2024-04-25T01:07:18.800 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T01:07:18.800 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/orig.config.yaml 2024-04-25T01:07:18.858 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:07:18.949 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672311' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ1/2jYADJmOZRMEHe0RipIy45LTe7GfMd4JyIT2cTCS704MzxsZ+aGY4HT/81vEwHosNnM0n3zXCmQoele0b2E= smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNnQ6ugAQ+jFR0CeCv+WpTaC9tg/RZzAFTMVg9lotTv1YCe9jGM6JHyD5A7k445jlHISkqZloSsytqvkmCjlew= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM70J+Oa/Oj10xv+r3o8Or7d5HKBYJd81EB5dsruig1sqkCi0Yx0vOdVxkR7QOvGforRd5BEQcD7M3VV1k+W0pc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - fs/test_o_trunc.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T01:07:18.949 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-25T01:07:18.949 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-25T01:07:18.950 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T01:07:18.951 INFO:teuthology.task.internal:Checking packages... 2024-04-25T01:07:18.974 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-25T01:07:18.974 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T01:07:18.974 INFO:teuthology.packaging:ref: None 2024-04-25T01:07:18.974 INFO:teuthology.packaging:tag: None 2024-04-25T01:07:18.975 INFO:teuthology.packaging:branch: squid 2024-04-25T01:07:18.975 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:07:18.975 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-25T01:07:19.173 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-25T01:07:19.175 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T01:07:19.182 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T01:07:19.182 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T01:07:19.194 INFO:teuthology.task.internal:Saving configuration 2024-04-25T01:07:19.215 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T01:07:19.221 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T01:07:19.246 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:00:51.107262', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ1/2jYADJmOZRMEHe0RipIy45LTe7GfMd4JyIT2cTCS704MzxsZ+aGY4HT/81vEwHosNnM0n3zXCmQoele0b2E='} 2024-04-25T01:07:19.269 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:00:51.111177', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNnQ6ugAQ+jFR0CeCv+WpTaC9tg/RZzAFTMVg9lotTv1YCe9jGM6JHyD5A7k445jlHISkqZloSsytqvkmCjlew='} 2024-04-25T01:07:19.294 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:00:51.109339', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM70J+Oa/Oj10xv+r3o8Or7d5HKBYJd81EB5dsruig1sqkCi0Yx0vOdVxkR7QOvGforRd5BEQcD7M3VV1k+W0pc='} 2024-04-25T01:07:19.294 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T01:07:19.301 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T01:07:19.301 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T01:07:19.301 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T01:07:19.301 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T01:07:19.406 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa5e2b34160>, signals=[15]) 2024-04-25T01:07:19.406 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T01:07:19.412 INFO:teuthology.task.internal:Opening connections... 2024-04-25T01:07:19.412 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:07:19.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:07:19.492 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2024-04-25T01:07:19.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:07:19.566 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-04-25T01:07:19.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:07:19.643 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T01:07:19.650 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-04-25T01:07:19.669 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-04-25T01:07:19.669 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-04-25T01:07:19.726 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:07:19.727 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:07:19.728 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-04-25T01:07:19.747 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2024-04-25T01:07:19.765 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2024-04-25T01:07:19.765 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2024-04-25T01:07:19.821 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="9" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="9" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:07:19.822 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:07:19.823 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2024-04-25T01:07:19.952 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-04-25T01:07:19.970 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-04-25T01:07:19.971 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-04-25T01:07:20.028 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-04-25T01:07:20.028 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-04-25T01:07:20.028 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-04-25T01:07:20.028 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:07:20.028 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-04-25T01:07:20.028 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:07:20.028 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:07:20.029 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-04-25T01:07:20.029 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:07:20.029 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:07:20.029 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:07:20.029 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:07:20.029 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:07:20.029 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:07:20.029 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-04-25T01:07:20.055 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T01:07:20.066 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T01:07:20.071 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T01:07:20.072 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:07:20.074 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:07:20.076 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:07:20.092 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T01:07:20.099 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T01:07:20.099 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:07:20.132 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:07:20.135 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:07:20.233 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T01:07:20.311 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T01:07:20.326 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-25T01:07:20.326 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-25T01:07:20.326 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-25T01:07:20.326 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:07:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:07:20.344 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-04-25T01:07:20.403 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:07:20.403 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-437.el9.x86_64 2024-04-25T01:07:20.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-04-25T01:07:22.615 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-25T01:07:24.341 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-04-25T01:07:28.717 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-04-25T01:07:31.077 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-25T01:07:33.213 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-25T01:07:38.410 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-04-25T01:07:40.300 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:07:40.300 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:07:40.392 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T01:07:40.393 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-25T01:07:40.393 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-25T01:07:40.539 DEBUG:teuthology.orchestra.run.smithi027:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:07:41.060 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:07:38 AM UTC. 2024-04-25T01:07:41.183 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T01:07:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalling: 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 255 k 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 0 2024-04-25T01:07:41.195 INFO:teuthology.orchestra.run.smithi027.stderr:Operation aborted. 2024-04-25T01:07:41.247 INFO:teuthology.orchestra.run.smithi027.stdout:Is this ok [y/N]: 2024-04-25T01:07:41.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum reinstall -y kernel || true 2024-04-25T01:07:41.763 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:07:38 AM UTC. 2024-04-25T01:07:41.884 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalling: 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 255 k 2024-04-25T01:07:41.885 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 0 2024-04-25T01:07:41.886 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-25T01:07:42.429 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-437.el9.x86_64.rpm 776 kB/s | 255 kB 00:00 2024-04-25T01:07:42.429 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:07:42.429 INFO:teuthology.orchestra.run.smithi027.stdout:Total 469 kB/s | 255 kB 00:00 2024-04-25T01:07:42.450 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-25T01:07:42.632 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-25T01:07:42.633 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-25T01:07:42.638 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-25T01:07:42.638 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-25T01:07:42.770 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-25T01:07:42.846 INFO:teuthology.orchestra.run.smithi027.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:07:42.847 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:07:43.571 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:07:43.572 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:07:44.250 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:07:44.250 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:07:44.250 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalled: 2024-04-25T01:07:44.251 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:07:44.251 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:07:44.251 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-25T01:07:44.401 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:07:44.448 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:07:44.449 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:07:44.449 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:07:44.449 DEBUG:teuthology.parallel:result is None 2024-04-25T01:07:44.449 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T01:07:44.460 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T01:07:44.460 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:07:44.463 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:07:44.465 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:07:44.483 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T01:07:44.490 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T01:07:44.521 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T01:07:44.522 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:07:44.524 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:07:44.526 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:07:44.858 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T01:07:44.865 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T01:07:44.865 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:07:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:07:44.882 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:07:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:07:44.899 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:07:44.901 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:07:44.924 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:07:44.932 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:07:44.947 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:07:44.955 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:07:44.973 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:07:44.997 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:07:45.014 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:07:45.015 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T01:07:45.023 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T01:07:45.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:07:45.026 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:07:45.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:07:45.055 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T01:07:45.064 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T01:07:45.064 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:07:45.091 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:07:45.094 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:07:45.111 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:07:45.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:07:45.231 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:07:45.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:07:45.352 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:07:45.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:07:45.416 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:07:45.462 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:07:45.525 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:07:45.568 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:07:45.607 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:07:45.608 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:07:45.672 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:07:45.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:07:45.773 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:07:45.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:07:45.901 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:07:45.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:07:45.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-04-25T01:07:45.969 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2024-04-25T01:07:45.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-04-25T01:07:46.001 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:07:46.005 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:07:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:07:46.494 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T01:07:46.502 INFO:teuthology.task.internal:Starting timer... 2024-04-25T01:07:46.502 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T01:07:46.512 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T01:07:46.519 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T01:07:46.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-04-25T01:07:46.597 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-04-25T01:07:46.599 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2024-04-25T01:07:46.689 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2024-04-25T01:07:46.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-04-25T01:07:46.765 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-04-25T01:07:46.767 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T01:07:46.767 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-04-25T01:07:46.791 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-04-25T01:07:46.791 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2024-04-25T01:07:46.818 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2024-04-25T01:07:46.818 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-04-25T01:07:46.871 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-04-25T01:07:46.871 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-04-25T01:07:46.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:07:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:07:46.897 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:07:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:07:46.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:07:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:07:46.952 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T01:07:46.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:07:46.977 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:07:47.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:07:47.027 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T01:07:47.037 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T01:07:47.053 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T01:07:47.054 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi139.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T01:11:49.470 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-04-25T01:11:49.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-25T01:11:49.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:11:49.551 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-04-25T01:11:49.591 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-25T01:11:49.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-04-25T01:11:49.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:11:49.666 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-04-25T01:11:49.749 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-04-25T01:11:49.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-25T01:11:49.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:11:49.826 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-25T01:11:49.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-25T01:11:49.906 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T01:11:49.917 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T01:11:49.918 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:11:49.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:11:49.920 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:11:49.920 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:11:49.922 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:11:49.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:11:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:11:49.949 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:11:49.962 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:11:49.963 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:11:49.976 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:11:49.985 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2024-04-25T01:11:49.986 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-04-25T01:11:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:11:49.995 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2024-04-25T01:11:49.997 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-04-25T01:11:50.009 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:11:50.011 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:11:50.013 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-04-25T01:11:50.022 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:11:50.023 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-04-25T01:11:50.024 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:11:50.037 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:11:50.050 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:11:50.074 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:11:50.077 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:11:50.077 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-04-25T01:11:50.078 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.078 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.078 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.078 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:11:50.086 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:11:50.086 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-25T01:11:50.086 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.086 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.086 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.086 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:11:50.111 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:11:50.111 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-25T01:11:50.112 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.112 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.112 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.112 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:11:50.113 INFO:teuthology.run_tasks:Running task install... 2024-04-25T01:11:50.121 DEBUG:teuthology.task.install:project ceph 2024-04-25T01:11:50.122 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T01:11:50.122 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T01:11:50.122 INFO:teuthology.task.install:Using flavor: default 2024-04-25T01:11:50.130 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T01:11:50.131 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T01:11:50.131 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:11:50.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:11:50.133 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:11:50.134 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:11:50.135 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:11:50.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:11:50.294 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:11:50.294 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:11:50.300 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:11:50.300 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:11:50.313 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:11:50.313 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:11:50.408 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:11:50.408 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:11:50.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:11:50.433 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:11:50.433 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:11:50.482 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:11:50.482 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:11:50.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:11:50.497 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:11:50.497 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:11:50.497 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:11:50.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:11:50.507 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:11:50.507 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:11:50.521 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:11:50.521 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:11:50.572 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:11:50.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:11:50.586 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:11:50.635 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-04-25T01:11:50.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-04-25T01:11:50.649 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:11:50.659 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:11:50.713 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-04-25T01:11:50.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-04-25T01:11:50.722 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2024-04-25T01:11:50.724 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2024-04-25T01:11:50.866 INFO:teuthology.orchestra.run.smithi153.stdout:48 files removed 2024-04-25T01:11:50.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:11:50.946 INFO:teuthology.orchestra.run.smithi139.stdout:48 files removed 2024-04-25T01:11:50.947 INFO:teuthology.orchestra.run.smithi027.stdout:48 files removed 2024-04-25T01:11:50.973 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:11:50.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:11:51.568 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 251 kB/s | 83 kB 00:00 2024-04-25T01:11:51.628 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 175 kB/s | 83 kB 00:00 2024-04-25T01:11:51.743 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 168 kB/s | 83 kB 00:00 2024-04-25T01:11:52.002 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-04-25T01:11:52.029 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-25T01:11:52.127 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-04-25T01:11:52.446 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-04-25T01:11:52.461 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-04-25T01:11:52.578 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-04-25T01:11:53.514 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-04-25T01:11:53.569 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-04-25T01:11:53.896 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-04-25T01:11:55.255 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-04-25T01:11:55.445 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-25T01:11:55.655 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-25T01:11:59.639 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-25T01:12:01.089 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-25T01:12:01.187 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.2 MB 00:02 2024-04-25T01:12:01.439 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.2 MB 00:02 2024-04-25T01:12:02.688 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-04-25T01:12:04.031 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-04-25T01:12:04.825 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-25T01:12:05.519 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-25T01:12:09.914 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-25T01:12:10.624 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-25T01:12:12.048 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:12:12.048 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:12:12.060 INFO:teuthology.orchestra.run.smithi139.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:12:12.060 INFO:teuthology.orchestra.run.smithi139.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:12:12.062 INFO:teuthology.orchestra.run.smithi139.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:12:12.062 INFO:teuthology.orchestra.run.smithi139.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:12:12.064 INFO:teuthology.orchestra.run.smithi139.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:12:12.066 INFO:teuthology.orchestra.run.smithi139.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:12:12.067 INFO:teuthology.orchestra.run.smithi139.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:12:12.128 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-04-25T01:12:12.156 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-04-25T01:12:12.156 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-04-25T01:12:12.156 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-04-25T01:12:12.156 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-04-25T01:12:12.156 INFO:teuthology.orchestra.run.smithi139.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:12:12.156 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:12:12.156 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:12:12.156 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:12:12.157 INFO:teuthology.orchestra.run.smithi139.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:12:12.158 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:12:12.159 INFO:teuthology.orchestra.run.smithi139.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:12:12.160 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:12:12.161 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:12:12.162 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:12:12.163 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:12:12.164 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:12:12.165 INFO:teuthology.orchestra.run.smithi139.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:12:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:12:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout:Install 148 Packages 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 4 Packages 2024-04-25T01:12:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:12:12.169 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 220 M 2024-04-25T01:12:12.169 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-04-25T01:12:12.753 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:12:12.753 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:12:12.765 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:12:12.765 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:12:12.766 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:12:12.767 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:12:12.770 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:12:12.771 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:12:12.772 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:12:12.832 INFO:teuthology.orchestra.run.smithi139.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-04-25T01:12:12.834 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:12:12.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:12:12.851 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:12:12.851 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:12:12.851 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:12:12.851 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:12:12.851 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:12:12.852 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-04-25T01:12:12.853 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:12:12.854 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:12:12.855 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:12:12.856 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:12:12.857 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:12:12.858 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:12:12.859 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:12:12.860 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:12:12.861 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-04-25T01:12:12.862 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:12:12.862 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:12:12.862 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-25T01:12:12.862 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-25T01:12:12.862 INFO:teuthology.orchestra.run.smithi153.stdout:Install 148 Packages 2024-04-25T01:12:12.862 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 4 Packages 2024-04-25T01:12:12.862 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:12:12.862 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 220 M 2024-04-25T01:12:12.863 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-25T01:12:13.140 INFO:teuthology.orchestra.run.smithi139.stdout:(2/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 9.5 MB/s | 5.4 MB 00:00 2024-04-25T01:12:13.198 INFO:teuthology.orchestra.run.smithi139.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.4 MB/s | 145 kB 00:00 2024-04-25T01:12:13.298 INFO:teuthology.orchestra.run.smithi139.stdout:(4/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.9 MB/s | 890 kB 00:00 2024-04-25T01:12:13.390 INFO:teuthology.orchestra.run.smithi139.stdout:(5/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 12 MB/s | 2.4 MB 00:00 2024-04-25T01:12:13.686 INFO:teuthology.orchestra.run.smithi153.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-25T01:12:13.749 INFO:teuthology.orchestra.run.smithi139.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 13 MB/s | 4.7 MB 00:00 2024-04-25T01:12:13.883 INFO:teuthology.orchestra.run.smithi139.stdout:(7/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-04-25T01:12:14.886 INFO:teuthology.orchestra.run.smithi153.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 742 kB/s | 890 kB 00:01 2024-04-25T01:12:15.028 INFO:teuthology.orchestra.run.smithi153.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.0 MB/s | 145 kB 00:00 2024-04-25T01:12:15.033 INFO:teuthology.orchestra.run.smithi139.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 13 MB/s | 17 MB 00:01 2024-04-25T01:12:15.075 INFO:teuthology.orchestra.run.smithi139.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 609 kB/s | 25 kB 00:00 2024-04-25T01:12:15.229 INFO:teuthology.orchestra.run.smithi153.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 2.9 MB/s | 5.4 MB 00:01 2024-04-25T01:12:15.545 INFO:teuthology.orchestra.run.smithi153.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-04-25T01:12:16.279 INFO:teuthology.orchestra.run.smithi153.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 6.4 MB/s | 4.7 MB 00:00 2024-04-25T01:12:17.225 INFO:teuthology.orchestra.run.smithi139.stdout:(10/152): ceph-common-19.0.0-2445.gd2c45d97.el9 4.5 MB/s | 21 MB 00:04 2024-04-25T01:12:17.429 INFO:teuthology.orchestra.run.smithi153.stdout:(7/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 1.0 MB/s | 2.4 MB 00:02 2024-04-25T01:12:18.000 INFO:teuthology.orchestra.run.smithi139.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 287 kB/s | 222 kB 00:00 2024-04-25T01:12:18.021 INFO:teuthology.orchestra.run.smithi153.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 9.6 MB/s | 17 MB 00:01 2024-04-25T01:12:18.075 INFO:teuthology.orchestra.run.smithi139.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 362 kB/s | 27 kB 00:00 2024-04-25T01:12:18.096 INFO:teuthology.orchestra.run.smithi153.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 335 kB/s | 25 kB 00:00 2024-04-25T01:12:18.200 INFO:teuthology.orchestra.run.smithi139.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 254 kB/s | 32 kB 00:00 2024-04-25T01:12:18.809 INFO:teuthology.orchestra.run.smithi139.stdout:(14/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 13 MB/s | 48 MB 00:03 2024-04-25T01:12:18.842 INFO:teuthology.orchestra.run.smithi139.stdout:(15/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 1.1 MB/s | 745 kB 00:00 2024-04-25T01:12:18.885 INFO:teuthology.orchestra.run.smithi139.stdout:(16/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.1 MB/s | 163 kB 00:00 2024-04-25T01:12:18.927 INFO:teuthology.orchestra.run.smithi139.stdout:(17/152): librados-devel-19.0.0-2445.gd2c45d97. 1.5 MB/s | 127 kB 00:00 2024-04-25T01:12:19.011 INFO:teuthology.orchestra.run.smithi139.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 3.9 MB/s | 501 kB 00:00 2024-04-25T01:12:19.078 INFO:teuthology.orchestra.run.smithi139.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 679 kB/s | 45 kB 00:00 2024-04-25T01:12:19.145 INFO:teuthology.orchestra.run.smithi139.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 2.0 MB/s | 133 kB 00:00 2024-04-25T01:12:19.212 INFO:teuthology.orchestra.run.smithi139.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.4 MB/s | 162 kB 00:00 2024-04-25T01:12:19.287 INFO:teuthology.orchestra.run.smithi139.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.2 MB/s | 321 kB 00:00 2024-04-25T01:12:19.362 INFO:teuthology.orchestra.run.smithi139.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.9 MB/s | 299 kB 00:00 2024-04-25T01:12:19.487 INFO:teuthology.orchestra.run.smithi139.stdout:(24/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 1.8 MB/s | 10 MB 00:05 2024-04-25T01:12:19.512 INFO:teuthology.orchestra.run.smithi139.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 663 kB/s | 99 kB 00:00 2024-04-25T01:12:19.571 INFO:teuthology.orchestra.run.smithi139.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-25T01:12:19.805 INFO:teuthology.orchestra.run.smithi139.stdout:(27/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 11 MB/s | 3.1 MB 00:00 2024-04-25T01:12:19.830 INFO:teuthology.orchestra.run.smithi139.stdout:(28/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 662 kB/s | 171 kB 00:00 2024-04-25T01:12:19.880 INFO:teuthology.orchestra.run.smithi139.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 423 kB/s | 32 kB 00:00 2024-04-25T01:12:19.906 INFO:teuthology.orchestra.run.smithi139.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.8 MB/s | 141 kB 00:00 2024-04-25T01:12:20.656 INFO:teuthology.orchestra.run.smithi139.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 9.9 MB/s | 7.4 MB 00:00 2024-04-25T01:12:20.781 INFO:teuthology.orchestra.run.smithi139.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.0 MB/s | 249 kB 00:00 2024-04-25T01:12:20.848 INFO:teuthology.orchestra.run.smithi139.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 740 kB/s | 49 kB 00:00 2024-04-25T01:12:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-04-25T01:12:20.915 INFO:teuthology.orchestra.run.smithi139.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 230 kB/s | 15 kB 00:00 2024-04-25T01:12:20.990 INFO:teuthology.orchestra.run.smithi139.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.4 MB/s | 264 kB 00:00 2024-04-25T01:12:21.090 INFO:teuthology.orchestra.run.smithi139.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 7.5 MB/s | 762 kB 00:00 2024-04-25T01:12:21.132 INFO:teuthology.orchestra.run.smithi139.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 437 kB/s | 18 kB 00:00 2024-04-25T01:12:21.241 INFO:teuthology.orchestra.run.smithi139.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 741 kB/s | 80 kB 00:00 2024-04-25T01:12:21.350 INFO:teuthology.orchestra.run.smithi139.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.3 MB/s | 803 kB 00:00 2024-04-25T01:12:21.392 INFO:teuthology.orchestra.run.smithi139.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-25T01:12:21.450 INFO:teuthology.orchestra.run.smithi139.stdout:(41/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 2.2 MB/s | 3.5 MB 00:01 2024-04-25T01:12:21.476 INFO:teuthology.orchestra.run.smithi139.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-25T01:12:21.518 INFO:teuthology.orchestra.run.smithi139.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-25T01:12:21.529 INFO:teuthology.orchestra.run.smithi153.stdout:(10/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 14 MB/s | 48 MB 00:03 2024-04-25T01:12:21.559 INFO:teuthology.orchestra.run.smithi139.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T01:12:21.602 INFO:teuthology.orchestra.run.smithi139.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T01:12:21.621 INFO:teuthology.orchestra.run.smithi153.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 222 kB 00:00 2024-04-25T01:12:21.652 INFO:teuthology.orchestra.run.smithi139.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 6.1 MB/s | 1.2 MB 00:00 2024-04-25T01:12:21.677 INFO:teuthology.orchestra.run.smithi139.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-25T01:12:21.714 INFO:teuthology.orchestra.run.smithi153.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 295 kB/s | 27 kB 00:00 2024-04-25T01:12:21.798 INFO:teuthology.orchestra.run.smithi153.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 381 kB/s | 32 kB 00:00 2024-04-25T01:12:21.914 INFO:teuthology.orchestra.run.smithi153.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 6.2 MB/s | 745 kB 00:00 2024-04-25T01:12:21.980 INFO:teuthology.orchestra.run.smithi153.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 163 kB 00:00 2024-04-25T01:12:22.048 INFO:teuthology.orchestra.run.smithi153.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.9 MB/s | 127 kB 00:00 2024-04-25T01:12:22.060 INFO:teuthology.orchestra.run.smithi139.stdout:(48/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 1.7 MB/s | 5.3 MB 00:03 2024-04-25T01:12:22.123 INFO:teuthology.orchestra.run.smithi153.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 6.5 MB/s | 501 kB 00:00 2024-04-25T01:12:22.453 INFO:teuthology.orchestra.run.smithi139.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 861 kB/s | 689 kB 00:00 2024-04-25T01:12:22.473 INFO:teuthology.orchestra.run.smithi153.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 15 MB/s | 5.3 MB 00:00 2024-04-25T01:12:22.494 INFO:teuthology.orchestra.run.smithi139.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-04-25T01:12:22.520 INFO:teuthology.orchestra.run.smithi139.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 233 kB/s | 107 kB 00:00 2024-04-25T01:12:22.540 INFO:teuthology.orchestra.run.smithi153.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 680 kB/s | 45 kB 00:00 2024-04-25T01:12:22.553 INFO:teuthology.orchestra.run.smithi139.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 214 kB/s | 21 kB 00:00 2024-04-25T01:12:22.604 INFO:teuthology.orchestra.run.smithi139.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-25T01:12:22.607 INFO:teuthology.orchestra.run.smithi153.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.9 MB/s | 133 kB 00:00 2024-04-25T01:12:22.629 INFO:teuthology.orchestra.run.smithi139.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 300 kB/s | 33 kB 00:00 2024-04-25T01:12:22.674 INFO:teuthology.orchestra.run.smithi153.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.4 MB/s | 162 kB 00:00 2024-04-25T01:12:22.721 INFO:teuthology.orchestra.run.smithi139.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 18 kB 00:00 2024-04-25T01:12:22.741 INFO:teuthology.orchestra.run.smithi153.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.7 MB/s | 321 kB 00:00 2024-04-25T01:12:22.746 INFO:teuthology.orchestra.run.smithi139.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-04-25T01:12:22.824 INFO:teuthology.orchestra.run.smithi153.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-25T01:12:22.871 INFO:teuthology.orchestra.run.smithi139.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-04-25T01:12:22.897 INFO:teuthology.orchestra.run.smithi139.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 301 kB/s | 45 kB 00:00 2024-04-25T01:12:22.908 INFO:teuthology.orchestra.run.smithi153.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.2 MB/s | 99 kB 00:00 2024-04-25T01:12:23.013 INFO:teuthology.orchestra.run.smithi139.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-04-25T01:12:23.080 INFO:teuthology.orchestra.run.smithi139.stdout:(60/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.7 MB/s | 3.0 MB 00:00 2024-04-25T01:12:23.092 INFO:teuthology.orchestra.run.smithi153.stdout:(25/152): ceph-common-19.0.0-2445.gd2c45d97.el9 2.2 MB/s | 21 MB 00:09 2024-04-25T01:12:23.114 INFO:teuthology.orchestra.run.smithi139.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 253 kB/s | 25 kB 00:00 2024-04-25T01:12:23.117 INFO:teuthology.orchestra.run.smithi153.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 409 kB/s | 85 kB 00:00 2024-04-25T01:12:23.217 INFO:teuthology.orchestra.run.smithi153.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-25T01:12:23.234 INFO:teuthology.orchestra.run.smithi139.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 315 kB/s | 47 kB 00:00 2024-04-25T01:12:23.309 INFO:teuthology.orchestra.run.smithi153.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 347 kB/s | 32 kB 00:00 2024-04-25T01:12:23.356 INFO:teuthology.orchestra.run.smithi139.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-04-25T01:12:23.393 INFO:teuthology.orchestra.run.smithi153.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.7 MB/s | 141 kB 00:00 2024-04-25T01:12:23.490 INFO:teuthology.orchestra.run.smithi139.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 264 kB/s | 35 kB 00:00 2024-04-25T01:12:23.743 INFO:teuthology.orchestra.run.smithi153.stdout:(30/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 1.6 MB/s | 10 MB 00:06 2024-04-25T01:12:23.802 INFO:teuthology.orchestra.run.smithi153.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 8.6 MB/s | 3.5 MB 00:00 2024-04-25T01:12:23.902 INFO:teuthology.orchestra.run.smithi153.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.4 MB/s | 249 kB 00:00 2024-04-25T01:12:24.011 INFO:teuthology.orchestra.run.smithi153.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 456 kB/s | 49 kB 00:00 2024-04-25T01:12:24.103 INFO:teuthology.orchestra.run.smithi153.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 167 kB/s | 15 kB 00:00 2024-04-25T01:12:24.186 INFO:teuthology.orchestra.run.smithi153.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.1 MB/s | 264 kB 00:00 2024-04-25T01:12:24.336 INFO:teuthology.orchestra.run.smithi153.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-25T01:12:24.403 INFO:teuthology.orchestra.run.smithi153.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 274 kB/s | 18 kB 00:00 2024-04-25T01:12:24.415 INFO:teuthology.orchestra.run.smithi139.stdout:(65/152): libtool-2.4.6-45.el9.x86_64.rpm 384 kB/s | 582 kB 00:01 2024-04-25T01:12:24.615 INFO:teuthology.orchestra.run.smithi139.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 195 kB/s | 292 kB 00:01 2024-04-25T01:12:24.854 INFO:teuthology.orchestra.run.smithi153.stdout:(38/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 1.8 MB/s | 3.1 MB 00:01 2024-04-25T01:12:24.879 INFO:teuthology.orchestra.run.smithi153.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 169 kB/s | 80 kB 00:00 2024-04-25T01:12:25.057 INFO:teuthology.orchestra.run.smithi139.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-04-25T01:12:25.354 INFO:teuthology.orchestra.run.smithi153.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-04-25T01:12:25.384 INFO:teuthology.orchestra.run.smithi139.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 2.7 MB/s | 911 kB 00:00 2024-04-25T01:12:25.463 INFO:teuthology.orchestra.run.smithi153.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 335 kB/s | 195 kB 00:00 2024-04-25T01:12:25.488 INFO:teuthology.orchestra.run.smithi153.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-04-25T01:12:25.493 INFO:teuthology.orchestra.run.smithi139.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-25T01:12:25.605 INFO:teuthology.orchestra.run.smithi153.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 913 kB/s | 106 kB 00:00 2024-04-25T01:12:25.714 INFO:teuthology.orchestra.run.smithi153.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-25T01:12:25.831 INFO:teuthology.orchestra.run.smithi153.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-04-25T01:12:25.948 INFO:teuthology.orchestra.run.smithi153.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 2.5 MB/s | 1.2 MB 00:00 2024-04-25T01:12:25.973 INFO:teuthology.orchestra.run.smithi153.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.5 MB/s | 215 kB 00:00 2024-04-25T01:12:26.081 INFO:teuthology.orchestra.run.smithi153.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 5.0 MB/s | 689 kB 00:00 2024-04-25T01:12:26.105 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-04-25T01:12:26.190 INFO:teuthology.orchestra.run.smithi153.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 989 kB/s | 107 kB 00:00 2024-04-25T01:12:26.341 INFO:teuthology.orchestra.run.smithi153.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 2.5 MB/s | 943 kB 00:00 2024-04-25T01:12:26.366 INFO:teuthology.orchestra.run.smithi153.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 122 kB/s | 21 kB 00:00 2024-04-25T01:12:26.458 INFO:teuthology.orchestra.run.smithi153.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-25T01:12:26.491 INFO:teuthology.orchestra.run.smithi153.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-04-25T01:12:26.608 INFO:teuthology.orchestra.run.smithi153.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 18 kB 00:00 2024-04-25T01:12:26.717 INFO:teuthology.orchestra.run.smithi153.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-25T01:12:26.742 INFO:teuthology.orchestra.run.smithi153.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 83 kB/s | 11 kB 00:00 2024-04-25T01:12:26.851 INFO:teuthology.orchestra.run.smithi153.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 419 kB/s | 45 kB 00:00 2024-04-25T01:12:26.968 INFO:teuthology.orchestra.run.smithi153.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-25T01:12:26.993 INFO:teuthology.orchestra.run.smithi153.stdout:(59/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 581 kB/s | 160 kB 00:00 2024-04-25T01:12:27.085 INFO:teuthology.orchestra.run.smithi153.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-25T01:12:27.110 INFO:teuthology.orchestra.run.smithi153.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 217 kB/s | 25 kB 00:00 2024-04-25T01:12:27.118 INFO:teuthology.orchestra.run.smithi139.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 3.6 MB/s | 5.8 MB 00:01 2024-04-25T01:12:27.244 INFO:teuthology.orchestra.run.smithi153.stdout:(62/152): ceph-mgr-diskprediction-local-19.0.0- 2.1 MB/s | 7.4 MB 00:03 2024-04-25T01:12:27.269 INFO:teuthology.orchestra.run.smithi153.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 258 kB/s | 47 kB 00:00 2024-04-25T01:12:27.378 INFO:teuthology.orchestra.run.smithi153.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 325 kB/s | 35 kB 00:00 2024-04-25T01:12:27.402 INFO:teuthology.orchestra.run.smithi139.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-04-25T01:12:27.510 INFO:teuthology.orchestra.run.smithi139.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 809 kB/s | 87 kB 00:00 2024-04-25T01:12:27.578 INFO:teuthology.orchestra.run.smithi153.stdout:(65/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 626 kB/s | 292 kB 00:00 2024-04-25T01:12:27.795 INFO:teuthology.orchestra.run.smithi153.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 342 kB/s | 188 kB 00:00 2024-04-25T01:12:28.088 INFO:teuthology.orchestra.run.smithi139.stdout:(73/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.2 MB/s | 4.4 MB 00:03 2024-04-25T01:12:28.194 INFO:teuthology.orchestra.run.smithi139.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-04-25T01:12:28.245 INFO:teuthology.orchestra.run.smithi153.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-04-25T01:12:28.319 INFO:teuthology.orchestra.run.smithi139.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-04-25T01:12:28.323 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:12:28.324 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:12:28.335 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:12:28.336 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:12:28.337 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:12:28.338 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:12:28.339 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:12:28.342 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:12:28.342 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:12:28.370 INFO:teuthology.orchestra.run.smithi153.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 7.1 MB/s | 911 kB 00:00 2024-04-25T01:12:28.407 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T01:12:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T01:12:28.416 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-25T01:12:28.416 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:12:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:12:28.418 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-04-25T01:12:28.419 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:12:28.420 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:12:28.421 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:12:28.422 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:12:28.423 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:12:28.424 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:12:28.425 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:12:28.426 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:12:28.427 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:12:28.428 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-25T01:12:28.428 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T01:12:28.428 INFO:teuthology.orchestra.run.smithi027.stdout:Install 148 Packages 2024-04-25T01:12:28.428 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 4 Packages 2024-04-25T01:12:28.428 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:12:28.428 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 220 M 2024-04-25T01:12:28.428 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-25T01:12:28.436 INFO:teuthology.orchestra.run.smithi139.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 220 kB/s | 26 kB 00:00 2024-04-25T01:12:28.504 INFO:teuthology.orchestra.run.smithi153.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-04-25T01:12:28.553 INFO:teuthology.orchestra.run.smithi139.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 164 kB/s | 19 kB 00:00 2024-04-25T01:12:28.662 INFO:teuthology.orchestra.run.smithi139.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-04-25T01:12:28.778 INFO:teuthology.orchestra.run.smithi139.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 298 kB/s | 35 kB 00:00 2024-04-25T01:12:28.854 INFO:teuthology.orchestra.run.smithi139.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-04-25T01:12:28.971 INFO:teuthology.orchestra.run.smithi139.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-25T01:12:29.079 INFO:teuthology.orchestra.run.smithi139.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-25T01:12:29.180 INFO:teuthology.orchestra.run.smithi139.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 772 kB/s | 77 kB 00:00 2024-04-25T01:12:29.257 INFO:teuthology.orchestra.run.smithi027.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-04-25T01:12:29.280 INFO:teuthology.orchestra.run.smithi139.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 421 kB/s | 42 kB 00:00 2024-04-25T01:12:29.397 INFO:teuthology.orchestra.run.smithi139.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-04-25T01:12:29.464 INFO:teuthology.orchestra.run.smithi139.stdout:(86/152): openblas-openmp-0.3.21-2.el9.x86_64.r 812 kB/s | 4.7 MB 00:05 2024-04-25T01:12:29.506 INFO:teuthology.orchestra.run.smithi139.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-25T01:12:29.573 INFO:teuthology.orchestra.run.smithi139.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-25T01:12:29.588 INFO:teuthology.orchestra.run.smithi153.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 2.1 MB/s | 4.7 MB 00:02 2024-04-25T01:12:29.606 INFO:teuthology.orchestra.run.smithi139.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 508 kB/s | 51 kB 00:00 2024-04-25T01:12:29.682 INFO:teuthology.orchestra.run.smithi139.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 493 kB/s | 54 kB 00:00 2024-04-25T01:12:29.698 INFO:teuthology.orchestra.run.smithi027.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 2.0 MB/s | 890 kB 00:00 2024-04-25T01:12:29.790 INFO:teuthology.orchestra.run.smithi139.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 385 kB/s | 42 kB 00:00 2024-04-25T01:12:29.849 INFO:teuthology.orchestra.run.smithi027.stdout:(3/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 6.6 MB/s | 5.4 MB 00:00 2024-04-25T01:12:29.874 INFO:teuthology.orchestra.run.smithi027.stdout:(4/152): ceph-immutable-object-cache-19.0.0-244 826 kB/s | 145 kB 00:00 2024-04-25T01:12:29.899 INFO:teuthology.orchestra.run.smithi139.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 407 kB/s | 44 kB 00:00 2024-04-25T01:12:30.007 INFO:teuthology.orchestra.run.smithi139.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-25T01:12:30.075 INFO:teuthology.orchestra.run.smithi027.stdout:(5/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 10 MB/s | 2.4 MB 00:00 2024-04-25T01:12:30.116 INFO:teuthology.orchestra.run.smithi139.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 331 kB/s | 36 kB 00:00 2024-04-25T01:12:30.225 INFO:teuthology.orchestra.run.smithi139.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-04-25T01:12:30.308 INFO:teuthology.orchestra.run.smithi027.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 3.4 MB/s | 1.5 MB 00:00 2024-04-25T01:12:30.333 INFO:teuthology.orchestra.run.smithi139.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-04-25T01:12:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 14 MB/s | 4.7 MB 00:00 2024-04-25T01:12:30.442 INFO:teuthology.orchestra.run.smithi139.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 534 kB/s | 58 kB 00:00 2024-04-25T01:12:30.559 INFO:teuthology.orchestra.run.smithi139.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 804 kB/s | 94 kB 00:00 2024-04-25T01:12:30.796 INFO:teuthology.orchestra.run.smithi139.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 269 kB/s | 64 kB 00:00 2024-04-25T01:12:30.910 INFO:teuthology.orchestra.run.smithi139.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 397 kB/s | 45 kB 00:00 2024-04-25T01:12:31.004 INFO:teuthology.orchestra.run.smithi153.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 729 kB/s | 1.0 MB 00:01 2024-04-25T01:12:31.110 INFO:teuthology.orchestra.run.smithi139.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-04-25T01:12:31.177 INFO:teuthology.orchestra.run.smithi139.stdout:(102/152): python3-numpy-1.20.1-5.el9.x86_64.rp 2.1 MB/s | 5.1 MB 00:02 2024-04-25T01:12:31.196 INFO:teuthology.orchestra.run.smithi153.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 457 kB/s | 87 kB 00:00 2024-04-25T01:12:31.252 INFO:teuthology.orchestra.run.smithi027.stdout:(8/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el9 12 MB/s | 10 MB 00:00 2024-04-25T01:12:31.326 INFO:teuthology.orchestra.run.smithi027.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 343 kB/s | 25 kB 00:00 2024-04-25T01:12:31.436 INFO:teuthology.orchestra.run.smithi139.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-04-25T01:12:31.511 INFO:teuthology.orchestra.run.smithi139.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 4.0 MB/s | 309 kB 00:00 2024-04-25T01:12:31.536 INFO:teuthology.orchestra.run.smithi139.stdout:(105/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-04-25T01:12:31.561 INFO:teuthology.orchestra.run.smithi139.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 385 kB/s | 19 kB 00:00 2024-04-25T01:12:31.620 INFO:teuthology.orchestra.run.smithi139.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-04-25T01:12:31.678 INFO:teuthology.orchestra.run.smithi139.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-04-25T01:12:31.754 INFO:teuthology.orchestra.run.smithi139.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 900 kB/s | 67 kB 00:00 2024-04-25T01:12:31.830 INFO:teuthology.orchestra.run.smithi139.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-25T01:12:31.946 INFO:teuthology.orchestra.run.smithi139.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-25T01:12:32.013 INFO:teuthology.orchestra.run.smithi139.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-04-25T01:12:32.038 INFO:teuthology.orchestra.run.smithi139.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-04-25T01:12:32.064 INFO:teuthology.orchestra.run.smithi139.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 868 kB/s | 43 kB 00:00 2024-04-25T01:12:32.088 INFO:teuthology.orchestra.run.smithi139.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-04-25T01:12:32.114 INFO:teuthology.orchestra.run.smithi139.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T01:12:32.281 INFO:teuthology.orchestra.run.smithi139.stdout:(117/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.1 MB/s | 16 MB 00:02 2024-04-25T01:12:32.315 INFO:teuthology.orchestra.run.smithi139.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 761 kB/s | 172 kB 00:00 2024-04-25T01:12:32.348 INFO:teuthology.orchestra.run.smithi139.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 1.5 MB/s | 358 kB 00:00 2024-04-25T01:12:32.373 INFO:teuthology.orchestra.run.smithi139.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 604 kB/s | 35 kB 00:00 2024-04-25T01:12:32.407 INFO:teuthology.orchestra.run.smithi139.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-25T01:12:32.435 INFO:teuthology.orchestra.run.smithi027.stdout:(10/152): ceph-common-19.0.0-2445.gd2c45d97.el9 6.2 MB/s | 21 MB 00:03 2024-04-25T01:12:32.491 INFO:teuthology.orchestra.run.smithi139.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-25T01:12:32.518 INFO:teuthology.orchestra.run.smithi027.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.6 MB/s | 222 kB 00:00 2024-04-25T01:12:32.533 INFO:teuthology.orchestra.run.smithi139.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-04-25T01:12:32.549 INFO:teuthology.orchestra.run.smithi139.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-25T01:12:32.575 INFO:teuthology.orchestra.run.smithi139.stdout:(125/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.0 MB/s | 303 kB 00:00 2024-04-25T01:12:32.592 INFO:teuthology.orchestra.run.smithi139.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 300 kB/s | 18 kB 00:00 2024-04-25T01:12:32.593 INFO:teuthology.orchestra.run.smithi027.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 362 kB/s | 27 kB 00:00 2024-04-25T01:12:32.609 INFO:teuthology.orchestra.run.smithi139.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 391 kB/s | 23 kB 00:00 2024-04-25T01:12:32.626 INFO:teuthology.orchestra.run.smithi139.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-04-25T01:12:32.642 INFO:teuthology.orchestra.run.smithi139.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-25T01:12:32.669 INFO:teuthology.orchestra.run.smithi027.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 422 kB/s | 32 kB 00:00 2024-04-25T01:12:32.685 INFO:teuthology.orchestra.run.smithi139.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-04-25T01:12:32.710 INFO:teuthology.orchestra.run.smithi139.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 553 kB/s | 46 kB 00:00 2024-04-25T01:12:32.735 INFO:teuthology.orchestra.run.smithi139.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 854 kB/s | 79 kB 00:00 2024-04-25T01:12:32.760 INFO:teuthology.orchestra.run.smithi139.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-04-25T01:12:32.777 INFO:teuthology.orchestra.run.smithi027.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 6.7 MB/s | 745 kB 00:00 2024-04-25T01:12:32.786 INFO:teuthology.orchestra.run.smithi139.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 329 kB/s | 16 kB 00:00 2024-04-25T01:12:32.811 INFO:teuthology.orchestra.run.smithi139.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-25T01:12:32.836 INFO:teuthology.orchestra.run.smithi139.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-25T01:12:32.861 INFO:teuthology.orchestra.run.smithi027.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.9 MB/s | 163 kB 00:00 2024-04-25T01:12:32.861 INFO:teuthology.orchestra.run.smithi139.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-25T01:12:32.887 INFO:teuthology.orchestra.run.smithi139.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-25T01:12:32.912 INFO:teuthology.orchestra.run.smithi139.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-25T01:12:32.937 INFO:teuthology.orchestra.run.smithi139.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-04-25T01:12:32.962 INFO:teuthology.orchestra.run.smithi139.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-04-25T01:12:32.988 INFO:teuthology.orchestra.run.smithi139.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-25T01:12:33.021 INFO:teuthology.orchestra.run.smithi139.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-25T01:12:33.045 INFO:teuthology.orchestra.run.smithi027.stdout:(16/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 6.1 MB/s | 17 MB 00:02 2024-04-25T01:12:33.047 INFO:teuthology.orchestra.run.smithi139.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-04-25T01:12:33.070 INFO:teuthology.orchestra.run.smithi027.stdout:(17/152): librados-devel-19.0.0-2445.gd2c45d97. 606 kB/s | 127 kB 00:00 2024-04-25T01:12:33.072 INFO:teuthology.orchestra.run.smithi139.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-25T01:12:33.097 INFO:teuthology.orchestra.run.smithi139.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-04-25T01:12:33.122 INFO:teuthology.orchestra.run.smithi139.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-25T01:12:33.154 INFO:teuthology.orchestra.run.smithi027.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 4.5 MB/s | 501 kB 00:00 2024-04-25T01:12:33.172 INFO:teuthology.orchestra.run.smithi139.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T01:12:33.246 INFO:teuthology.orchestra.run.smithi027.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 492 kB/s | 45 kB 00:00 2024-04-25T01:12:33.332 INFO:teuthology.orchestra.run.smithi139.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 8.5 MB/s | 1.3 MB 00:00 2024-04-25T01:12:33.338 INFO:teuthology.orchestra.run.smithi027.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.4 MB/s | 133 kB 00:00 2024-04-25T01:12:33.415 INFO:teuthology.orchestra.run.smithi139.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-25T01:12:33.430 INFO:teuthology.orchestra.run.smithi027.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.7 MB/s | 162 kB 00:00 2024-04-25T01:12:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 321 kB 00:00 2024-04-25T01:12:33.613 INFO:teuthology.orchestra.run.smithi153.stdout:(73/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 751 kB/s | 4.4 MB 00:06 2024-04-25T01:12:33.680 INFO:teuthology.orchestra.run.smithi027.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 2.5 MB/s | 299 kB 00:00 2024-04-25T01:12:33.747 INFO:teuthology.orchestra.run.smithi027.stdout:(24/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 7.9 MB/s | 5.3 MB 00:00 2024-04-25T01:12:33.789 INFO:teuthology.orchestra.run.smithi027.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 917 kB/s | 99 kB 00:00 2024-04-25T01:12:33.807 INFO:teuthology.orchestra.run.smithi139.stdout:(151/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 4.6 MB/s | 3.1 MB 00:00 2024-04-25T01:12:33.823 INFO:teuthology.orchestra.run.smithi027.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-25T01:12:33.906 INFO:teuthology.orchestra.run.smithi027.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-25T01:12:33.963 INFO:teuthology.orchestra.run.smithi153.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 698 kB/s | 244 kB 00:00 2024-04-25T01:12:34.106 INFO:teuthology.orchestra.run.smithi027.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 159 kB/s | 32 kB 00:00 2024-04-25T01:12:34.196 INFO:teuthology.orchestra.run.smithi153.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 1.0 MB/s | 5.8 MB 00:05 2024-04-25T01:12:34.198 INFO:teuthology.orchestra.run.smithi027.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.5 MB/s | 141 kB 00:00 2024-04-25T01:12:34.249 INFO:teuthology.orchestra.run.smithi027.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 6.7 MB/s | 3.1 MB 00:00 2024-04-25T01:12:34.313 INFO:teuthology.orchestra.run.smithi153.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 220 kB/s | 26 kB 00:00 2024-04-25T01:12:34.338 INFO:teuthology.orchestra.run.smithi153.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 660 kB/s | 248 kB 00:00 2024-04-25T01:12:34.422 INFO:teuthology.orchestra.run.smithi153.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 176 kB/s | 19 kB 00:00 2024-04-25T01:12:34.531 INFO:teuthology.orchestra.run.smithi153.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 321 kB/s | 35 kB 00:00 2024-04-25T01:12:34.582 INFO:teuthology.orchestra.run.smithi139.stdout:(152/152): librados2-19.0.0-2445.gd2c45d97.el9. 2.3 MB/s | 3.4 MB 00:01 2024-04-25T01:12:34.585 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:12:34.585 INFO:teuthology.orchestra.run.smithi139.stdout:Total 9.8 MB/s | 220 MB 00:22 2024-04-25T01:12:34.597 INFO:teuthology.orchestra.run.smithi153.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 665 kB/s | 172 kB 00:00 2024-04-25T01:12:34.658 INFO:teuthology.orchestra.run.smithi027.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 7.7 MB/s | 3.5 MB 00:00 2024-04-25T01:12:34.741 INFO:teuthology.orchestra.run.smithi027.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.9 MB/s | 249 kB 00:00 2024-04-25T01:12:34.817 INFO:teuthology.orchestra.run.smithi027.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 658 kB/s | 49 kB 00:00 2024-04-25T01:12:34.948 INFO:teuthology.orchestra.run.smithi153.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 716 kB/s | 250 kB 00:00 2024-04-25T01:12:35.298 INFO:teuthology.orchestra.run.smithi153.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 635 kB/s | 222 kB 00:00 2024-04-25T01:12:35.300 INFO:teuthology.orchestra.run.smithi027.stdout:(34/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 12 MB/s | 48 MB 00:03 2024-04-25T01:12:35.326 INFO:teuthology.orchestra.run.smithi027.stdout:(35/152): ceph-prometheus-alerts-19.0.0-2445.gd 30 kB/s | 15 kB 00:00 2024-04-25T01:12:35.359 INFO:teuthology.orchestra.run.smithi027.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 4.4 MB/s | 264 kB 00:00 2024-04-25T01:12:35.459 INFO:teuthology.orchestra.run.smithi027.stdout:(37/152): ceph-mgr-diskprediction-local-19.0.0- 6.1 MB/s | 7.4 MB 00:01 2024-04-25T01:12:35.481 INFO:teuthology.orchestra.run.smithi153.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 421 kB/s | 77 kB 00:00 2024-04-25T01:12:35.485 INFO:teuthology.orchestra.run.smithi027.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 146 kB/s | 18 kB 00:00 2024-04-25T01:12:35.518 INFO:teuthology.orchestra.run.smithi027.stdout:(39/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 3.9 MB/s | 762 kB 00:00 2024-04-25T01:12:35.599 INFO:teuthology.orchestra.run.smithi153.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 361 kB/s | 42 kB 00:00 2024-04-25T01:12:35.610 INFO:teuthology.orchestra.run.smithi027.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 535 kB/s | 80 kB 00:00 2024-04-25T01:12:35.644 INFO:teuthology.orchestra.run.smithi027.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-04-25T01:12:35.694 INFO:teuthology.orchestra.run.smithi027.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-25T01:12:35.699 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-04-25T01:12:35.728 INFO:teuthology.orchestra.run.smithi027.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-04-25T01:12:35.753 INFO:teuthology.orchestra.run.smithi027.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-25T01:12:35.778 INFO:teuthology.orchestra.run.smithi027.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T01:12:35.804 INFO:teuthology.orchestra.run.smithi027.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T01:12:35.829 INFO:teuthology.orchestra.run.smithi027.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T01:12:35.846 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-04-25T01:12:35.846 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-04-25T01:12:35.862 INFO:teuthology.orchestra.run.smithi027.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-04-25T01:12:36.032 INFO:teuthology.orchestra.run.smithi153.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 621 kB/s | 269 kB 00:00 2024-04-25T01:12:36.071 INFO:teuthology.orchestra.run.smithi027.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 513 kB/s | 107 kB 00:00 2024-04-25T01:12:36.130 INFO:teuthology.orchestra.run.smithi027.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-04-25T01:12:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 257 kB/s | 21 kB 00:00 2024-04-25T01:12:36.263 INFO:teuthology.orchestra.run.smithi027.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 2.1 MB/s | 943 kB 00:00 2024-04-25T01:12:36.299 INFO:teuthology.orchestra.run.smithi153.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 595 kB/s | 159 kB 00:00 2024-04-25T01:12:36.381 INFO:teuthology.orchestra.run.smithi027.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 145 kB/s | 33 kB 00:00 2024-04-25T01:12:36.406 INFO:teuthology.orchestra.run.smithi027.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-25T01:12:36.440 INFO:teuthology.orchestra.run.smithi027.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-04-25T01:12:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 190 kB/s | 11 kB 00:00 2024-04-25T01:12:36.499 INFO:teuthology.orchestra.run.smithi027.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-25T01:12:36.523 INFO:teuthology.orchestra.run.smithi027.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 45 kB 00:00 2024-04-25T01:12:36.574 INFO:teuthology.orchestra.run.smithi027.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.6 MB/s | 3.0 MB 00:00 2024-04-25T01:12:36.618 INFO:teuthology.orchestra.run.smithi027.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.4 MB/s | 662 kB 00:00 2024-04-25T01:12:36.643 INFO:teuthology.orchestra.run.smithi027.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 369 kB/s | 25 kB 00:00 2024-04-25T01:12:36.668 INFO:teuthology.orchestra.run.smithi027.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 3.9 MB/s | 582 kB 00:00 2024-04-25T01:12:36.694 INFO:teuthology.orchestra.run.smithi027.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-04-25T01:12:36.719 INFO:teuthology.orchestra.run.smithi027.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-25T01:12:36.724 INFO:teuthology.orchestra.run.smithi153.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 656 kB/s | 279 kB 00:00 2024-04-25T01:12:36.744 INFO:teuthology.orchestra.run.smithi027.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-25T01:12:36.769 INFO:teuthology.orchestra.run.smithi027.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-04-25T01:12:36.825 INFO:teuthology.orchestra.run.smithi153.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 511 kB/s | 51 kB 00:00 2024-04-25T01:12:36.986 INFO:teuthology.orchestra.run.smithi027.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-25T01:12:37.053 INFO:teuthology.orchestra.run.smithi027.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-04-25T01:12:37.154 INFO:teuthology.orchestra.run.smithi027.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-25T01:12:37.179 INFO:teuthology.orchestra.run.smithi027.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-04-25T01:12:37.183 INFO:teuthology.orchestra.run.smithi153.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 150 kB/s | 54 kB 00:00 2024-04-25T01:12:37.212 INFO:teuthology.orchestra.run.smithi027.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 3.9 MB/s | 911 kB 00:00 2024-04-25T01:12:37.263 INFO:teuthology.orchestra.run.smithi027.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-25T01:12:37.296 INFO:teuthology.orchestra.run.smithi027.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-04-25T01:12:37.363 INFO:teuthology.orchestra.run.smithi027.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-25T01:12:37.430 INFO:teuthology.orchestra.run.smithi027.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-04-25T01:12:37.489 INFO:teuthology.orchestra.run.smithi027.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 439 kB/s | 26 kB 00:00 2024-04-25T01:12:37.558 INFO:teuthology.orchestra.run.smithi027.stdout:(77/152): postgresql-server-13.14-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-04-25T01:12:37.581 INFO:teuthology.orchestra.run.smithi027.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 208 kB/s | 19 kB 00:00 2024-04-25T01:12:37.614 INFO:teuthology.orchestra.run.smithi027.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-25T01:12:37.640 INFO:teuthology.orchestra.run.smithi027.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 593 kB/s | 35 kB 00:00 2024-04-25T01:12:37.676 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-04-25T01:12:37.677 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-04-25T01:12:37.698 INFO:teuthology.orchestra.run.smithi027.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-25T01:12:37.766 INFO:teuthology.orchestra.run.smithi027.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-25T01:12:37.816 INFO:teuthology.orchestra.run.smithi027.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T01:12:37.891 INFO:teuthology.orchestra.run.smithi027.stdout:(84/152): python3-babel-2.9.1-2.el9.noarch.rpm 9.5 MB/s | 6.0 MB 00:00 2024-04-25T01:12:37.916 INFO:teuthology.orchestra.run.smithi027.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 422 kB/s | 42 kB 00:00 2024-04-25T01:12:37.983 INFO:teuthology.orchestra.run.smithi027.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-04-25T01:12:38.008 INFO:teuthology.orchestra.run.smithi027.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-25T01:12:38.034 INFO:teuthology.orchestra.run.smithi027.stdout:(88/152): python3-protobuf-3.14.0-13.el9.noarch 1.8 MB/s | 269 kB 00:00 2024-04-25T01:12:38.059 INFO:teuthology.orchestra.run.smithi027.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-25T01:12:38.093 INFO:teuthology.orchestra.run.smithi027.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 605 kB/s | 51 kB 00:00 2024-04-25T01:12:38.118 INFO:teuthology.orchestra.run.smithi027.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 637 kB/s | 54 kB 00:00 2024-04-25T01:12:38.152 INFO:teuthology.orchestra.run.smithi027.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 714 kB/s | 42 kB 00:00 2024-04-25T01:12:38.177 INFO:teuthology.orchestra.run.smithi027.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 757 kB/s | 44 kB 00:00 2024-04-25T01:12:38.211 INFO:teuthology.orchestra.run.smithi027.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-25T01:12:38.236 INFO:teuthology.orchestra.run.smithi027.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 611 kB/s | 36 kB 00:00 2024-04-25T01:12:38.269 INFO:teuthology.orchestra.run.smithi027.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-04-25T01:12:38.295 INFO:teuthology.orchestra.run.smithi027.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-25T01:12:38.328 INFO:teuthology.orchestra.run.smithi027.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 986 kB/s | 58 kB 00:00 2024-04-25T01:12:38.354 INFO:teuthology.orchestra.run.smithi027.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-25T01:12:38.387 INFO:teuthology.orchestra.run.smithi027.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T01:12:38.413 INFO:teuthology.orchestra.run.smithi027.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 765 kB/s | 45 kB 00:00 2024-04-25T01:12:38.730 INFO:teuthology.orchestra.run.smithi027.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-04-25T01:12:38.746 INFO:teuthology.orchestra.run.smithi027.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 62 kB/s | 22 kB 00:00 2024-04-25T01:12:38.980 INFO:teuthology.orchestra.run.smithi027.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-04-25T01:12:39.005 INFO:teuthology.orchestra.run.smithi027.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-04-25T01:12:39.039 INFO:teuthology.orchestra.run.smithi027.stdout:(106/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.3 MB/s | 863 kB 00:00 2024-04-25T01:12:39.064 INFO:teuthology.orchestra.run.smithi027.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 232 kB/s | 19 kB 00:00 2024-04-25T01:12:39.123 INFO:teuthology.orchestra.run.smithi027.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-04-25T01:12:39.156 INFO:teuthology.orchestra.run.smithi027.stdout:(109/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-04-25T01:12:39.173 INFO:teuthology.orchestra.run.smithi027.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-25T01:12:39.290 INFO:teuthology.orchestra.run.smithi027.stdout:(111/152): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-25T01:12:39.332 INFO:teuthology.orchestra.run.smithi027.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-04-25T01:12:39.366 INFO:teuthology.orchestra.run.smithi027.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-04-25T01:12:39.394 INFO:teuthology.orchestra.run.smithi027.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 744 kB/s | 43 kB 00:00 2024-04-25T01:12:39.416 INFO:teuthology.orchestra.run.smithi027.stdout:(115/152): luarocks-3.9.2-1.el9.noarch.rpm 582 kB/s | 151 kB 00:00 2024-04-25T01:12:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 426 kB/s | 32 kB 00:00 2024-04-25T01:12:39.467 INFO:teuthology.orchestra.run.smithi027.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-04-25T01:12:39.509 INFO:teuthology.orchestra.run.smithi027.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 5.3 MB/s | 358 kB 00:00 2024-04-25T01:12:39.534 INFO:teuthology.orchestra.run.smithi027.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-04-25T01:12:39.550 INFO:teuthology.orchestra.run.smithi153.stdout:(90/152): python3-babel-2.9.1-2.el9.noarch.rpm 731 kB/s | 6.0 MB 00:08 2024-04-25T01:12:39.568 INFO:teuthology.orchestra.run.smithi027.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 600 kB/s | 35 kB 00:00 2024-04-25T01:12:39.593 INFO:teuthology.orchestra.run.smithi027.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-25T01:12:39.627 INFO:teuthology.orchestra.run.smithi027.stdout:(122/152): python3-cheroot-8.6.0-4.el9.noarch.r 817 kB/s | 172 kB 00:00 2024-04-25T01:12:39.650 INFO:teuthology.orchestra.run.smithi153.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 418 kB/s | 42 kB 00:00 2024-04-25T01:12:39.652 INFO:teuthology.orchestra.run.smithi027.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-25T01:12:39.702 INFO:teuthology.orchestra.run.smithi027.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 15 MB/s | 2.0 MB 00:00 2024-04-25T01:12:39.736 INFO:teuthology.orchestra.run.smithi027.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-25T01:12:39.750 INFO:teuthology.orchestra.run.smithi153.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 442 kB/s | 44 kB 00:00 2024-04-25T01:12:39.770 INFO:teuthology.orchestra.run.smithi027.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-04-25T01:12:39.795 INFO:teuthology.orchestra.run.smithi027.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-04-25T01:12:39.820 INFO:teuthology.orchestra.run.smithi027.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-04-25T01:12:39.845 INFO:teuthology.orchestra.run.smithi027.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-25T01:12:39.887 INFO:teuthology.orchestra.run.smithi027.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-25T01:12:39.913 INFO:teuthology.orchestra.run.smithi027.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 502 kB/s | 46 kB 00:00 2024-04-25T01:12:39.938 INFO:teuthology.orchestra.run.smithi027.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 855 kB/s | 79 kB 00:00 2024-04-25T01:12:39.963 INFO:teuthology.orchestra.run.smithi027.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-04-25T01:12:39.988 INFO:teuthology.orchestra.run.smithi027.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-04-25T01:12:40.022 INFO:teuthology.orchestra.run.smithi027.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-25T01:12:40.047 INFO:teuthology.orchestra.run.smithi027.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 483 kB/s | 28 kB 00:00 2024-04-25T01:12:40.072 INFO:teuthology.orchestra.run.smithi027.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-25T01:12:40.106 INFO:teuthology.orchestra.run.smithi027.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-25T01:12:40.117 INFO:teuthology.orchestra.run.smithi153.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 602 kB/s | 221 kB 00:00 2024-04-25T01:12:40.131 INFO:teuthology.orchestra.run.smithi027.stdout:(139/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-04-25T01:12:40.157 INFO:teuthology.orchestra.run.smithi027.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-04-25T01:12:40.182 INFO:teuthology.orchestra.run.smithi027.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-04-25T01:12:40.207 INFO:teuthology.orchestra.run.smithi027.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-25T01:12:40.233 INFO:teuthology.orchestra.run.smithi027.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-25T01:12:40.234 INFO:teuthology.orchestra.run.smithi153.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 308 kB/s | 36 kB 00:00 2024-04-25T01:12:40.249 INFO:teuthology.orchestra.run.smithi027.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-25T01:12:40.283 INFO:teuthology.orchestra.run.smithi027.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-04-25T01:12:40.308 INFO:teuthology.orchestra.run.smithi027.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-25T01:12:40.359 INFO:teuthology.orchestra.run.smithi153.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 379 kB/s | 47 kB 00:00 2024-04-25T01:12:40.383 INFO:teuthology.orchestra.run.smithi027.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T01:12:40.451 INFO:teuthology.orchestra.run.smithi027.stdout:(148/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.7 MB/s | 427 kB 00:00 2024-04-25T01:12:40.493 INFO:teuthology.orchestra.run.smithi153.stdout:(96/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 875 kB/s | 5.1 MB 00:05 2024-04-25T01:12:40.609 INFO:teuthology.orchestra.run.smithi027.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 8.5 MB/s | 1.3 MB 00:00 2024-04-25T01:12:40.627 INFO:teuthology.orchestra.run.smithi153.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 434 kB/s | 58 kB 00:00 2024-04-25T01:12:40.693 INFO:teuthology.orchestra.run.smithi027.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-25T01:12:40.760 INFO:teuthology.orchestra.run.smithi153.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 704 kB/s | 94 kB 00:00 2024-04-25T01:12:40.885 INFO:teuthology.orchestra.run.smithi153.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 509 kB/s | 64 kB 00:00 2024-04-25T01:12:40.927 INFO:teuthology.orchestra.run.smithi153.stdout:(100/152): socat-1.7.4.1-5.el9.x86_64.rpm 537 kB/s | 305 kB 00:00 2024-04-25T01:12:41.011 INFO:teuthology.orchestra.run.smithi153.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 358 kB/s | 45 kB 00:00 2024-04-25T01:12:41.018 INFO:teuthology.orchestra.run.smithi027.stdout:(151/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 4.9 MB/s | 3.1 MB 00:00 2024-04-25T01:12:41.076 INFO:teuthology.orchestra.run.smithi027.stdout:(152/152): librados2-19.0.0-2445.gd2c45d97.el9. 4.4 MB/s | 3.4 MB 00:00 2024-04-25T01:12:41.079 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:12:41.079 INFO:teuthology.orchestra.run.smithi027.stdout:Total 17 MB/s | 220 MB 00:12 2024-04-25T01:12:41.094 INFO:teuthology.orchestra.run.smithi153.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-04-25T01:12:41.228 INFO:teuthology.orchestra.run.smithi153.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 4.0 MB/s | 548 kB 00:00 2024-04-25T01:12:41.279 INFO:teuthology.orchestra.run.smithi153.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 6.0 MB/s | 309 kB 00:00 2024-04-25T01:12:41.320 INFO:teuthology.orchestra.run.smithi153.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 467 kB/s | 19 kB 00:00 2024-04-25T01:12:41.622 INFO:teuthology.orchestra.run.smithi153.stdout:(106/152): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-25T01:12:41.830 INFO:teuthology.orchestra.run.smithi153.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 125 kB/s | 26 kB 00:00 2024-04-25T01:12:41.863 INFO:teuthology.orchestra.run.smithi153.stdout:(108/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-04-25T01:12:41.894 INFO:teuthology.orchestra.run.smithi153.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-04-25T01:12:41.930 INFO:teuthology.orchestra.run.smithi153.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-04-25T01:12:41.958 INFO:teuthology.orchestra.run.smithi153.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-04-25T01:12:41.997 INFO:teuthology.orchestra.run.smithi153.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-04-25T01:12:42.031 INFO:teuthology.orchestra.run.smithi153.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-04-25T01:12:42.074 INFO:teuthology.orchestra.run.smithi153.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-25T01:12:42.099 INFO:teuthology.orchestra.run.smithi153.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 480 kB/s | 32 kB 00:00 2024-04-25T01:12:42.124 INFO:teuthology.orchestra.run.smithi153.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T01:12:42.150 INFO:teuthology.orchestra.run.smithi153.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-04-25T01:12:42.191 INFO:teuthology.orchestra.run.smithi153.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-04-25T01:12:42.200 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-25T01:12:42.216 INFO:teuthology.orchestra.run.smithi153.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-04-25T01:12:42.241 INFO:teuthology.orchestra.run.smithi153.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 701 kB/s | 35 kB 00:00 2024-04-25T01:12:42.267 INFO:teuthology.orchestra.run.smithi153.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-25T01:12:42.309 INFO:teuthology.orchestra.run.smithi153.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-04-25T01:12:42.342 INFO:teuthology.orchestra.run.smithi153.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 321 kB/s | 11 kB 00:00 2024-04-25T01:12:42.358 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-25T01:12:42.358 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-25T01:12:42.393 INFO:teuthology.orchestra.run.smithi153.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-25T01:12:42.418 INFO:teuthology.orchestra.run.smithi153.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-04-25T01:12:42.443 INFO:teuthology.orchestra.run.smithi153.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-25T01:12:42.468 INFO:teuthology.orchestra.run.smithi153.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-25T01:12:42.493 INFO:teuthology.orchestra.run.smithi153.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-04-25T01:12:42.527 INFO:teuthology.orchestra.run.smithi153.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-04-25T01:12:42.560 INFO:teuthology.orchestra.run.smithi153.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-04-25T01:12:42.594 INFO:teuthology.orchestra.run.smithi153.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-04-25T01:12:42.619 INFO:teuthology.orchestra.run.smithi153.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 987 kB/s | 58 kB 00:00 2024-04-25T01:12:42.644 INFO:teuthology.orchestra.run.smithi153.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-25T01:12:42.661 INFO:teuthology.orchestra.run.smithi153.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 392 kB/s | 16 kB 00:00 2024-04-25T01:12:42.687 INFO:teuthology.orchestra.run.smithi153.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-04-25T01:12:42.704 INFO:teuthology.orchestra.run.smithi153.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 674 kB/s | 28 kB 00:00 2024-04-25T01:12:42.721 INFO:teuthology.orchestra.run.smithi153.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 921 kB/s | 31 kB 00:00 2024-04-25T01:12:42.746 INFO:teuthology.orchestra.run.smithi153.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2024-04-25T01:12:42.763 INFO:teuthology.orchestra.run.smithi153.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-25T01:12:42.780 INFO:teuthology.orchestra.run.smithi153.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-04-25T01:12:42.797 INFO:teuthology.orchestra.run.smithi153.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-04-25T01:12:42.822 INFO:teuthology.orchestra.run.smithi153.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-04-25T01:12:42.847 INFO:teuthology.orchestra.run.smithi153.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-25T01:12:42.872 INFO:teuthology.orchestra.run.smithi153.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-04-25T01:12:42.898 INFO:teuthology.orchestra.run.smithi153.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 396 kB/s | 20 kB 00:00 2024-04-25T01:12:42.923 INFO:teuthology.orchestra.run.smithi153.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-25T01:12:43.040 INFO:teuthology.orchestra.run.smithi153.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-25T01:12:43.432 INFO:teuthology.orchestra.run.smithi153.stdout:(148/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 8.0 MB/s | 3.1 MB 00:00 2024-04-25T01:12:43.621 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-04-25T01:12:43.727 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:12:43.849 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:12:44.024 INFO:teuthology.orchestra.run.smithi153.stdout:(149/152): librados2-19.0.0-2445.gd2c45d97.el9. 3.1 MB/s | 3.4 MB 00:01 2024-04-25T01:12:44.030 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:12:44.078 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:12:44.112 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:12:44.156 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:12:44.217 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-25T01:12:44.218 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-25T01:12:44.299 INFO:teuthology.orchestra.run.smithi153.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 1.6 MB/s | 1.3 MB 00:00 2024-04-25T01:12:44.484 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:12:44.579 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:12:44.699 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:12:44.730 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:12:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:12:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:12:44.980 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:12:44.991 INFO:teuthology.orchestra.run.smithi153.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.5 MB/s | 2.4 MB 00:00 2024-04-25T01:12:45.031 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:12:45.068 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:12:45.171 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:12:45.194 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:12:45.239 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:12:45.261 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:12:45.282 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:12:45.478 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:12:45.519 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:12:45.627 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:12:45.721 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:12:45.752 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:12:45.786 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:12:45.851 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:12:45.894 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:12:45.937 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:12:45.969 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:12:45.998 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:12:46.065 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:12:46.128 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:12:46.159 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:12:46.194 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:12:46.270 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:12:46.305 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:12:46.348 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:12:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:12:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:12:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:12:46.573 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:12:46.714 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:12:46.769 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:12:47.286 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:12:47.372 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:12:47.402 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:12:47.444 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:12:47.482 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:12:47.524 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:12:47.684 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:12:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:12:47.799 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:12:47.879 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:12:47.919 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:12:47.955 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:12:48.002 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:12:48.051 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:12:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:12:48.374 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:12:48.509 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:12:48.554 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:12:48.584 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:12:48.633 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:12:48.736 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:12:48.775 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:12:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:12:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:12:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:12:48.956 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:12:49.076 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:12:49.106 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:12:49.134 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:12:49.177 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:12:49.224 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:12:49.264 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:12:49.294 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:12:49.328 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:12:49.361 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:12:49.405 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:12:49.438 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:12:49.524 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:12:49.576 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:12:49.616 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:12:49.663 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:12:49.761 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:12:49.809 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:12:50.271 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-25T01:12:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:12:50.411 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:12:50.473 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:12:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:12:50.528 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:12:50.549 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:12:50.572 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:12:50.593 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:12:50.623 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:12:50.720 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:12:50.773 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:12:50.859 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:12:50.900 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:12:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:12:51.049 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:12:51.086 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:12:51.126 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:12:51.182 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:12:51.208 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:12:51.209 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:12:51.301 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:12:51.330 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:12:51.487 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:12:51.536 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:12:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:12:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:12:51.846 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:12:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:12:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:12:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:12:51.962 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:12:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:12:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:12:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:12:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:12:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:12:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:12:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:12:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:12:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:12:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:12:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:12:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:12:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:12:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:12:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:12:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:12:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:12:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:12:53.062 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:12:53.090 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:12:53.162 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:12:53.172 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:12:53.213 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:12:53.267 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:12:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:12:53.295 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:12:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:12:53.445 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:12:53.500 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:12:53.641 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:12:53.691 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:12:53.783 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:12:53.813 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:12:53.849 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:12:54.063 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:12:54.123 INFO:teuthology.orchestra.run.smithi153.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 982 kB/s | 16 MB 00:16 2024-04-25T01:12:54.126 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:12:54.126 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.3 MB/s | 220 MB 00:41 2024-04-25T01:12:54.158 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:12:54.190 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:12:54.233 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:12:54.271 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:12:54.313 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:12:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:12:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:12:54.415 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:12:54.445 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:12:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:12:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:12:54.597 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:12:54.671 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:12:54.708 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:12:54.745 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:12:54.793 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:12:54.827 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:12:54.871 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:12:54.921 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:12:55.067 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:12:55.118 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:12:55.148 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:12:55.191 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:12:55.213 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-25T01:12:55.290 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:12:55.323 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:12:55.356 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-25T01:12:55.356 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-25T01:12:55.363 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:12:55.388 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:12:55.413 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:12:55.483 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:12:55.620 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:12:55.804 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:12:55.841 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:12:55.875 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:12:55.915 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:12:55.963 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:12:55.992 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:12:56.027 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:12:56.060 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:12:56.103 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:12:56.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:12:56.228 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:12:56.292 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:12:56.322 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:12:56.371 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:12:56.480 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:12:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:12:56.699 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:12:56.734 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:12:57.135 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-25T01:12:57.135 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-25T01:12:57.269 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:12:57.296 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:12:57.312 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:12:57.346 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:12:57.382 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:12:57.408 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:12:57.409 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:12:57.409 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:12:57.409 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:12:57.446 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:12:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:12:57.539 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:12:57.631 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:12:57.665 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:12:57.843 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:12:57.871 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:12:57.951 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:12:58.635 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:12:58.758 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:13:00.064 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:13:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:13:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:13:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:13:00.539 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:13:00.583 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:13:00.674 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:13:00.704 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:13:00.741 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:13:01.321 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:13:01.369 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:13:01.440 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:13:01.470 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:13:03.109 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-25T01:13:03.193 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:13:03.231 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:13:03.285 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:13:03.327 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:13:03.353 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:13:03.395 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:13:03.717 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:13:03.820 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:13:03.881 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:13:03.918 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:13:03.942 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:13:03.993 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:13:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:13:04.117 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:13:04.265 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:13:04.322 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:13:04.354 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:13:04.454 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:13:04.485 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:13:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:13:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:13:04.547 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:13:04.569 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:13:04.589 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:13:04.637 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:13:04.637 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:13:04.637 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:13:04.637 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:04.665 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:13:04.785 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:13:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:13:04.925 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:13:05.031 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:13:05.068 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:13:05.102 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:13:05.168 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:13:05.201 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:13:05.253 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:13:05.285 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:13:05.313 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:13:05.382 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:13:05.435 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:13:05.466 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:13:05.501 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:13:05.576 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:13:05.612 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:13:05.655 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:13:05.684 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:13:05.755 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:13:05.799 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:13:05.872 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:13:06.005 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:13:06.050 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:13:06.562 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:13:06.647 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:13:06.675 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:13:06.718 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:13:06.756 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:13:06.798 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:13:06.953 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:13:07.009 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:13:07.063 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:13:07.138 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:13:07.193 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:13:07.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:13:07.275 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:13:07.312 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:13:07.355 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:13:07.397 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:13:07.531 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:13:07.603 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:13:07.633 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:13:07.674 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:13:07.768 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:13:07.807 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:13:07.847 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:13:07.890 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:13:07.915 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:13:07.988 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:13:08.115 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:13:08.147 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:13:08.174 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:13:08.209 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:13:08.249 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:13:08.288 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:13:08.318 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:13:08.352 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:13:08.385 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:13:08.418 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:13:08.454 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:13:08.538 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:13:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:13:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:13:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:13:08.759 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:13:08.816 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /root 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2024-04-25T01:13:09.036 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:09.199 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:13:09.215 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:13:09.215 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:09.215 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:13:09.215 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:09.535 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:13:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:13:09.613 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:13:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:13:09.762 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:13:09.868 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:13:09.915 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:13:10.093 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:13:10.121 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:13:10.199 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:13:10.710 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:13:10.736 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:13:10.736 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:10.736 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:13:10.736 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:13:10.737 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:13:10.737 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:10.839 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:13:10.960 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:13:10.983 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:13:10.983 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:10.983 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:13:10.983 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:13:10.983 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:13:10.983 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:11.419 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:13:11.442 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:13:11.442 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:11.442 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:13:11.442 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:13:11.443 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:13:11.443 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:11.557 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:13:11.606 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:13:11.661 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:13:11.847 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:13:11.895 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:13:11.980 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:13:12.069 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:13:12.172 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:13:12.208 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:13:12.254 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:13:12.258 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:13:12.285 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:13:12.285 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:13:12.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:13:12.323 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:13:12.630 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:13:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:13:12.763 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:13:12.766 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:13:12.787 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:13:12.795 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:13:12.818 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:13:12.831 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:13:12.879 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:13:12.918 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:13:13.301 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:13:13.334 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:13:13.405 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:13:13.436 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:13:13.939 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:13:13.978 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:13:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:13:14.969 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:13:15.070 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:13:15.232 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:13:15.271 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:13:15.294 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:13:15.295 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:15.295 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:13:15.295 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:13:15.295 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:13:15.295 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:15.339 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:13:15.350 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:13:15.456 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:13:15.504 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:13:15.705 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:13:15.749 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:13:16.291 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:13:16.335 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:13:16.407 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:13:16.433 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:13:16.436 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:13:16.436 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:13:16.436 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:13:16.436 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:16.457 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:13:16.457 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:16.457 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:13:16.457 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:13:16.457 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:13:16.458 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:13:16.500 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:13:16.522 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:13:16.523 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:16.523 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:13:16.523 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:16.567 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:13:16.590 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:13:16.590 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:16.590 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:13:16.591 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:13:16.591 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:13:16.591 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-04-25T01:13:16.719 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:16.893 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:13:16.893 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:13:16.910 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:13:16.910 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:16.910 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:13:16.910 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:16.915 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:13:16.915 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:16.916 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:13:16.916 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:13:16.916 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:13:16.916 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:13:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:13:18.476 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:18.476 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:13:18.477 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:13:18.477 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:13:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:18.714 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:13:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:13:18.738 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:18.738 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:13:18.738 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:13:18.738 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:13:18.738 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:19.198 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:13:19.221 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:13:19.221 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:19.222 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:13:19.222 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:13:19.222 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:13:19.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:19.328 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:13:19.386 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:13:19.450 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:13:19.655 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:13:19.707 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:13:19.787 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:13:19.876 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:13:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:13:20.074 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:13:20.106 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:13:20.133 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:13:20.584 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:13:20.625 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:13:20.664 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:13:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:13:20.763 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:13:21.421 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:13:21.471 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:13:21.511 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:13:21.544 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:13:21.571 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:13:21.754 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:13:21.792 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:13:21.822 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:13:21.857 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:13:21.898 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:13:22.378 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:22.411 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:22.431 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:22.917 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:13:22.949 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:13:23.060 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:13:23.228 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:13:23.256 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:13:23.259 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:13:23.283 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:13:23.284 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:23.284 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:13:23.284 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:13:23.284 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:13:23.284 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:23.335 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:13:23.346 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:13:23.349 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:13:23.390 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:13:23.448 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:13:23.469 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:13:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:13:23.503 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:13:23.503 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:13:23.586 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:13:23.611 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:13:23.611 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:13:23.709 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:13:23.776 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:24.440 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:13:24.471 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:13:24.495 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:13:24.495 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:24.496 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:13:24.496 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:13:24.496 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:13:24.496 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:13:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:13:24.562 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:24.562 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:13:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:13:24.638 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:13:24.638 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:24.638 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:13:24.638 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:13:24.638 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:13:24.638 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:24.956 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:13:24.980 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:13:24.980 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:24.980 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:13:24.980 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:13:24.981 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:13:24.981 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:28.069 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:13:28.070 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-04-25T01:13:28.070 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-04-25T01:13:28.070 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-04-25T01:13:28.070 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-04-25T01:13:28.070 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-04-25T01:13:28.070 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-04-25T01:13:28.070 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-04-25T01:13:28.070 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:28.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:13:28.248 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:13:28.248 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:28.248 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:13:28.248 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:29.726 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:13:29.731 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:13:29.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:13:29.733 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:13:29.734 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:13:29.735 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:13:29.736 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:13:29.737 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:13:29.738 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:13:29.739 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:13:29.740 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:13:29.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:13:29.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:13:29.743 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:13:29.743 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:13:29.743 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:13:29.743 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:13:29.744 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:13:29.744 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:13:29.744 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:13:29.744 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:29.744 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:13:29.744 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:13:29.745 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:13:29.745 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:29.785 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:13:29.826 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:13:29.858 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:13:29.876 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:13:29.967 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:13:29.990 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:13:29.990 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:29.990 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:13:29.990 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:13:29.990 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:13:29.990 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:30.067 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:13:30.106 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:13:30.144 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:13:30.426 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:13:30.448 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:13:30.448 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:30.448 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:13:30.448 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:13:30.448 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:13:30.449 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:30.548 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:13:30.605 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:13:30.661 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:13:30.781 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:30.825 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:30.845 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:13:30.846 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:30.893 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:13:30.971 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:13:31.060 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:13:31.189 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:13:31.243 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:13:31.275 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:13:31.303 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:13:31.702 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:13:31.726 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:13:31.755 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:13:31.757 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:13:31.789 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:31.789 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:31.790 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2024-04-25T01:13:31.790 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.790 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.790 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:31.790 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:31.791 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-04-25T01:13:31.791 INFO:teuthology.orchestra.run.smithi139.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:13:31.791 INFO:teuthology.orchestra.run.smithi139.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:13:31.791 INFO:teuthology.orchestra.run.smithi139.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:13:31.791 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:13:31.791 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:13:31.793 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:13:31.794 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:13:31.795 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:13:31.796 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:13:31.797 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:13:31.798 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:13:31.799 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:13:31.799 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:13:31.799 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:13:31.799 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:13:31.799 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:13:31.799 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:13:31.799 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:13:31.800 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:13:31.800 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:13:31.800 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:13:31.800 INFO:teuthology.orchestra.run.smithi139.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:13:31.800 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:13:31.800 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:13:31.801 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:13:31.801 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:13:31.801 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:13:31.801 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:13:31.801 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.801 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.801 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:13:31.801 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:13:31.802 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:13:31.803 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:13:31.804 INFO:teuthology.orchestra.run.smithi139.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:13:31.804 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-04-25T01:13:31.804 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-04-25T01:13:31.820 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:13:31.846 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:13:31.849 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:13:31.889 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:13:31.974 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:13:32.009 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:13:32.009 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:13:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:13:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:13:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:13:32.160 DEBUG:teuthology.parallel:result is None 2024-04-25T01:13:32.207 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:13:32.277 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:13:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:13:33.870 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:13:33.901 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:13:34.002 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:13:34.164 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:13:34.186 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:13:34.209 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:13:34.209 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:34.209 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:13:34.209 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:13:34.209 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:13:34.209 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:34.246 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:13:34.256 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:13:34.355 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:13:34.419 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:13:35.319 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:13:35.348 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:13:35.370 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:13:35.370 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:35.370 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:13:35.370 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:13:35.370 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:13:35.370 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:35.415 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:13:35.437 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:13:35.437 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:35.437 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:13:35.437 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:35.482 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:13:35.505 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:13:35.505 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:35.505 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:13:35.505 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:13:35.505 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:13:35.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:35.808 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:13:35.830 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:13:35.831 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:13:35.831 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:13:35.831 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:13:35.831 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:13:35.831 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:38.136 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:38.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:13:38.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:13:38.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:13:38.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:13:38.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:13:38.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:13:38.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:13:38.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:13:38.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:13:38.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:13:38.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:13:38.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:13:38.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:13:38.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:13:38.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:13:38.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:13:38.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:13:38.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:13:38.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:13:38.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:13:38.162 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:13:38.162 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:13:38.162 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:13:38.162 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:13:38.162 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:13:38.162 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:13:38.162 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:13:38.163 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:13:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:13:38.165 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:13:38.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:13:38.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:13:38.168 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:13:38.168 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:13:38.168 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:13:38.168 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:13:38.168 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:13:38.168 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:13:38.168 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:13:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:13:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:13:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:13:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:13:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:13:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:13:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:13:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:13:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:13:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:13:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:13:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:13:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:13:38.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:13:38.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:13:38.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:13:38.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:13:38.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:13:38.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:13:38.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:13:38.171 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:13:38.172 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:13:38.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:13:38.174 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:13:38.174 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:13:38.174 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:13:38.174 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:13:38.174 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:13:38.174 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:13:38.174 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:13:38.175 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:13:38.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:13:38.177 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:13:40.361 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:13:40.368 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:40.368 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:40.368 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-04-25T01:13:40.368 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.368 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.368 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:40.368 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:40.368 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.370 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.371 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:13:40.372 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:13:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:13:40.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:13:40.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:13:40.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:13:40.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:40.378 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:13:40.379 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:13:40.379 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:13:40.379 INFO:teuthology.orchestra.run.smithi027.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:13:40.379 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:13:40.379 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:13:40.379 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:13:40.379 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:13:40.379 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-25T01:13:40.403 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:13:40.444 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:13:40.476 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:13:40.503 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:13:40.691 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:13:40.732 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:13:40.762 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:13:40.794 DEBUG:teuthology.parallel:result is None 2024-04-25T01:13:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:13:42.206 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:13:42.297 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:13:42.338 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:13:42.419 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:13:42.476 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:13:42.476 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:13:42.560 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:13:42.593 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:13:42.593 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:13:42.689 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:13:42.757 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:48.529 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:13:48.530 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:13:48.531 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:13:48.532 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:13:48.533 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:13:48.534 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:13:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:13:48.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:13:48.537 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:13:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:13:48.539 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:13:48.540 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:13:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:13:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:13:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:13:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:13:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:13:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:13:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:13:48.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:13:50.623 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:13:50.624 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:13:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:13:50.627 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:13:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:13:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:13:50.630 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:13:50.631 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:13:50.632 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:13:50.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:13:50.634 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:13:50.635 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:13:50.635 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:13:50.635 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:13:50.635 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:13:50.635 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:13:50.635 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:13:50.635 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-25T01:13:51.020 DEBUG:teuthology.parallel:result is None 2024-04-25T01:13:51.021 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:51.246 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:13:51.272 INFO:teuthology.orchestra.run.smithi027.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:13:51.273 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:13:51.273 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:13:51.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:51.491 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:13:51.517 INFO:teuthology.orchestra.run.smithi139.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:13:51.518 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:13:51.518 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:13:51.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:51.662 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:13:51.683 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:13:51.684 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:13:51.684 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:13:51.685 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T01:13:51.686 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:13:51.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:13:51.717 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:13:51.718 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:13:51.749 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:13:51.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:13:51.776 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T01:13:51.776 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:13:51.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:13:51.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:13:51.873 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:13:51.873 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:13:51.901 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:13:51.968 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:13:51.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:13:51.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:13:52.056 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T01:13:52.056 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:13:52.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:13:52.084 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:13:52.152 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:13:52.152 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:13:52.180 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:13:52.247 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:13:52.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:13:52.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:13:52.334 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T01:13:52.334 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:13:52.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:13:52.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:13:52.431 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:13:52.431 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:13:52.459 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:13:52.526 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:13:52.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:13:52.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:13:52.616 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T01:13:52.724 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T01:13:52.724 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:52.725 INFO:tasks.cephadm:Cluster fsid is 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:13:52.725 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T01:13:52.725 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.27', 'mon.b': '172.21.15.139', 'mon.c': '172.21.15.153'} 2024-04-25T01:13:52.725 INFO:tasks.cephadm:First mon is mon.a on smithi027 2024-04-25T01:13:52.725 INFO:tasks.cephadm:First mgr is x 2024-04-25T01:13:52.725 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T01:13:52.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-04-25T01:13:52.755 DEBUG:teuthology.orchestra.run.smithi139:> sudo hostname $(hostname -s) 2024-04-25T01:13:52.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-04-25T01:13:52.809 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T01:13:52.809 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:52.949 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54987/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 22:37:46.422067', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54981/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 17:00:00.872117', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-25T01:13:53.106 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 d2c45d977142e024a59b68a8f1dce8f0b9840551 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:53.108 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm 2024-04-25T01:13:53.108 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:53.489 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:13 /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:53.489 DEBUG:teuthology.orchestra.run.smithi139:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:54.017 INFO:teuthology.orchestra.run.smithi139.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:13 /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:54.017 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:54.527 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:13 /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:54.527 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:54.547 DEBUG:teuthology.orchestra.run.smithi139:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:54.566 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:13:54.597 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 on all hosts... 2024-04-25T01:13:54.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:13:54.600 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:13:54.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:13:56.179 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:13:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:13:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:14:29.047 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:14:29.047 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:14:29.047 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:14:29.048 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-04-25T01:14:29.048 INFO:teuthology.orchestra.run.smithi027.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:14:29.048 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:14:29.048 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:14:29.364 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T01:14:29.365 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:14:29.365 INFO:teuthology.orchestra.run.smithi139.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:14:29.365 INFO:teuthology.orchestra.run.smithi139.stdout: "repo_digests": [ 2024-04-25T01:14:29.365 INFO:teuthology.orchestra.run.smithi139.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:14:29.365 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-04-25T01:14:29.365 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T01:14:30.856 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T01:14:30.857 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:14:30.857 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:14:30.857 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-04-25T01:14:30.857 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:14:30.857 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-25T01:14:30.857 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T01:14:30.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-04-25T01:14:30.914 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph 2024-04-25T01:14:30.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-04-25T01:14:30.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-04-25T01:14:31.001 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /etc/ceph 2024-04-25T01:14:31.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-04-25T01:14:31.053 INFO:tasks.cephadm:Writing seed config... 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T01:14:31.055 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T01:14:31.056 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T01:14:31.057 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T01:14:31.058 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-25T01:14:31.059 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:14:31.059 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T01:14:31.077 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1469ec2e-02a1-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T01:14:31.077 DEBUG:teuthology.orchestra.run.smithi027:mon.a> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T01:14:31.120 DEBUG:teuthology.orchestra.run.smithi027:mgr.x> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T01:14:31.162 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T01:14:31.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 -v bootstrap --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:14:31.363 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:14:31.364 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551', '-v', 'bootstrap', '--fsid', '1469ec2e-02a1-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.27', '--skip-admin-label'] 2024-04-25T01:14:31.364 INFO:teuthology.orchestra.run.smithi027.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T01:14:31.364 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying podman|docker is present... 2024-04-25T01:14:31.391 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:14:31.392 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying lvm2 is present... 2024-04-25T01:14:31.392 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying time synchronization is in place... 2024-04-25T01:14:31.399 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:14:31.399 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:14:31.405 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:14:31.405 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T01:14:31.412 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-04-25T01:14:31.418 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-04-25T01:14:31.418 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:14:31.418 INFO:teuthology.orchestra.run.smithi027.stdout:Repeating the final host check... 2024-04-25T01:14:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:14:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T01:14:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl is present 2024-04-25T01:14:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:lvcreate is present 2024-04-25T01:14:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:14:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:14:31.466 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:14:31.466 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T01:14:31.472 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Host looks OK 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Cluster fsid: 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Acquiring lock 139797487649696 on /run/cephadm/1469ec2e-02a1-11ef-bc93-c7b262605968.lock 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Lock 139797487649696 acquired on /run/cephadm/1469ec2e-02a1-11ef-bc93-c7b262605968.lock 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 3300 ... 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 6789 ... 2024-04-25T01:14:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Base mon IP(s) is [172.21.15.27:3300, 172.21.15.27:6789], mon addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-04-25T01:14:31.483 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2024-04-25T01:14:31.483 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2024-04-25T01:14:31.485 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T01:14:31.486 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T01:14:31.486 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T01:14:31.488 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T01:14:31.488 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T01:14:31.488 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:14:31.488 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T01:14:31.488 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-25T01:14:31.488 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:14:31.489 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-04-25T01:14:31.489 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-04-25T01:14:31.489 INFO:teuthology.orchestra.run.smithi027.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T01:14:31.489 INFO:teuthology.orchestra.run.smithi027.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T01:14:31.490 INFO:teuthology.orchestra.run.smithi027.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:14:31.952 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:14:31.952 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:14:31.952 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T01:14:31.952 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:58a99251f41e8a467bafa0ec849428709568906ca732f10972304f37056be6c5 2024-04-25T01:14:31.952 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T01:14:31.952 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying config sha256:f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:14:31.952 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T01:14:32.929 INFO:teuthology.orchestra.run.smithi027.stdout:ceph: stdout ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:14:32.929 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph version: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:14:32.930 INFO:teuthology.orchestra.run.smithi027.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T01:14:33.810 INFO:teuthology.orchestra.run.smithi027.stdout:stat: stdout 167 167 2024-04-25T01:14:33.811 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial keys... 2024-04-25T01:14:34.665 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQB6rilmwUYRChAARmuGuBB8oXjwTmSm/2CZSw== 2024-04-25T01:14:35.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQB6rilmvNuIOxAA9BRn8G/WojEA8xpJ/a2NZg== 2024-04-25T01:14:36.541 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQB7rilmTbWgOhAAFdifPAHwJgMNjXF8ClsjOQ== 2024-04-25T01:14:36.553 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial monmap... 2024-04-25T01:14:37.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:14:37.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T01:14:37.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:37.440 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:14:37.440 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool for a [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:14:37.441 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-04-25T01:14:37.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: set fsid to 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:37.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:14:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:14:37.442 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mon... 2024-04-25T01:14:41.479 INFO:teuthology.orchestra.run.smithi027.stdout:create mon.a on 2024-04-25T01:14:42.062 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T01:14:42.488 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T01:14:43.266 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968.target. 2024-04-25T01:14:43.267 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968.target. 2024-04-25T01:14:44.002 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a 2024-04-25T01:14:44.002 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T01:14:44.330 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968.target.wants/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@.service. 2024-04-25T01:14:44.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:44 smithi027 podman[28811]: 2024-04-25 01:14:44.50952468 +0000 UTC m=+0.018424759 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:14:45.834 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:45 smithi027 podman[28811]: 2024-04-25 01:14:45.510020077 +0000 UTC m=+1.018920135 container create fdd80d5403d297ac517630b3a8875427867c53cc118711ac27f2b72209ed334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T01:14:45.972 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:14:45.972 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T01:14:45.979 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:14:45.979 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T01:14:45.979 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T01:14:45.979 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:14:45.980 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon to start... 2024-04-25T01:14:45.980 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon... 2024-04-25T01:14:46.184 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:45 smithi027 podman[28811]: 2024-04-25 01:14:45.914451745 +0000 UTC m=+1.423351803 container init fdd80d5403d297ac517630b3a8875427867c53cc118711ac27f2b72209ed334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T01:14:46.185 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:45 smithi027 podman[28811]: 2024-04-25 01:14:45.917779009 +0000 UTC m=+1.426679074 container start fdd80d5403d297ac517630b3a8875427867c53cc118711ac27f2b72209ed334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:14:46.185 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:45 smithi027 ceph-mon[28830]: mkfs 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:46.185 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:45 smithi027 ceph-mon[28830]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:14:46.185 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:45 smithi027 bash[28811]: fdd80d5403d297ac517630b3a8875427867c53cc118711ac27f2b72209ed334f 2024-04-25T01:14:46.185 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:45 smithi027 systemd[1]: Started Ceph mon.a for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:14:47.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:14:47.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: monmap epoch 1 2024-04-25T01:14:47.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:47.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: last_changed 2024-04-25T01:14:36.895688+0000 2024-04-25T01:14:47.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:14:47.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: min_mon_release 19 (squid) 2024-04-25T01:14:47.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: election_strategy: 1 2024-04-25T01:14:47.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:14:47.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: fsmap 2024-04-25T01:14:47.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:14:47.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: mgrmap e1: no daemons active 2024-04-25T01:14:47.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:46 smithi027 ceph-mon[28830]: from='client.? 172.21.15.27:0/2621491865' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout id: 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout services: 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.52187s) 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout data: 2024-04-25T01:14:47.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.401+0000 7f42a13f7700 1 Processor -- start 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.401+0000 7f42a13f7700 1 -- start start 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.401+0000 7f42a13f7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c10ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.401+0000 7f42a13f7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f429c07be50 con 0x7f429c108780 2024-04-25T01:14:47.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.402+0000 7f429affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c10ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.402+0000 7f429affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c10ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41726/0 (socket says 172.21.15.27:41726) 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.402+0000 7f429affd700 1 -- 172.21.15.27:0/248870837 learned_addr learned my addr 172.21.15.27:0/248870837 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.402+0000 7f429affd700 1 -- 172.21.15.27:0/248870837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c10b0b0 con 0x7f429c108780 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.402+0000 7f429affd700 1 --2- 172.21.15.27:0/248870837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c10ab70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f428c009e80 tx=0x7f428c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64f819b42507f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.402+0000 7f4299ffb700 1 -- 172.21.15.27:0/248870837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c017070 con 0x7f429c108780 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.402+0000 7f4299ffb700 1 -- 172.21.15.27:0/248870837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f428c003a20 con 0x7f429c108780 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.402+0000 7f4299ffb700 1 -- 172.21.15.27:0/248870837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c003d20 con 0x7f429c108780 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.403+0000 7f42a13f7700 1 -- 172.21.15.27:0/248870837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 msgr2=0x7f429c10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.403+0000 7f42a13f7700 1 --2- 172.21.15.27:0/248870837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c10ab70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f428c009e80 tx=0x7f428c00b330 comp rx=0 tx=0).stop 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.403+0000 7f42a13f7700 1 -- 172.21.15.27:0/248870837 shutdown_connections 2024-04-25T01:14:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.403+0000 7f42a13f7700 1 --2- 172.21.15.27:0/248870837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c10ab70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:47.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.403+0000 7f42a13f7700 1 -- 172.21.15.27:0/248870837 >> 172.21.15.27:0/248870837 conn(0x7f429c1021d0 msgr2=0x7f429c1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:47.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.403+0000 7f42a13f7700 1 -- 172.21.15.27:0/248870837 shutdown_connections 2024-04-25T01:14:47.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.403+0000 7f42a13f7700 1 -- 172.21.15.27:0/248870837 wait complete. 2024-04-25T01:14:47.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.404+0000 7f42a13f7700 1 Processor -- start 2024-04-25T01:14:47.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.404+0000 7f42a13f7700 1 -- start start 2024-04-25T01:14:47.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.404+0000 7f42a13f7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:47.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.404+0000 7f42a13f7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f429c10b580 con 0x7f429c108780 2024-04-25T01:14:47.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.404+0000 7f429affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:47.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.404+0000 7f429affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c19b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41738/0 (socket says 172.21.15.27:41738) 2024-04-25T01:14:47.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.404+0000 7f429affd700 1 -- 172.21.15.27:0/2621491865 learned_addr learned my addr 172.21.15.27:0/2621491865 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:47.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.404+0000 7f429affd700 1 -- 172.21.15.27:0/2621491865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c19b960 con 0x7f429c108780 2024-04-25T01:14:47.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.405+0000 7f429affd700 1 --2- 172.21.15.27:0/2621491865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c19b420 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f429c07b290 tx=0x7f428c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:47.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.405+0000 7f4293fff700 1 -- 172.21.15.27:0/2621491865 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c022020 con 0x7f429c108780 2024-04-25T01:14:47.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.405+0000 7f4293fff700 1 -- 172.21.15.27:0/2621491865 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f428c018040 con 0x7f429c108780 2024-04-25T01:14:47.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.405+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f429c19bbf0 con 0x7f429c108780 2024-04-25T01:14:47.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.405+0000 7f4293fff700 1 -- 172.21.15.27:0/2621491865 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c017070 con 0x7f429c108780 2024-04-25T01:14:47.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.405+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f429c19e8e0 con 0x7f429c108780 2024-04-25T01:14:47.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.405+0000 7f4293fff700 1 -- 172.21.15.27:0/2621491865 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f428c025020 con 0x7f429c108780 2024-04-25T01:14:47.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.405+0000 7f4293fff700 1 -- 172.21.15.27:0/2621491865 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f428c00f050 con 0x7f429c108780 2024-04-25T01:14:47.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.406+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f429c063890 con 0x7f429c108780 2024-04-25T01:14:47.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.408+0000 7f4293fff700 1 -- 172.21.15.27:0/2621491865 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f428c02c030 con 0x7f429c108780 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.481+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f429c02dd80 con 0x7f429c108780 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.482+0000 7f4293fff700 1 -- 172.21.15.27:0/2621491865 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f428c01e480 con 0x7f429c108780 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.482+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 msgr2=0x7f429c19b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.482+0000 7f42a13f7700 1 --2- 172.21.15.27:0/2621491865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c19b420 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f429c07b290 tx=0x7f428c004ad0 comp rx=0 tx=0).stop 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.482+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 shutdown_connections 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.482+0000 7f42a13f7700 1 --2- 172.21.15.27:0/2621491865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f429c108780 0x7f429c19b420 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.482+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 >> 172.21.15.27:0/2621491865 conn(0x7f429c1021d0 msgr2=0x7f429c1044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.482+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 shutdown_connections 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:46.482+0000 7f42a13f7700 1 -- 172.21.15.27:0/2621491865 wait complete. 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:mon is available 2024-04-25T01:14:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T01:14:48.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:48 smithi027 ceph-mon[28830]: from='client.? 172.21.15.27:0/3897657066' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:14:48.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:48 smithi027 ceph-mon[28830]: from='client.? 172.21.15.27:0/3897657066' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T01:14:48.605 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:14:48.606 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.075+0000 7f4c3ce51700 1 Processor -- start 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.075+0000 7f4c3ce51700 1 -- start start 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.075+0000 7f4c3ce51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3810ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.075+0000 7f4c3ce51700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3807be50 con 0x7f4c38108780 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.076+0000 7f4c3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3810ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.076+0000 7f4c3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3810ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54040/0 (socket says 172.21.15.27:54040) 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.076+0000 7f4c3659c700 1 -- 172.21.15.27:0/7890014 learned_addr learned my addr 172.21.15.27:0/7890014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:48.607 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.076+0000 7f4c3659c700 1 -- 172.21.15.27:0/7890014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3810b0b0 con 0x7f4c38108780 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.076+0000 7f4c3659c700 1 --2- 172.21.15.27:0/7890014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3810ab70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4c280098a0 tx=0x7f4c28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d87120c2747fd0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.076+0000 7f4c3559a700 1 -- 172.21.15.27:0/7890014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c28004160 con 0x7f4c38108780 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.076+0000 7f4c3559a700 1 -- 172.21.15.27:0/7890014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c28004300 con 0x7f4c38108780 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.076+0000 7f4c3559a700 1 -- 172.21.15.27:0/7890014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c280036a0 con 0x7f4c38108780 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.077+0000 7f4c3ce51700 1 -- 172.21.15.27:0/7890014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 msgr2=0x7f4c3810ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.077+0000 7f4c3ce51700 1 --2- 172.21.15.27:0/7890014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3810ab70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4c280098a0 tx=0x7f4c28009bb0 comp rx=0 tx=0).stop 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.077+0000 7f4c3ce51700 1 -- 172.21.15.27:0/7890014 shutdown_connections 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.077+0000 7f4c3ce51700 1 --2- 172.21.15.27:0/7890014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3810ab70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.077+0000 7f4c3ce51700 1 -- 172.21.15.27:0/7890014 >> 172.21.15.27:0/7890014 conn(0x7f4c381021d0 msgr2=0x7f4c381045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.077+0000 7f4c3ce51700 1 -- 172.21.15.27:0/7890014 shutdown_connections 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.077+0000 7f4c3ce51700 1 -- 172.21.15.27:0/7890014 wait complete. 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.078+0000 7f4c3ce51700 1 Processor -- start 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.078+0000 7f4c3ce51700 1 -- start start 2024-04-25T01:14:48.608 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.078+0000 7f4c3ce51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.078+0000 7f4c3ce51700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3810b580 con 0x7f4c38108780 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.078+0000 7f4c3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.078+0000 7f4c3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3819b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54052/0 (socket says 172.21.15.27:54052) 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.078+0000 7f4c3659c700 1 -- 172.21.15.27:0/3897657066 learned_addr learned my addr 172.21.15.27:0/3897657066 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c3659c700 1 -- 172.21.15.27:0/3897657066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3819b8d0 con 0x7f4c38108780 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c3659c700 1 --2- 172.21.15.27:0/3897657066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3819b390 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4c28000c00 tx=0x7f4c28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c277fe700 1 -- 172.21.15.27:0/3897657066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c28003db0 con 0x7f4c38108780 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c277fe700 1 -- 172.21.15.27:0/3897657066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c28011630 con 0x7f4c38108780 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c3819bb60 con 0x7f4c38108780 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c277fe700 1 -- 172.21.15.27:0/3897657066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c28011950 con 0x7f4c38108780 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c3819e850 con 0x7f4c38108780 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c277fe700 1 -- 172.21.15.27:0/3897657066 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4c28011af0 con 0x7f4c38108780 2024-04-25T01:14:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.079+0000 7f4c277fe700 1 -- 172.21.15.27:0/3897657066 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4c2801c450 con 0x7f4c38108780 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.080+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c38063890 con 0x7f4c38108780 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.082+0000 7f4c277fe700 1 -- 172.21.15.27:0/3897657066 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4c28032020 con 0x7f4c38108780 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.153+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4c3802dcb0 con 0x7f4c38108780 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.156+0000 7f4c277fe700 1 -- 172.21.15.27:0/3897657066 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f4c28004260 con 0x7f4c38108780 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.156+0000 7f4c277fe700 1 -- 172.21.15.27:0/3897657066 <== mon.0 v2:172.21.15.27:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4c2801cb60 con 0x7f4c38108780 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.157+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 msgr2=0x7f4c3819b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.157+0000 7f4c3ce51700 1 --2- 172.21.15.27:0/3897657066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3819b390 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4c28000c00 tx=0x7f4c28004750 comp rx=0 tx=0).stop 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.157+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 shutdown_connections 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.157+0000 7f4c3ce51700 1 --2- 172.21.15.27:0/3897657066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c38108780 0x7f4c3819b390 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.157+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 >> 172.21.15.27:0/3897657066 conn(0x7f4c381021d0 msgr2=0x7f4c38103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.157+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 shutdown_connections 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:48.157+0000 7f4c3ce51700 1 -- 172.21.15.27:0/3897657066 wait complete. 2024-04-25T01:14:48.610 INFO:teuthology.orchestra.run.smithi027.stdout:Generating new minimal ceph.conf... 2024-04-25T01:14:49.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:49 smithi027 ceph-mon[28830]: from='client.? 172.21.15.27:0/910689112' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:14:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.029+0000 7fc439f81700 1 Processor -- start 2024-04-25T01:14:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.029+0000 7fc439f81700 1 -- start start 2024-04-25T01:14:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.029+0000 7fc439f81700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc434107030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.029+0000 7fc439f81700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc434107600 con 0x7fc434106c10 2024-04-25T01:14:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.030+0000 7fc4337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc434107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:49.644 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.030+0000 7fc4337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc434107030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54068/0 (socket says 172.21.15.27:54068) 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.030+0000 7fc4337fe700 1 -- 172.21.15.27:0/3828174054 learned_addr learned my addr 172.21.15.27:0/3828174054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.030+0000 7fc4337fe700 1 -- 172.21.15.27:0/3828174054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc434107e10 con 0x7fc434106c10 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.030+0000 7fc4337fe700 1 --2- 172.21.15.27:0/3828174054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc434107030 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc4240098a0 tx=0x7fc424009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe58901dd75cd9b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.030+0000 7fc4327fc700 1 -- 172.21.15.27:0/3828174054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424004160 con 0x7fc434106c10 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.030+0000 7fc4327fc700 1 -- 172.21.15.27:0/3828174054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc424004300 con 0x7fc434106c10 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.030+0000 7fc4327fc700 1 -- 172.21.15.27:0/3828174054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424011b50 con 0x7fc434106c10 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.031+0000 7fc439f81700 1 -- 172.21.15.27:0/3828174054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 msgr2=0x7fc434107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.031+0000 7fc439f81700 1 --2- 172.21.15.27:0/3828174054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc434107030 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc4240098a0 tx=0x7fc424009bb0 comp rx=0 tx=0).stop 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.031+0000 7fc439f81700 1 -- 172.21.15.27:0/3828174054 shutdown_connections 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.031+0000 7fc439f81700 1 --2- 172.21.15.27:0/3828174054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc434107030 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.031+0000 7fc439f81700 1 -- 172.21.15.27:0/3828174054 >> 172.21.15.27:0/3828174054 conn(0x7fc4341021d0 msgr2=0x7fc4341045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.031+0000 7fc439f81700 1 -- 172.21.15.27:0/3828174054 shutdown_connections 2024-04-25T01:14:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.031+0000 7fc439f81700 1 -- 172.21.15.27:0/3828174054 wait complete. 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc439f81700 1 Processor -- start 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc439f81700 1 -- start start 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc439f81700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc43419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc439f81700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc43410b1c0 con 0x7fc434106c10 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc4337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc43419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc4337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc43419aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54084/0 (socket says 172.21.15.27:54084) 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc4337fe700 1 -- 172.21.15.27:0/910689112 learned_addr learned my addr 172.21.15.27:0/910689112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc4337fe700 1 -- 172.21.15.27:0/910689112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43419b0e0 con 0x7fc434106c10 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.032+0000 7fc4337fe700 1 --2- 172.21.15.27:0/910689112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc43419aba0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc424000c00 tx=0x7fc424011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.033+0000 7fc430ff9700 1 -- 172.21.15.27:0/910689112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424003b20 con 0x7fc434106c10 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.033+0000 7fc430ff9700 1 -- 172.21.15.27:0/910689112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc424003cc0 con 0x7fc434106c10 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.033+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43419b370 con 0x7fc434106c10 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.033+0000 7fc430ff9700 1 -- 172.21.15.27:0/910689112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424024910 con 0x7fc434106c10 2024-04-25T01:14:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.033+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43419e060 con 0x7fc434106c10 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.033+0000 7fc430ff9700 1 -- 172.21.15.27:0/910689112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc424022020 con 0x7fc434106c10 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.033+0000 7fc430ff9700 1 -- 172.21.15.27:0/910689112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc424011e00 con 0x7fc434106c10 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.034+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc434063890 con 0x7fc434106c10 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.036+0000 7fc430ff9700 1 -- 172.21.15.27:0/910689112 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc42402ddf0 con 0x7fc434106c10 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.106+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc43419e230 con 0x7fc434106c10 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.107+0000 7fc430ff9700 1 -- 172.21.15.27:0/910689112 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fc424032030 con 0x7fc434106c10 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.107+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 msgr2=0x7fc43419aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.107+0000 7fc439f81700 1 --2- 172.21.15.27:0/910689112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc43419aba0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc424000c00 tx=0x7fc424011c10 comp rx=0 tx=0).stop 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.107+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 shutdown_connections 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.107+0000 7fc439f81700 1 --2- 172.21.15.27:0/910689112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106c10 0x7fc43419aba0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.107+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 >> 172.21.15.27:0/910689112 conn(0x7fc4341021d0 msgr2=0x7fc434102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.107+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 shutdown_connections 2024-04-25T01:14:49.647 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:49.108+0000 7fc439f81700 1 -- 172.21.15.27:0/910689112 wait complete. 2024-04-25T01:14:49.648 INFO:teuthology.orchestra.run.smithi027.stdout:Restarting the monitor... 2024-04-25T01:14:49.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:49 smithi027 systemd[1]: Stopping Ceph mon.a for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:14:50.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:50 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a[28826]: 2024-04-25T01:14:50.440+0000 7f3321c8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:14:50.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:50 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a[28826]: 2024-04-25T01:14:50.440+0000 7f3321c8f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T01:14:50.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:50 smithi027 podman[29074]: 2024-04-25 01:14:50.593807443 +0000 UTC m=+0.768411656 container died fdd80d5403d297ac517630b3a8875427867c53cc118711ac27f2b72209ed334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:14:52.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:51 smithi027 podman[29074]: 2024-04-25 01:14:51.793817791 +0000 UTC m=+1.968422031 container cleanup fdd80d5403d297ac517630b3a8875427867c53cc118711ac27f2b72209ed334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T01:14:52.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:51 smithi027 bash[29074]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a 2024-04-25T01:14:54.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:53 smithi027 podman[29088]: 2024-04-25 01:14:53.757817915 +0000 UTC m=+3.163439388 container remove fdd80d5403d297ac517630b3a8875427867c53cc118711ac27f2b72209ed334f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T01:14:54.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:53 smithi027 systemd[1]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T01:14:54.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:53 smithi027 systemd[1]: Stopped Ceph mon.a for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:14:54.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:53 smithi027 systemd[1]: Starting Ceph mon.a for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:14:54.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 podman[29177]: 2024-04-25 01:14:54.090066543 +0000 UTC m=+0.018848754 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 podman[29177]: 2024-04-25 01:14:54.572860256 +0000 UTC m=+0.501642490 container create be8d93f92e99d480d18cc736384ac0575dfbca7d47fc4efe4bd66caeb6398132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5) 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 podman[29177]: 2024-04-25 01:14:54.77498068 +0000 UTC m=+0.703762872 container init be8d93f92e99d480d18cc736384ac0575dfbca7d47fc4efe4bd66caeb6398132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, ceph=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 podman[29177]: 2024-04-25 01:14:54.778203192 +0000 UTC m=+0.706985385 container start be8d93f92e99d480d18cc736384ac0575dfbca7d47fc4efe4bd66caeb6398132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: pidfile_write: ignore empty --pid-file 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: load: jerasure load: lrc 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Git sha 0 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:14:54.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: DB SUMMARY 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: DB Session ID: BAW37M6Z0W6H34A1FJT1 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105312 ; 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.env: 0x557c3e735c60 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:14:54.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.info_log: 0x557c40ef9aa0 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.statistics: (nil) 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.use_fsync: 0 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:14:54.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.db_log_dir: 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.wal_dir: 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.write_buffer_manager: 0x557c40ecc1e0 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:14:54.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:14:54.848 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:14:54.848 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:14:54.848 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:14:54.848 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:14:54.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.unordered_write: 0 2024-04-25T01:14:54.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:14:54.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:14:54.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:14:54.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:14:54.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.row_cache: None 2024-04-25T01:14:54.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.wal_filter: None 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.wal_compression: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:14:54.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:14:54.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_open_files: -1 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:14:54.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Compression algorithms supported: 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: kZSTD supported: 0 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:14:54.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.merge_operator: 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_filter: None 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557c40ef9ba0) 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_type: 0 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_index_type: 0 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_shortening: 1 2024-04-25T01:14:54.854 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: checksum: 4 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: no_block_cache: 0 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache: 0x557c40efd090 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_options: 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: capacity : 536870912 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_shard_bits : 4 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: strict_capacity_limit : 0 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_compressed: (nil) 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: persistent_cache: (nil) 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size: 4096 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size_deviation: 10 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_restart_interval: 16 2024-04-25T01:14:54.855 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_block_restart_interval: 1 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: metadata_block_size: 4096 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: partition_filters: 0 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: use_delta_encoding: 1 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: filter_policy: bloomfilter 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: whole_key_filtering: 1 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: verify_compression: 0 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: format_version: 5 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_index_compression: 1 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_align: 0 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: max_auto_readahead_size: 262144 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: prepopulate_block_cache: 0 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:14:54.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression: NoCompression 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.num_levels: 7 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:14:54.857 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:14:54.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:14:54.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:14:54.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:14:54.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.ttl: 2592000 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:14:54.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2c67e3fc-b270-4e2e-a96d-2ad57e512256 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007694810360, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007694811528, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100355, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98275, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14306, "raw_average_key_size": 49, "raw_value_size": 90521, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714007694, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2c67e3fc-b270-4e2e-a96d-2ad57e512256", "db_session_id": "BAW37M6Z0W6H34A1FJT1", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007694811623, "job": 1, "event": "recovery_finished"} 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557c40fee000 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: DB pointer 0x557c40fd8000 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** DB Stats ** 2024-04-25T01:14:54.863 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: L0 2/0 99.84 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Sum 2/0 99.84 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T01:14:54.864 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 129.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative compaction: 0.00 GB write, 28.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:14:54.865 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval compaction: 0.00 GB write, 28.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache BinnedLRUCache@0x557c40efd090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: starting mon.a rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a@-1(???) e1 preinit fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a@-1(???).mds e1 new map 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a@-1(???).mds e1 print_map 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout: e1 2024-04-25T01:14:54.866 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout: legacy client fscid: -1 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout: No filesystems configured 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 bash[29177]: be8d93f92e99d480d18cc736384ac0575dfbca7d47fc4efe4bd66caeb6398132 2024-04-25T01:14:54.867 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 systemd[1]: Started Ceph mon.a for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:14:54.867 INFO:teuthology.orchestra.run.smithi027.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: monmap epoch 1 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: last_changed 2024-04-25T01:14:36.895688+0000 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: min_mon_release 19 (squid) 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: election_strategy: 1 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:14:55.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: fsmap 2024-04-25T01:14:55.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:14:55.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:54 smithi027 ceph-mon[29197]: mgrmap e1: no daemons active 2024-04-25T01:14:56.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:56 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/465954691' entity='client.admin' 2024-04-25T01:14:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.274+0000 7fa8db58c700 1 Processor -- start 2024-04-25T01:14:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.274+0000 7fa8db58c700 1 -- start start 2024-04-25T01:14:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.274+0000 7fa8db58c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d410ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.274+0000 7fa8db58c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d407be50 con 0x7fa8d4108780 2024-04-25T01:14:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.275+0000 7fa8d9328700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d410ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.275+0000 7fa8d9328700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d410ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54096/0 (socket says 172.21.15.27:54096) 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.275+0000 7fa8d9328700 1 -- 172.21.15.27:0/6738461 learned_addr learned my addr 172.21.15.27:0/6738461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.275+0000 7fa8d9328700 1 -- 172.21.15.27:0/6738461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d410b0b0 con 0x7fa8d4108780 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.275+0000 7fa8d9328700 1 --2- 172.21.15.27:0/6738461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d410ab70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00098a0 tx=0x7fa8d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f06a70edf6b6600 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.275+0000 7fa8cbfff700 1 -- 172.21.15.27:0/6738461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8d0004160 con 0x7fa8d4108780 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8cbfff700 1 -- 172.21.15.27:0/6738461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa8d0004300 con 0x7fa8d4108780 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8cbfff700 1 -- 172.21.15.27:0/6738461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8d0011b50 con 0x7fa8d4108780 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8db58c700 1 -- 172.21.15.27:0/6738461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 msgr2=0x7fa8d410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8db58c700 1 --2- 172.21.15.27:0/6738461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d410ab70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00098a0 tx=0x7fa8d0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8db58c700 1 -- 172.21.15.27:0/6738461 shutdown_connections 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8db58c700 1 --2- 172.21.15.27:0/6738461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d410ab70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8db58c700 1 -- 172.21.15.27:0/6738461 >> 172.21.15.27:0/6738461 conn(0x7fa8d41021d0 msgr2=0x7fa8d41045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8db58c700 1 -- 172.21.15.27:0/6738461 shutdown_connections 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.276+0000 7fa8db58c700 1 -- 172.21.15.27:0/6738461 wait complete. 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.277+0000 7fa8db58c700 1 Processor -- start 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.277+0000 7fa8db58c700 1 -- start start 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.277+0000 7fa8db58c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.277+0000 7fa8db58c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d410c0e0 con 0x7fa8d4108780 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.277+0000 7fa8d9328700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.277+0000 7fa8d9328700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d419bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54112/0 (socket says 172.21.15.27:54112) 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.277+0000 7fa8d9328700 1 -- 172.21.15.27:0/465954691 learned_addr learned my addr 172.21.15.27:0/465954691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8d9328700 1 -- 172.21.15.27:0/465954691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d419c1c0 con 0x7fa8d4108780 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8d9328700 1 --2- 172.21.15.27:0/465954691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d419bc80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0011670 tx=0x7fa8d0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8ca7fc700 1 -- 172.21.15.27:0/465954691 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8d00039c0 con 0x7fa8d4108780 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8d419c450 con 0x7fa8d4108780 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8ca7fc700 1 -- 172.21.15.27:0/465954691 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa8d0011e50 con 0x7fa8d4108780 2024-04-25T01:14:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8ca7fc700 1 -- 172.21.15.27:0/465954691 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8d001a5c0 con 0x7fa8d4108780 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8d419f140 con 0x7fa8d4108780 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8ca7fc700 1 -- 172.21.15.27:0/465954691 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa8d001ac50 con 0x7fa8d4108780 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.278+0000 7fa8ca7fc700 1 -- 172.21.15.27:0/465954691 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa8d0019700 con 0x7fa8d4108780 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.279+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8d4063890 con 0x7fa8d4108780 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.281+0000 7fa8ca7fc700 1 -- 172.21.15.27:0/465954691 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa8d0003b60 con 0x7fa8d4108780 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.352+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fa8d419f310 con 0x7fa8d4108780 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.354+0000 7fa8ca7fc700 1 -- 172.21.15.27:0/465954691 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fa8d0019900 con 0x7fa8d4108780 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.354+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 msgr2=0x7fa8d419bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.354+0000 7fa8db58c700 1 --2- 172.21.15.27:0/465954691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d419bc80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0011670 tx=0x7fa8d0011b60 comp rx=0 tx=0).stop 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.354+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 shutdown_connections 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.354+0000 7fa8db58c700 1 --2- 172.21.15.27:0/465954691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4108780 0x7fa8d419bc80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.354+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 >> 172.21.15.27:0/465954691 conn(0x7fa8d41021d0 msgr2=0x7fa8d4102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.355+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 shutdown_connections 2024-04-25T01:14:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:55.355+0000 7fa8db58c700 1 -- 172.21.15.27:0/465954691 wait complete. 2024-04-25T01:14:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T01:14:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:14:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mgr... 2024-04-25T01:14:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T01:14:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T01:14:57.140 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.x 2024-04-25T01:14:57.140 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T01:14:57.302 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968.target.wants/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@.service. 2024-04-25T01:14:57.451 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:57 smithi027 systemd[1]: Starting Ceph mgr.x for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:14:57.845 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:57 smithi027 podman[29412]: 2024-04-25 01:14:57.462480666 +0000 UTC m=+0.018372861 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:14:57.845 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:57 smithi027 podman[29412]: 2024-04-25 01:14:57.566987008 +0000 UTC m=+0.122879183 container create 63362b3c482ce2f56f3b187a2c898c0d40173e6db67380bf294e0a24015b982b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T01:14:57.846 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:57 smithi027 podman[29412]: 2024-04-25 01:14:57.777655804 +0000 UTC m=+0.333547983 container init 63362b3c482ce2f56f3b187a2c898c0d40173e6db67380bf294e0a24015b982b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:14:57.846 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:57 smithi027 podman[29412]: 2024-04-25 01:14:57.780112081 +0000 UTC m=+0.336004257 container start 63362b3c482ce2f56f3b187a2c898c0d40173e6db67380bf294e0a24015b982b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:14:57.846 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:57 smithi027 bash[29412]: 63362b3c482ce2f56f3b187a2c898c0d40173e6db67380bf294e0a24015b982b 2024-04-25T01:14:57.862 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:14:57.862 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T01:14:57.871 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:14:57.871 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T01:14:57.871 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T01:14:57.871 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:14:57.881 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:14:57.881 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T01:14:57.890 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:14:57.890 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T01:14:57.890 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T01:14:57.890 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T01:14:57.890 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr to start... 2024-04-25T01:14:57.890 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr... 2024-04-25T01:14:58.125 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:57 smithi027 systemd[1]: Started Ceph mgr.x for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:14:58.126 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:57 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:14:57.944+0000 7f55a37a0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:14:58.126 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:14:58.139+0000 7f55a37a0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:14:58.375 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:14:58.248+0000 7f55a37a0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:14:58.658 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:14:58 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3773330328' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:14:58.659 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:14:58.584+0000 7f55a37a0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:14:58.659 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:14:58.672+0000 7f55a37a0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:14:58.990 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:14:58.991 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:14:58.991 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:14:58.991 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:14:58.991 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:14:58.991 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:14:58.991 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:14:58.991 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-25T01:14:58.992 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:14:58.993 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:14:59.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:14:59.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:14:59.002 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:14:59.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:14:45.961872+0000", 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.382+0000 7f8bac7c4700 1 Processor -- start 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.382+0000 7f8bac7c4700 1 -- start start 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.382+0000 7f8bac7c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba4108140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.382+0000 7f8bac7c4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba4108710 con 0x7f8ba4107d20 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.383+0000 7f8baa560700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba4108140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:59.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.383+0000 7f8baa560700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba4108140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57142/0 (socket says 172.21.15.27:57142) 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.383+0000 7f8baa560700 1 -- 172.21.15.27:0/2669529515 learned_addr learned my addr 172.21.15.27:0/2669529515 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.383+0000 7f8baa560700 1 -- 172.21.15.27:0/2669529515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba4108ed0 con 0x7f8ba4107d20 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.383+0000 7f8baa560700 1 --2- 172.21.15.27:0/2669529515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba4108140 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00098a0 tx=0x7f8ba0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f26f4e0709c0c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.383+0000 7f8ba955e700 1 -- 172.21.15.27:0/2669529515 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ba0004160 con 0x7f8ba4107d20 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.383+0000 7f8ba955e700 1 -- 172.21.15.27:0/2669529515 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8ba0004300 con 0x7f8ba4107d20 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.383+0000 7f8ba955e700 1 -- 172.21.15.27:0/2669529515 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ba0011b50 con 0x7f8ba4107d20 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.384+0000 7f8bac7c4700 1 -- 172.21.15.27:0/2669529515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 msgr2=0x7f8ba4108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.384+0000 7f8bac7c4700 1 --2- 172.21.15.27:0/2669529515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba4108140 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00098a0 tx=0x7f8ba0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.384+0000 7f8bac7c4700 1 -- 172.21.15.27:0/2669529515 shutdown_connections 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.384+0000 7f8bac7c4700 1 --2- 172.21.15.27:0/2669529515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba4108140 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.384+0000 7f8bac7c4700 1 -- 172.21.15.27:0/2669529515 >> 172.21.15.27:0/2669529515 conn(0x7f8ba4103320 msgr2=0x7f8ba4105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.384+0000 7f8bac7c4700 1 -- 172.21.15.27:0/2669529515 shutdown_connections 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.384+0000 7f8bac7c4700 1 -- 172.21.15.27:0/2669529515 wait complete. 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.385+0000 7f8bac7c4700 1 Processor -- start 2024-04-25T01:14:59.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.385+0000 7f8bac7c4700 1 -- start start 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.385+0000 7f8bac7c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.385+0000 7f8bac7c4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba410c280 con 0x7f8ba4107d20 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.385+0000 7f8baa560700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.385+0000 7f8baa560700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba419bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57144/0 (socket says 172.21.15.27:57144) 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.385+0000 7f8baa560700 1 -- 172.21.15.27:0/3773330328 learned_addr learned my addr 172.21.15.27:0/3773330328 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8baa560700 1 -- 172.21.15.27:0/3773330328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba419c1c0 con 0x7f8ba4107d20 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8baa560700 1 --2- 172.21.15.27:0/3773330328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba419bc80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0011670 tx=0x7f8ba0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8b9b7fe700 1 -- 172.21.15.27:0/3773330328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ba00039c0 con 0x7f8ba4107d20 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8b9b7fe700 1 -- 172.21.15.27:0/3773330328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8ba0011e50 con 0x7f8ba4107d20 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba419c450 con 0x7f8ba4107d20 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8b9b7fe700 1 -- 172.21.15.27:0/3773330328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ba001a5c0 con 0x7f8ba4107d20 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba419f140 con 0x7f8ba4107d20 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8b9b7fe700 1 -- 172.21.15.27:0/3773330328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8ba001ac00 con 0x7f8ba4107d20 2024-04-25T01:14:59.006 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.386+0000 7f8b9b7fe700 1 -- 172.21.15.27:0/3773330328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8ba0019700 con 0x7f8ba4107d20 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.387+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba4063890 con 0x7f8ba4107d20 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.389+0000 7f8b9b7fe700 1 -- 172.21.15.27:0/3773330328 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8ba0003b60 con 0x7f8ba4107d20 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.463+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8ba419f310 con 0x7f8ba4107d20 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.463+0000 7f8b9b7fe700 1 -- 172.21.15.27:0/3773330328 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8ba00199a0 con 0x7f8ba4107d20 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.465+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 msgr2=0x7f8ba419bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.465+0000 7f8bac7c4700 1 --2- 172.21.15.27:0/3773330328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba419bc80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0011670 tx=0x7f8ba0011b60 comp rx=0 tx=0).stop 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.465+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 shutdown_connections 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.465+0000 7f8bac7c4700 1 --2- 172.21.15.27:0/3773330328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ba4107d20 0x7f8ba419bc80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.465+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 >> 172.21.15.27:0/3773330328 conn(0x7f8ba4103320 msgr2=0x7f8ba4103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.465+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 shutdown_connections 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:14:58.465+0000 7f8bac7c4700 1 -- 172.21.15.27:0/3773330328 wait complete. 2024-04-25T01:14:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (1/15)... 2024-04-25T01:15:00.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:14:59 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:14:59.930+0000 7f55a37a0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:15:00.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:00.106+0000 7f55a37a0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:15:00.564 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:00.281+0000 7f55a37a0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:15:00.564 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:00.578+0000 7f55a37a0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:15:00.869 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:00.673+0000 7f55a37a0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:15:00.869 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:00.882+0000 7f55a37a0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:15:01.121 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:01 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:01.087+0000 7f55a37a0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:15:01.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:01 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1481213346' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:15:02.083 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:01 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:01.727+0000 7f55a37a0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:15:02.088 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:15:02.088 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:15:02.088 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:15:02.088 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:15:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:15:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:15:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:15:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:15:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:15:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:15:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:15:02.090 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:15:02.099 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:15:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:15:02.101 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:14:45.961872+0000", 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:15:02.102 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.403+0000 7f8da2971700 1 Processor -- start 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.403+0000 7f8da2971700 1 -- start start 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.403+0000 7f8da2971700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c10ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.403+0000 7f8da2971700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c07be50 con 0x7f8d9c108790 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.403+0000 7f8d9bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c10ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.404+0000 7f8d9bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c10ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57152/0 (socket says 172.21.15.27:57152) 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.404+0000 7f8d9bfff700 1 -- 172.21.15.27:0/3915413453 learned_addr learned my addr 172.21.15.27:0/3915413453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.404+0000 7f8d9bfff700 1 -- 172.21.15.27:0/3915413453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c10b0c0 con 0x7f8d9c108790 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.404+0000 7f8d9bfff700 1 --2- 172.21.15.27:0/3915413453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c10ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0098a0 tx=0x7f8d8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa067f9f996d49c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.404+0000 7f8d9affd700 1 -- 172.21.15.27:0/3915413453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d8c004160 con 0x7f8d9c108790 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.404+0000 7f8d9affd700 1 -- 172.21.15.27:0/3915413453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8d8c004300 con 0x7f8d9c108790 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.404+0000 7f8d9affd700 1 -- 172.21.15.27:0/3915413453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d8c011b50 con 0x7f8d9c108790 2024-04-25T01:15:02.103 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.405+0000 7f8da2971700 1 -- 172.21.15.27:0/3915413453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 msgr2=0x7f8d9c10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.405+0000 7f8da2971700 1 --2- 172.21.15.27:0/3915413453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c10ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0098a0 tx=0x7f8d8c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.405+0000 7f8da2971700 1 -- 172.21.15.27:0/3915413453 shutdown_connections 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.405+0000 7f8da2971700 1 --2- 172.21.15.27:0/3915413453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c10ab80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.405+0000 7f8da2971700 1 -- 172.21.15.27:0/3915413453 >> 172.21.15.27:0/3915413453 conn(0x7f8d9c1021c0 msgr2=0x7f8d9c104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.405+0000 7f8da2971700 1 -- 172.21.15.27:0/3915413453 shutdown_connections 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.405+0000 7f8da2971700 1 -- 172.21.15.27:0/3915413453 wait complete. 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.406+0000 7f8da2971700 1 Processor -- start 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.406+0000 7f8da2971700 1 -- start start 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.406+0000 7f8da2971700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.406+0000 7f8da2971700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c10c0f0 con 0x7f8d9c108790 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.406+0000 7f8d9bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.406+0000 7f8d9bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c19bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57168/0 (socket says 172.21.15.27:57168) 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.406+0000 7f8d9bfff700 1 -- 172.21.15.27:0/1481213346 learned_addr learned my addr 172.21.15.27:0/1481213346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8d9bfff700 1 -- 172.21.15.27:0/1481213346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c19c1d0 con 0x7f8d9c108790 2024-04-25T01:15:02.104 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8d9bfff700 1 --2- 172.21.15.27:0/1481213346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c19bc90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c011670 tx=0x7f8d8c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8d997fa700 1 -- 172.21.15.27:0/1481213346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d8c0039c0 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8d997fa700 1 -- 172.21.15.27:0/1481213346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8d8c011e50 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9c19c460 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8d997fa700 1 -- 172.21.15.27:0/1481213346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d8c01a5c0 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9c19f150 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8d997fa700 1 -- 172.21.15.27:0/1481213346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d8c01ac00 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.407+0000 7f8d997fa700 1 -- 172.21.15.27:0/1481213346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8d8c019700 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.408+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d9c063890 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.410+0000 7f8d997fa700 1 -- 172.21.15.27:0/1481213346 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8d8c003b60 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.484+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8d9c19f320 con 0x7f8d9c108790 2024-04-25T01:15:02.105 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.485+0000 7f8d997fa700 1 -- 172.21.15.27:0/1481213346 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8d8c0199a0 con 0x7f8d9c108790 2024-04-25T01:15:02.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.486+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 msgr2=0x7f8d9c19bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:02.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.486+0000 7f8da2971700 1 --2- 172.21.15.27:0/1481213346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c19bc90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c011670 tx=0x7f8d8c011b60 comp rx=0 tx=0).stop 2024-04-25T01:15:02.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.486+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 shutdown_connections 2024-04-25T01:15:02.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.486+0000 7f8da2971700 1 --2- 172.21.15.27:0/1481213346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c108790 0x7f8d9c19bc90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:02.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.486+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 >> 172.21.15.27:0/1481213346 conn(0x7f8d9c1021c0 msgr2=0x7f8d9c102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:02.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.486+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 shutdown_connections 2024-04-25T01:15:02.106 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:01.486+0000 7f8da2971700 1 -- 172.21.15.27:0/1481213346 wait complete. 2024-04-25T01:15:02.106 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (2/15)... 2024-04-25T01:15:02.451 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:02 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:02.188+0000 7f55a37a0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:15:02.451 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:02 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:02.292+0000 7f55a37a0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:15:02.951 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:02 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:02.498+0000 7f55a37a0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:15:03.374 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:03.043+0000 7f55a37a0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:15:03.374 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:03.137+0000 7f55a37a0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:15:03.701 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:03.387+0000 7f55a37a0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:15:03.701 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:03.524+0000 7f55a37a0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:15:03.701 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:03.622+0000 7f55a37a0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:15:04.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:03.741+0000 7f55a37a0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:15:04.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:03.835+0000 7f55a37a0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:15:04.932 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1532831888' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:15:04.932 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:04 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:04.722+0000 7f55a37a0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:15:05.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:04 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:04.950+0000 7f55a37a0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:15:05.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:15:05.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:15:05.264 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:15:05.265 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:15:05.266 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:15:05.267 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:14:45.961872+0000", 2024-04-25T01:15:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.546+0000 7f59622cb700 1 Processor -- start 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.546+0000 7f59622cb700 1 -- start start 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.546+0000 7f59622cb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c10ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.546+0000 7f59622cb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f595c07be50 con 0x7f595c108790 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.547+0000 7f595bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c10ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.547+0000 7f595bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c10ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57174/0 (socket says 172.21.15.27:57174) 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.547+0000 7f595bfff700 1 -- 172.21.15.27:0/3047815973 learned_addr learned my addr 172.21.15.27:0/3047815973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.547+0000 7f595bfff700 1 -- 172.21.15.27:0/3047815973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595c10b0c0 con 0x7f595c108790 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.547+0000 7f595bfff700 1 --2- 172.21.15.27:0/3047815973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c10ab80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f594c0098a0 tx=0x7f594c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f075faa56160ed21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.547+0000 7f595affd700 1 -- 172.21.15.27:0/3047815973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f594c004160 con 0x7f595c108790 2024-04-25T01:15:05.269 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.547+0000 7f595affd700 1 -- 172.21.15.27:0/3047815973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f594c004300 con 0x7f595c108790 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.547+0000 7f595affd700 1 -- 172.21.15.27:0/3047815973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f594c011b50 con 0x7f595c108790 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.548+0000 7f59622cb700 1 -- 172.21.15.27:0/3047815973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 msgr2=0x7f595c10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.548+0000 7f59622cb700 1 --2- 172.21.15.27:0/3047815973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c10ab80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f594c0098a0 tx=0x7f594c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.548+0000 7f59622cb700 1 -- 172.21.15.27:0/3047815973 shutdown_connections 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.548+0000 7f59622cb700 1 --2- 172.21.15.27:0/3047815973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c10ab80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.548+0000 7f59622cb700 1 -- 172.21.15.27:0/3047815973 >> 172.21.15.27:0/3047815973 conn(0x7f595c1021c0 msgr2=0x7f595c104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.548+0000 7f59622cb700 1 -- 172.21.15.27:0/3047815973 shutdown_connections 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.548+0000 7f59622cb700 1 -- 172.21.15.27:0/3047815973 wait complete. 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.549+0000 7f59622cb700 1 Processor -- start 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.549+0000 7f59622cb700 1 -- start start 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.549+0000 7f59622cb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c108010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.549+0000 7f59622cb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f595c10c0f0 con 0x7f595c108790 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.549+0000 7f595bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:05.270 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.549+0000 7f595bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57180/0 (socket says 172.21.15.27:57180) 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.549+0000 7f595bfff700 1 -- 172.21.15.27:0/1532831888 learned_addr learned my addr 172.21.15.27:0/1532831888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.550+0000 7f595bfff700 1 -- 172.21.15.27:0/1532831888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595c108550 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.550+0000 7f595bfff700 1 --2- 172.21.15.27:0/1532831888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c108010 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f594c011670 tx=0x7f594c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.550+0000 7f59597fa700 1 -- 172.21.15.27:0/1532831888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f594c0039c0 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.550+0000 7f59597fa700 1 -- 172.21.15.27:0/1532831888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f594c011e50 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.551+0000 7f59597fa700 1 -- 172.21.15.27:0/1532831888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f594c01a5c0 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.551+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f595c106740 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.551+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f595c106c80 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.551+0000 7f59597fa700 1 -- 172.21.15.27:0/1532831888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f594c0247c0 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.551+0000 7f59597fa700 1 -- 172.21.15.27:0/1532831888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f594c019700 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.552+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f595c063890 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.554+0000 7f59597fa700 1 -- 172.21.15.27:0/1532831888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f594c029060 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.628+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f595c1071b0 con 0x7f595c108790 2024-04-25T01:15:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.629+0000 7f59597fa700 1 -- 172.21.15.27:0/1532831888 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f594c003b60 con 0x7f595c108790 2024-04-25T01:15:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.630+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 msgr2=0x7f595c108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.630+0000 7f59622cb700 1 --2- 172.21.15.27:0/1532831888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c108010 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f594c011670 tx=0x7f594c011b60 comp rx=0 tx=0).stop 2024-04-25T01:15:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.630+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 shutdown_connections 2024-04-25T01:15:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.630+0000 7f59622cb700 1 --2- 172.21.15.27:0/1532831888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f595c108790 0x7f595c108010 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.630+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 >> 172.21.15.27:0/1532831888 conn(0x7f595c1021c0 msgr2=0x7f595c102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.630+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 shutdown_connections 2024-04-25T01:15:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:04.630+0000 7f59622cb700 1 -- 172.21.15.27:0/1532831888 wait complete. 2024-04-25T01:15:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (3/15)... 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: Activating manager daemon x 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: mgrmap e2: x(active, starting, since 0.0030681s) 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: Manager daemon x is now available 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' 2024-04-25T01:15:05.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' 2024-04-25T01:15:05.952 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:05 smithi027 ceph-mon[29197]: from='mgr.14100 172.21.15.27:0/887009137' entity='mgr.x' 2024-04-25T01:15:07.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:06 smithi027 ceph-mon[29197]: mgrmap e3: x(active, since 1.00651s) 2024-04-25T01:15:08.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:07 smithi027 ceph-mon[29197]: mgrmap e4: x(active, since 2s) 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:15:08.619 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:08.620 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:15:08.621 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:08.621 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:15:08.621 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:15:08.621 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:15:08.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:15:08.631 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:14:45.961872+0000", 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.720+0000 7fc01216c700 1 Processor -- start 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.720+0000 7fc01216c700 1 -- start start 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.720+0000 7fc01216c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c1074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.720+0000 7fc01216c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c107ac0 con 0x7fc00c1070d0 2024-04-25T01:15:08.632 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.721+0000 7fc00b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c1074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.721+0000 7fc00b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c1074f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44406/0 (socket says 172.21.15.27:44406) 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.721+0000 7fc00b7fe700 1 -- 172.21.15.27:0/3375243640 learned_addr learned my addr 172.21.15.27:0/3375243640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.721+0000 7fc00b7fe700 1 -- 172.21.15.27:0/3375243640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c107df0 con 0x7fc00c1070d0 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.721+0000 7fc00b7fe700 1 --2- 172.21.15.27:0/3375243640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c1074f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0098a0 tx=0x7fbffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f33fe2f2035170 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc00a7fc700 1 -- 172.21.15.27:0/3375243640 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbffc004160 con 0x7fc00c1070d0 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc00a7fc700 1 -- 172.21.15.27:0/3375243640 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbffc004300 con 0x7fc00c1070d0 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc00a7fc700 1 -- 172.21.15.27:0/3375243640 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbffc011b50 con 0x7fc00c1070d0 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc01216c700 1 -- 172.21.15.27:0/3375243640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 msgr2=0x7fc00c1074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc01216c700 1 --2- 172.21.15.27:0/3375243640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c1074f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0098a0 tx=0x7fbffc009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc01216c700 1 -- 172.21.15.27:0/3375243640 shutdown_connections 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc01216c700 1 --2- 172.21.15.27:0/3375243640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c1074f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:08.633 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc01216c700 1 -- 172.21.15.27:0/3375243640 >> 172.21.15.27:0/3375243640 conn(0x7fc00c101910 msgr2=0x7fc00c103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc01216c700 1 -- 172.21.15.27:0/3375243640 shutdown_connections 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.722+0000 7fc01216c700 1 -- 172.21.15.27:0/3375243640 wait complete. 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.723+0000 7fc01216c700 1 Processor -- start 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.723+0000 7fc01216c700 1 -- start start 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.723+0000 7fc01216c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.723+0000 7fc01216c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c10b1a0 con 0x7fc00c1070d0 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.723+0000 7fc00b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc00b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c19bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44418/0 (socket says 172.21.15.27:44418) 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc00b7fe700 1 -- 172.21.15.27:0/2242175141 learned_addr learned my addr 172.21.15.27:0/2242175141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc00b7fe700 1 -- 172.21.15.27:0/2242175141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c19c1d0 con 0x7fc00c1070d0 2024-04-25T01:15:08.634 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc00b7fe700 1 --2- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c19bc90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009fb0 tx=0x7fbffc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc008ff9700 1 -- 172.21.15.27:0/2242175141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbffc0038b0 con 0x7fc00c1070d0 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc008ff9700 1 -- 172.21.15.27:0/2242175141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbffc011e50 con 0x7fc00c1070d0 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00c19c460 con 0x7fc00c1070d0 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc008ff9700 1 -- 172.21.15.27:0/2242175141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbffc019420 con 0x7fc00c1070d0 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.724+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00c19f150 con 0x7fc00c1070d0 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.725+0000 7fc008ff9700 1 -- 172.21.15.27:0/2242175141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fbffc01a430 con 0x7fc00c1070d0 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.725+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00c063890 con 0x7fc00c1070d0 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.725+0000 7fc008ff9700 1 --2- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fbff4038fb0 0x7fbff403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:08.635 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.725+0000 7fc008ff9700 1 -- 172.21.15.27:0/2242175141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbffc04d1b0 con 0x7fc00c1070d0 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.726+0000 7fc00affd700 1 --2- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fbff4038fb0 0x7fbff403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.726+0000 7fc00affd700 1 --2- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fbff4038fb0 0x7fbff403b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc000006fd0 tx=0x7fc000006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:07.731+0000 7fc008ff9700 1 -- 172.21.15.27:0/2242175141 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbffc003b00 con 0x7fc00c1070d0 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.029+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc00c0008d0 con 0x7fc00c1070d0 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.029+0000 7fc008ff9700 1 -- 172.21.15.27:0/2242175141 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fbffc01a720 con 0x7fc00c1070d0 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.032+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fbff4038fb0 msgr2=0x7fbff403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.032+0000 7fc01216c700 1 --2- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fbff4038fb0 0x7fbff403b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc000006fd0 tx=0x7fc000006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.032+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 msgr2=0x7fc00c19bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.032+0000 7fc01216c700 1 --2- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c19bc90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009fb0 tx=0x7fbffc011bf0 comp rx=0 tx=0).stop 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.032+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 shutdown_connections 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.032+0000 7fc01216c700 1 --2- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fbff4038fb0 0x7fbff403b470 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.032+0000 7fc01216c700 1 --2- 172.21.15.27:0/2242175141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00c1070d0 0x7fc00c19bc90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:08.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.032+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 >> 172.21.15.27:0/2242175141 conn(0x7fc00c101910 msgr2=0x7fc00c1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:08.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.033+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 shutdown_connections 2024-04-25T01:15:08.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:08.033+0000 7fc01216c700 1 -- 172.21.15.27:0/2242175141 wait complete. 2024-04-25T01:15:08.637 INFO:teuthology.orchestra.run.smithi027.stdout:mgr is available 2024-04-25T01:15:09.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:08 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2242175141' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:15:09.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:15:09.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:15:09.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:15:09.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.041+0000 7fac71f4a700 1 Processor -- start 2024-04-25T01:15:09.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.041+0000 7fac71f4a700 1 -- start start 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.041+0000 7fac71f4a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.041+0000 7fac71f4a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c1086e0 con 0x7fac6c107cf0 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.041+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.041+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44426/0 (socket says 172.21.15.27:44426) 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.041+0000 7fac6b7fe700 1 -- 172.21.15.27:0/191505844 learned_addr learned my addr 172.21.15.27:0/191505844 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.041+0000 7fac6b7fe700 1 -- 172.21.15.27:0/191505844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6c108ef0 con 0x7fac6c107cf0 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac6b7fe700 1 --2- 172.21.15.27:0/191505844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c108110 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0098a0 tx=0x7fac5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2362d1c1cc21844 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac6a7fc700 1 -- 172.21.15.27:0/191505844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac5c004160 con 0x7fac6c107cf0 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac6a7fc700 1 -- 172.21.15.27:0/191505844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac5c004300 con 0x7fac6c107cf0 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac6a7fc700 1 -- 172.21.15.27:0/191505844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac5c011b50 con 0x7fac6c107cf0 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac71f4a700 1 -- 172.21.15.27:0/191505844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 msgr2=0x7fac6c108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac71f4a700 1 --2- 172.21.15.27:0/191505844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c108110 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0098a0 tx=0x7fac5c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:09.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac71f4a700 1 -- 172.21.15.27:0/191505844 shutdown_connections 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac71f4a700 1 --2- 172.21.15.27:0/191505844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c108110 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.042+0000 7fac71f4a700 1 -- 172.21.15.27:0/191505844 >> 172.21.15.27:0/191505844 conn(0x7fac6c1032d0 msgr2=0x7fac6c1056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.043+0000 7fac71f4a700 1 -- 172.21.15.27:0/191505844 shutdown_connections 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.043+0000 7fac71f4a700 1 -- 172.21.15.27:0/191505844 wait complete. 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.043+0000 7fac71f4a700 1 Processor -- start 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.043+0000 7fac71f4a700 1 -- start start 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.043+0000 7fac71f4a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac71f4a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c10c2a0 con 0x7fac6c107cf0 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c19bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44442/0 (socket says 172.21.15.27:44442) 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac6b7fe700 1 -- 172.21.15.27:0/231796280 learned_addr learned my addr 172.21.15.27:0/231796280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac6b7fe700 1 -- 172.21.15.27:0/231796280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6c19c1c0 con 0x7fac6c107cf0 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac6b7fe700 1 --2- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c19bc80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fac5c011670 tx=0x7fac5c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:09.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac68ff9700 1 -- 172.21.15.27:0/231796280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac5c0039c0 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac6c19c450 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac68ff9700 1 -- 172.21.15.27:0/231796280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac5c011e50 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac68ff9700 1 -- 172.21.15.27:0/231796280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac5c01a5c0 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.044+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac6c19f140 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.045+0000 7fac68ff9700 1 -- 172.21.15.27:0/231796280 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fac5c024430 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.045+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac6c063890 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.045+0000 7fac68ff9700 1 --2- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fac54038fb0 0x7fac5403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.045+0000 7fac68ff9700 1 -- 172.21.15.27:0/231796280 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fac5c04d500 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.045+0000 7fac6affd700 1 --2- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fac54038fb0 0x7fac5403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.046+0000 7fac6affd700 1 --2- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fac54038fb0 0x7fac5403b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fac60006fd0 tx=0x7fac60006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.050+0000 7fac68ff9700 1 -- 172.21.15.27:0/231796280 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fac5c04cf20 con 0x7fac6c107cf0 2024-04-25T01:15:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.254+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fac6c19f310 con 0x7fac6c107cf0 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.255+0000 7fac68ff9700 1 -- 172.21.15.27:0/231796280 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fac6c19f310 con 0x7fac6c107cf0 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fac54038fb0 msgr2=0x7fac5403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 --2- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fac54038fb0 0x7fac5403b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fac60006fd0 tx=0x7fac60006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 msgr2=0x7fac6c19bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 --2- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c19bc80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fac5c011670 tx=0x7fac5c011b60 comp rx=0 tx=0).stop 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 shutdown_connections 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 --2- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7fac54038fb0 0x7fac5403b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 --2- 172.21.15.27:0/231796280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac6c107cf0 0x7fac6c19bc80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 >> 172.21.15.27:0/231796280 conn(0x7fac6c1032d0 msgr2=0x7fac6c103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 shutdown_connections 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:09.257+0000 7fac71f4a700 1 -- 172.21.15.27:0/231796280 wait complete. 2024-04-25T01:15:09.800 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling cephadm module... 2024-04-25T01:15:10.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:09 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/231796280' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:15:11.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:10 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/284950586' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T01:15:11.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: ignoring --setuser ceph since I am not root 2024-04-25T01:15:11.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: ignoring --setgroup ceph since I am not root 2024-04-25T01:15:11.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:11.136+0000 7f03a57f7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:15:11.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.277+0000 7f1628a50700 1 Processor -- start 2024-04-25T01:15:11.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.277+0000 7f1628a50700 1 -- start start 2024-04-25T01:15:11.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.277+0000 7f1628a50700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162410abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:11.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.277+0000 7f1628a50700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f162407bdd0 con 0x7f1624108800 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.277+0000 7f162259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162410abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.277+0000 7f162259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162410abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44458/0 (socket says 172.21.15.27:44458) 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.277+0000 7f162259c700 1 -- 172.21.15.27:0/1417581031 learned_addr learned my addr 172.21.15.27:0/1417581031 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.278+0000 7f162259c700 1 -- 172.21.15.27:0/1417581031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162410b130 con 0x7f1624108800 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.278+0000 7f162259c700 1 --2- 172.21.15.27:0/1417581031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162410abf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f16140098a0 tx=0x7f1614009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230ef7564ae5281e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.278+0000 7f162159a700 1 -- 172.21.15.27:0/1417581031 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1614004160 con 0x7f1624108800 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.278+0000 7f162159a700 1 -- 172.21.15.27:0/1417581031 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1614004300 con 0x7f1624108800 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.278+0000 7f162159a700 1 -- 172.21.15.27:0/1417581031 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1614011b50 con 0x7f1624108800 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.279+0000 7f1628a50700 1 -- 172.21.15.27:0/1417581031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 msgr2=0x7f162410abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.279+0000 7f1628a50700 1 --2- 172.21.15.27:0/1417581031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162410abf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f16140098a0 tx=0x7f1614009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.279+0000 7f1628a50700 1 -- 172.21.15.27:0/1417581031 shutdown_connections 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.279+0000 7f1628a50700 1 --2- 172.21.15.27:0/1417581031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162410abf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.279+0000 7f1628a50700 1 -- 172.21.15.27:0/1417581031 >> 172.21.15.27:0/1417581031 conn(0x7f16241021d0 msgr2=0x7f16241045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:11.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.279+0000 7f1628a50700 1 -- 172.21.15.27:0/1417581031 shutdown_connections 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.279+0000 7f1628a50700 1 -- 172.21.15.27:0/1417581031 wait complete. 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.279+0000 7f1628a50700 1 Processor -- start 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f1628a50700 1 -- start start 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f1628a50700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f1628a50700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f162410c0f0 con 0x7f1624108800 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f162259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f162259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162419bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44474/0 (socket says 172.21.15.27:44474) 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f162259c700 1 -- 172.21.15.27:0/284950586 learned_addr learned my addr 172.21.15.27:0/284950586 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f162259c700 1 -- 172.21.15.27:0/284950586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162419c1b0 con 0x7f1624108800 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f162259c700 1 --2- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162419bc70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1614011670 tx=0x7f1614011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f16137fe700 1 -- 172.21.15.27:0/284950586 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16140039c0 con 0x7f1624108800 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f16137fe700 1 -- 172.21.15.27:0/284950586 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1614011e50 con 0x7f1624108800 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.280+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f162419c440 con 0x7f1624108800 2024-04-25T01:15:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.281+0000 7f16137fe700 1 -- 172.21.15.27:0/284950586 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f161401a5c0 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.281+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f162419f130 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.281+0000 7f16137fe700 1 -- 172.21.15.27:0/284950586 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f161401a8c0 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.281+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1624063890 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.282+0000 7f16137fe700 1 --2- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f160c038fb0 0x7f160c03b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.282+0000 7f16137fe700 1 -- 172.21.15.27:0/284950586 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f161404d530 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.282+0000 7f1621d9b700 1 --2- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f160c038fb0 0x7f160c03b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.282+0000 7f1621d9b700 1 --2- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f160c038fb0 0x7f160c03b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1618006fd0 tx=0x7f1618006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.287+0000 7f16137fe700 1 -- 172.21.15.27:0/284950586 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1614019c70 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.551+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f162419f300 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.963+0000 7f16137fe700 1 -- 172.21.15.27:0/284950586 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f162419f300 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.965+0000 7f16137fe700 1 -- 172.21.15.27:0/284950586 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f161402d910 con 0x7f1624108800 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f160c038fb0 msgr2=0x7f160c03b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 --2- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f160c038fb0 0x7f160c03b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1618006fd0 tx=0x7f1618006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 msgr2=0x7f162419bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:11.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 --2- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162419bc70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1614011670 tx=0x7f1614011b60 comp rx=0 tx=0).stop 2024-04-25T01:15:11.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 shutdown_connections 2024-04-25T01:15:11.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 --2- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f160c038fb0 0x7f160c03b470 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:11.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 --2- 172.21.15.27:0/284950586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1624108800 0x7f162419bc70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:11.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 >> 172.21.15.27:0/284950586 conn(0x7f16241021d0 msgr2=0x7f1624102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:11.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 shutdown_connections 2024-04-25T01:15:11.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:10.967+0000 7f1628a50700 1 -- 172.21.15.27:0/284950586 wait complete. 2024-04-25T01:15:11.653 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:11.332+0000 7f03a57f7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:15:11.653 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:11.438+0000 7f03a57f7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:15:11.937 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:11.767+0000 7f03a57f7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:15:11.937 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:11.858+0000 7f03a57f7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:15:12.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/284950586' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T01:15:12.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:11 smithi027 ceph-mon[29197]: mgrmap e5: x(active, since 6s) 2024-04-25T01:15:12.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:15:12.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T01:15:12.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:15:12.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:15:12.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:15:12.865 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.987+0000 7f97ebda1700 1 Processor -- start 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.987+0000 7f97ebda1700 1 -- start start 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.987+0000 7f97ebda1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e4106400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.987+0000 7f97ebda1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97e41069d0 con 0x7f97e4105fe0 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.988+0000 7f97e9b3d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e4106400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.988+0000 7f97e9b3d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e4106400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44514/0 (socket says 172.21.15.27:44514) 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.988+0000 7f97e9b3d700 1 -- 172.21.15.27:0/1227262933 learned_addr learned my addr 172.21.15.27:0/1227262933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.988+0000 7f97e9b3d700 1 -- 172.21.15.27:0/1227262933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e4106d00 con 0x7f97e4105fe0 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.988+0000 7f97e9b3d700 1 --2- 172.21.15.27:0/1227262933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e4106400 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97d40098a0 tx=0x7f97d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b61beea4f0050 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.988+0000 7f97e8b3b700 1 -- 172.21.15.27:0/1227262933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97d4004160 con 0x7f97e4105fe0 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.988+0000 7f97e8b3b700 1 -- 172.21.15.27:0/1227262933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f97d4004300 con 0x7f97e4105fe0 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.988+0000 7f97e8b3b700 1 -- 172.21.15.27:0/1227262933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97d4011b50 con 0x7f97e4105fe0 2024-04-25T01:15:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.989+0000 7f97ebda1700 1 -- 172.21.15.27:0/1227262933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 msgr2=0x7f97e4106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.989+0000 7f97ebda1700 1 --2- 172.21.15.27:0/1227262933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e4106400 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97d40098a0 tx=0x7f97d4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.989+0000 7f97ebda1700 1 -- 172.21.15.27:0/1227262933 shutdown_connections 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.989+0000 7f97ebda1700 1 --2- 172.21.15.27:0/1227262933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e4106400 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.989+0000 7f97ebda1700 1 -- 172.21.15.27:0/1227262933 >> 172.21.15.27:0/1227262933 conn(0x7f97e4101990 msgr2=0x7f97e4103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.989+0000 7f97ebda1700 1 -- 172.21.15.27:0/1227262933 shutdown_connections 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.989+0000 7f97ebda1700 1 -- 172.21.15.27:0/1227262933 wait complete. 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.990+0000 7f97ebda1700 1 Processor -- start 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.990+0000 7f97ebda1700 1 -- start start 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.990+0000 7f97ebda1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.990+0000 7f97ebda1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97e41080c0 con 0x7f97e4105fe0 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.990+0000 7f97e9b3d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.990+0000 7f97e9b3d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e419dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44528/0 (socket says 172.21.15.27:44528) 2024-04-25T01:15:12.867 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.990+0000 7f97e9b3d700 1 -- 172.21.15.27:0/2854031477 learned_addr learned my addr 172.21.15.27:0/2854031477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.990+0000 7f97e9b3d700 1 -- 172.21.15.27:0/2854031477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e419e3e0 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.991+0000 7f97e9b3d700 1 --2- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e419dea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97d4000c00 tx=0x7f97d4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.991+0000 7f97daffd700 1 -- 172.21.15.27:0/2854031477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97d4003b20 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.991+0000 7f97daffd700 1 -- 172.21.15.27:0/2854031477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f97d4003cc0 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.991+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e419e670 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.991+0000 7f97daffd700 1 -- 172.21.15.27:0/2854031477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97d4024910 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.991+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e41a1360 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.992+0000 7f97daffd700 1 -- 172.21.15.27:0/2854031477 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f97d4022020 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.992+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e4063890 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.992+0000 7f97daffd700 1 --2- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f97d00390a0 0x7f97d003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.992+0000 7f97daffd700 1 -- 172.21.15.27:0/2854031477 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f97d4011e00 con 0x7f97e4105fe0 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.992+0000 7f97e933c700 1 -- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f97d00390a0 msgr2=0x7f97d003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/219526886 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:11.992+0000 7f97e933c700 1 --2- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f97d00390a0 0x7f97d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:15:12.868 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.001+0000 7f97daffd700 1 -- 172.21.15.27:0/2854031477 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f97d4017ca0 con 0x7f97e4105fe0 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.193+0000 7f97e933c700 1 -- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f97d00390a0 msgr2=0x7f97d003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/219526886 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.193+0000 7f97e933c700 1 --2- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f97d00390a0 0x7f97d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.303+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f97e41a1530 con 0x7f97e4105fe0 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.305+0000 7f97daffd700 1 -- 172.21.15.27:0/2854031477 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f97e41a1530 con 0x7f97e4105fe0 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.311+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f97d00390a0 msgr2=0x7f97d003b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.311+0000 7f97ebda1700 1 --2- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f97d00390a0 0x7f97d003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.311+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 msgr2=0x7f97e419dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.311+0000 7f97ebda1700 1 --2- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e419dea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97d4000c00 tx=0x7f97d4011c10 comp rx=0 tx=0).stop 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.316+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 shutdown_connections 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.316+0000 7f97ebda1700 1 --2- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f97d00390a0 0x7f97d003b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.316+0000 7f97ebda1700 1 --2- 172.21.15.27:0/2854031477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e4105fe0 0x7f97e419dea0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.316+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 >> 172.21.15.27:0/2854031477 conn(0x7f97e4101990 msgr2=0x7f97e41944e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.316+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 shutdown_connections 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:12.316+0000 7f97ebda1700 1 -- 172.21.15.27:0/2854031477 wait complete. 2024-04-25T01:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-04-25T01:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 5... 2024-04-25T01:15:13.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:12 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2854031477' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:15:13.434 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:13.135+0000 7f03a57f7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:15:13.434 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:13.316+0000 7f03a57f7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:15:13.700 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:13.492+0000 7f03a57f7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:15:14.083 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:13.793+0000 7f03a57f7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:15:14.084 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:13.889+0000 7f03a57f7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:15:14.084 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:14 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:14.098+0000 7f03a57f7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:15:14.450 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:14 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:14.298+0000 7f03a57f7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:15:15.200 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:14 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:14.924+0000 7f03a57f7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:15:15.665 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:15.366+0000 7f03a57f7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:15:15.665 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:15.466+0000 7f03a57f7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:15:15.665 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:15.680+0000 7f03a57f7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:15:16.558 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:16.225+0000 7f03a57f7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:15:16.558 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:16.320+0000 7f03a57f7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:15:16.915 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:16.573+0000 7f03a57f7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:15:16.915 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:16.712+0000 7f03a57f7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:15:16.915 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:16.810+0000 7f03a57f7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:15:16.915 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:16.930+0000 7f03a57f7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:15:17.200 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:17 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:17.025+0000 7f03a57f7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:15:18.138 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:17 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:17.876+0000 7f03a57f7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:15:18.138 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:18.100+0000 7f03a57f7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: Active manager daemon x restarted 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: Activating manager daemon x 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: mgrmap e6: x(active, starting, since 0.00344498s) 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: Manager daemon x is now available 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:18.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:18.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:15:18.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:15:18.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:15:18.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:18 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:15:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:19 smithi027 ceph-mon[29197]: Found migration_current of "None". Setting to last migration. 2024-04-25T01:15:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:19 smithi027 ceph-mon[29197]: mgrmap e7: x(active, since 1.00591s) 2024-04-25T01:15:19.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:15:19.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T01:15:19.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:15:19.691 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.243+0000 7f7f9be81700 1 Processor -- start 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.243+0000 7f7f9be81700 1 -- start start 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.243+0000 7f7f9be81700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f9410abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.243+0000 7f7f9be81700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9407bdf0 con 0x7f7f941087f0 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.243+0000 7f7f99c1d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f9410abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.243+0000 7f7f99c1d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f9410abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44540/0 (socket says 172.21.15.27:44540) 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.243+0000 7f7f99c1d700 1 -- 172.21.15.27:0/1487345629 learned_addr learned my addr 172.21.15.27:0/1487345629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.244+0000 7f7f99c1d700 1 -- 172.21.15.27:0/1487345629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9410b120 con 0x7f7f941087f0 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.244+0000 7f7f99c1d700 1 --2- 172.21.15.27:0/1487345629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f9410abe0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7f900098a0 tx=0x7f7f90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95303be2ce98dd31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.244+0000 7f7f98c1b700 1 -- 172.21.15.27:0/1487345629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f90004160 con 0x7f7f941087f0 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.244+0000 7f7f98c1b700 1 -- 172.21.15.27:0/1487345629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f90004300 con 0x7f7f941087f0 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.244+0000 7f7f98c1b700 1 -- 172.21.15.27:0/1487345629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f90011b50 con 0x7f7f941087f0 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 -- 172.21.15.27:0/1487345629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 msgr2=0x7f7f9410abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 --2- 172.21.15.27:0/1487345629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f9410abe0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7f900098a0 tx=0x7f7f90009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 -- 172.21.15.27:0/1487345629 shutdown_connections 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 --2- 172.21.15.27:0/1487345629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f9410abe0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 -- 172.21.15.27:0/1487345629 >> 172.21.15.27:0/1487345629 conn(0x7f7f941021c0 msgr2=0x7f7f94104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 -- 172.21.15.27:0/1487345629 shutdown_connections 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 -- 172.21.15.27:0/1487345629 wait complete. 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 Processor -- start 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 -- start start 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f94108070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.245+0000 7f7f9be81700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9410c0f0 con 0x7f7f941087f0 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f99c1d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f94108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f99c1d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f94108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44548/0 (socket says 172.21.15.27:44548) 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f99c1d700 1 -- 172.21.15.27:0/1513350435 learned_addr learned my addr 172.21.15.27:0/1513350435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f99c1d700 1 -- 172.21.15.27:0/1513350435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f941085b0 con 0x7f7f941087f0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f99c1d700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f94108070 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7f90011b60 tx=0x7f7f90011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f90003b20 con 0x7f7f941087f0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f941067a0 con 0x7f7f941087f0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f90003cc0 con 0x7f7f941087f0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f90024840 con 0x7f7f941087f0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.246+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f94106ce0 con 0x7f7f941087f0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.247+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f7f90022020 con 0x7f7f941087f0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.247+0000 7f7f8affd700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 0x7f7f8003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.247+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 --> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7f94063890 con 0x7f7f80039050 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.247+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7f90011e00 con 0x7f7f941087f0 2024-04-25T01:15:19.694 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.247+0000 7f7f9941c700 1 -- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 msgr2=0x7f7f8003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/219526886 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.247+0000 7f7f9941c700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 0x7f7f8003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.448+0000 7f7f9941c700 1 -- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 msgr2=0x7f7f8003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/219526886 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.448+0000 7f7f9941c700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 0x7f7f8003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.849+0000 7f7f9941c700 1 -- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 msgr2=0x7f7f8003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/219526886 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:13.849+0000 7f7f9941c700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 0x7f7f8003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:14.650+0000 7f7f9941c700 1 -- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 msgr2=0x7f7f8003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/219526886 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:14.650+0000 7f7f9941c700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 0x7f7f8003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:16.252+0000 7f7f9941c700 1 -- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 msgr2=0x7f7f8003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/219526886 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:16.252+0000 7f7f9941c700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 0x7f7f8003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:18.104+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f7f9002d910 con 0x7f7f941087f0 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:18.104+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 msgr2=0x7f7f8003b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:18.104+0000 7f7f8affd700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 0x7f7f8003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:19.695 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.105+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f7f9004eb00 con 0x7f7f941087f0 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.105+0000 7f7f8affd700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f7f8003c790 0x7f7f8003eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.105+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7f94063890 con 0x7f7f8003c790 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.106+0000 7f7f9941c700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f7f8003c790 0x7f7f8003eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.106+0000 7f7f9941c700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f7f8003c790 0x7f7f8003eb80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f84003a10 tx=0x7f7f84009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.107+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mgr.14120 v2:172.21.15.27:6800/3353153293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f7f94063890 con 0x7f7f8003c790 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.113+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f7f94103410 con 0x7f7f8003c790 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f8affd700 1 -- 172.21.15.27:0/1513350435 <== mgr.14120 v2:172.21.15.27:6800/3353153293 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f7f94103410 con 0x7f7f8003c790 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f7f8003c790 msgr2=0x7f7f8003eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f7f8003c790 0x7f7f8003eb80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f84003a10 tx=0x7f7f84009290 comp rx=0 tx=0).stop 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 msgr2=0x7f7f94108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:19.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f94108070 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7f90011b60 tx=0x7f7f90011c40 comp rx=0 tx=0).stop 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 shutdown_connections 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f7f8003c790 0x7f7f8003eb80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941087f0 0x7f7f94108070 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 --2- 172.21.15.27:0/1513350435 >> [v2:172.21.15.27:6800/219526886,v1:172.21.15.27:6801/219526886] conn(0x7f7f80039050 0x7f7f8003b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 >> 172.21.15.27:0/1513350435 conn(0x7f7f941021c0 msgr2=0x7f7f94102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 shutdown_connections 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:19.114+0000 7f7f9be81700 1 -- 172.21.15.27:0/1513350435 wait complete. 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 5 is available 2024-04-25T01:15:19.697 INFO:teuthology.orchestra.run.smithi027.stdout:Setting orchestrator backend to cephadm... 2024-04-25T01:15:20.338 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:20 smithi027 ceph-mon[29197]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:15:20.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:20 smithi027 ceph-mon[29197]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:15:20.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:20 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:20.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:20 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:20.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:20 smithi027 ceph-mon[29197]: [25/Apr/2024:01:15:19] ENGINE Bus STARTING 2024-04-25T01:15:20.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:20 smithi027 ceph-mon[29197]: [25/Apr/2024:01:15:19] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T01:15:20.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:20 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:15:20.939 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.118+0000 7f9bd4b36700 1 Processor -- start 2024-04-25T01:15:20.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.118+0000 7f9bd4b36700 1 -- start start 2024-04-25T01:15:20.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.118+0000 7f9bd4b36700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd007c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:20.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.118+0000 7f9bd4b36700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd007c870 con 0x7f9bd007de40 2024-04-25T01:15:20.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.118+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd007c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:20.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.118+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd007c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50362/0 (socket says 172.21.15.27:50362) 2024-04-25T01:15:20.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.118+0000 7f9bce59c700 1 -- 172.21.15.27:0/324340482 learned_addr learned my addr 172.21.15.27:0/324340482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:20.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.118+0000 7f9bce59c700 1 -- 172.21.15.27:0/324340482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd007c9f0 con 0x7f9bd007de40 2024-04-25T01:15:20.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bce59c700 1 --2- 172.21.15.27:0/324340482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd007c2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009e80 tx=0x7f9bc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47beb83b5568c23e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:20.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bcd59a700 1 -- 172.21.15.27:0/324340482 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc0017070 con 0x7f9bd007de40 2024-04-25T01:15:20.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bcd59a700 1 -- 172.21.15.27:0/324340482 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9bc000b8c0 con 0x7f9bd007de40 2024-04-25T01:15:20.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bcd59a700 1 -- 172.21.15.27:0/324340482 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc00045c0 con 0x7f9bd007de40 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bd4b36700 1 -- 172.21.15.27:0/324340482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 msgr2=0x7f9bd007c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bd4b36700 1 --2- 172.21.15.27:0/324340482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd007c2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009e80 tx=0x7f9bc000b330 comp rx=0 tx=0).stop 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bd4b36700 1 -- 172.21.15.27:0/324340482 shutdown_connections 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bd4b36700 1 --2- 172.21.15.27:0/324340482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd007c2a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bd4b36700 1 -- 172.21.15.27:0/324340482 >> 172.21.15.27:0/324340482 conn(0x7f9bd0103270 msgr2=0x7f9bd01056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.119+0000 7f9bd4b36700 1 -- 172.21.15.27:0/324340482 shutdown_connections 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.120+0000 7f9bd4b36700 1 -- 172.21.15.27:0/324340482 wait complete. 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.120+0000 7f9bd4b36700 1 Processor -- start 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.120+0000 7f9bd4b36700 1 -- start start 2024-04-25T01:15:20.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.120+0000 7f9bd4b36700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:20.951 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.120+0000 7f9bd4b36700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd010e2e0 con 0x7f9bd007de40 2024-04-25T01:15:20.951 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:20.951 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50368/0 (socket says 172.21.15.27:50368) 2024-04-25T01:15:20.951 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bce59c700 1 -- 172.21.15.27:0/1914034423 learned_addr learned my addr 172.21.15.27:0/1914034423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:20.951 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bce59c700 1 -- 172.21.15.27:0/1914034423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd019e3e0 con 0x7f9bd007de40 2024-04-25T01:15:20.951 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bce59c700 1 --2- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd019dea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009e50 tx=0x7f9bc000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:20.951 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bc77fe700 1 -- 172.21.15.27:0/1914034423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc0020020 con 0x7f9bd007de40 2024-04-25T01:15:20.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd019e670 con 0x7f9bd007de40 2024-04-25T01:15:20.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bc77fe700 1 -- 172.21.15.27:0/1914034423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9bc0003b50 con 0x7f9bd007de40 2024-04-25T01:15:20.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bc77fe700 1 -- 172.21.15.27:0/1914034423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc0017040 con 0x7f9bd007de40 2024-04-25T01:15:20.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.121+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd01a1360 con 0x7f9bd007de40 2024-04-25T01:15:20.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.122+0000 7f9bc77fe700 1 -- 172.21.15.27:0/1914034423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f9bc0023020 con 0x7f9bd007de40 2024-04-25T01:15:20.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.122+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd0063890 con 0x7f9bd007de40 2024-04-25T01:15:20.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.122+0000 7f9bc77fe700 1 --2- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9bbc038ed0 0x7f9bbc03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:20.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.122+0000 7f9bc77fe700 1 -- 172.21.15.27:0/1914034423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9bc000f050 con 0x7f9bd007de40 2024-04-25T01:15:20.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.122+0000 7f9bcdd9b700 1 --2- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9bbc038ed0 0x7f9bbc03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:20.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.123+0000 7f9bcdd9b700 1 --2- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9bbc038ed0 0x7f9bbc03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8006fd0 tx=0x7f9bb8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:20.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.128+0000 7f9bc77fe700 1 -- 172.21.15.27:0/1914034423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9bc002a020 con 0x7f9bd007de40 2024-04-25T01:15:20.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.260+0000 7f9bc77fe700 1 -- 172.21.15.27:0/1914034423 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f9bc0025400 con 0x7f9bd007de40 2024-04-25T01:15:20.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.353+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f9bd01044e0 con 0x7f9bbc038ed0 2024-04-25T01:15:20.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.358+0000 7f9bc77fe700 1 -- 172.21.15.27:0/1914034423 <== mgr.14120 v2:172.21.15.27:6800/3353153293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9bd01044e0 con 0x7f9bbc038ed0 2024-04-25T01:15:20.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9bbc038ed0 msgr2=0x7f9bbc03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:20.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 --2- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9bbc038ed0 0x7f9bbc03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8006fd0 tx=0x7f9bb8006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:20.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 msgr2=0x7f9bd019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:20.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 --2- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd019dea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009e50 tx=0x7f9bc000bd30 comp rx=0 tx=0).stop 2024-04-25T01:15:20.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 shutdown_connections 2024-04-25T01:15:20.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 --2- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9bbc038ed0 0x7f9bbc03b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:20.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 --2- 172.21.15.27:0/1914034423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd007de40 0x7f9bd019dea0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:20.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 >> 172.21.15.27:0/1914034423 conn(0x7f9bd0103270 msgr2=0x7f9bd0103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:20.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 shutdown_connections 2024-04-25T01:15:20.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:20.360+0000 7f9bd4b36700 1 -- 172.21.15.27:0/1914034423 wait complete. 2024-04-25T01:15:21.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:21 smithi027 ceph-mon[29197]: [25/Apr/2024:01:15:20] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T01:15:21.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:21 smithi027 ceph-mon[29197]: [25/Apr/2024:01:15:20] ENGINE Bus STARTED 2024-04-25T01:15:21.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:21 smithi027 ceph-mon[29197]: mgrmap e8: x(active, since 2s) 2024-04-25T01:15:21.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:21 smithi027 ceph-mon[29197]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:21.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:21 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:21.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:21 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:15:22.196 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T01:15:22.196 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.404+0000 7fcafe4d7700 1 Processor -- start 2024-04-25T01:15:22.196 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.404+0000 7fcafe4d7700 1 -- start start 2024-04-25T01:15:22.196 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.404+0000 7fcafe4d7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf810abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.404+0000 7fcafe4d7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf807bdd0 con 0x7fcaf8108800 2024-04-25T01:15:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.404+0000 7fcaf7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf810abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.404+0000 7fcaf7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf810abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50382/0 (socket says 172.21.15.27:50382) 2024-04-25T01:15:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.404+0000 7fcaf7fff700 1 -- 172.21.15.27:0/3430443860 learned_addr learned my addr 172.21.15.27:0/3430443860 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.404+0000 7fcaf7fff700 1 -- 172.21.15.27:0/3430443860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf810b130 con 0x7fcaf8108800 2024-04-25T01:15:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcaf7fff700 1 --2- 172.21.15.27:0/3430443860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf810abf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcae80098a0 tx=0x7fcae8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4a0f3cc93602df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcaf6ffd700 1 -- 172.21.15.27:0/3430443860 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcae8004160 con 0x7fcaf8108800 2024-04-25T01:15:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcaf6ffd700 1 -- 172.21.15.27:0/3430443860 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcae8004300 con 0x7fcaf8108800 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcaf6ffd700 1 -- 172.21.15.27:0/3430443860 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcae8011b50 con 0x7fcaf8108800 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcafe4d7700 1 -- 172.21.15.27:0/3430443860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 msgr2=0x7fcaf810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcafe4d7700 1 --2- 172.21.15.27:0/3430443860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf810abf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcae80098a0 tx=0x7fcae8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcafe4d7700 1 -- 172.21.15.27:0/3430443860 shutdown_connections 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcafe4d7700 1 --2- 172.21.15.27:0/3430443860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf810abf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcafe4d7700 1 -- 172.21.15.27:0/3430443860 >> 172.21.15.27:0/3430443860 conn(0x7fcaf81021d0 msgr2=0x7fcaf81045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcafe4d7700 1 -- 172.21.15.27:0/3430443860 shutdown_connections 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.405+0000 7fcafe4d7700 1 -- 172.21.15.27:0/3430443860 wait complete. 2024-04-25T01:15:22.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.406+0000 7fcafe4d7700 1 Processor -- start 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.406+0000 7fcafe4d7700 1 -- start start 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.406+0000 7fcafe4d7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf819bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.406+0000 7fcafe4d7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf810c0f0 con 0x7fcaf8108800 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.406+0000 7fcaf7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf819bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.406+0000 7fcaf7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf819bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50384/0 (socket says 172.21.15.27:50384) 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.406+0000 7fcaf7fff700 1 -- 172.21.15.27:0/1756719368 learned_addr learned my addr 172.21.15.27:0/1756719368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.407+0000 7fcaf7fff700 1 -- 172.21.15.27:0/1756719368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf819c1d0 con 0x7fcaf8108800 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.407+0000 7fcaf7fff700 1 --2- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf819bc90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcae8011670 tx=0x7fcae8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.407+0000 7fcaf57fa700 1 -- 172.21.15.27:0/1756719368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcae80039c0 con 0x7fcaf8108800 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.407+0000 7fcaf57fa700 1 -- 172.21.15.27:0/1756719368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcae8011e50 con 0x7fcaf8108800 2024-04-25T01:15:22.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.407+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf819c460 con 0x7fcaf8108800 2024-04-25T01:15:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.407+0000 7fcaf57fa700 1 -- 172.21.15.27:0/1756719368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcae801a5c0 con 0x7fcaf8108800 2024-04-25T01:15:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.407+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf819f150 con 0x7fcaf8108800 2024-04-25T01:15:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.408+0000 7fcaf57fa700 1 -- 172.21.15.27:0/1756719368 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fcae801a870 con 0x7fcaf8108800 2024-04-25T01:15:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.408+0000 7fcaf57fa700 1 --2- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fcae0039050 0x7fcae003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.408+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf8063890 con 0x7fcaf8108800 2024-04-25T01:15:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.408+0000 7fcaf57fa700 1 -- 172.21.15.27:0/1756719368 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcae804d560 con 0x7fcaf8108800 2024-04-25T01:15:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.408+0000 7fcaf77fe700 1 --2- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fcae0039050 0x7fcae003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.409+0000 7fcaf77fe700 1 --2- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fcae0039050 0x7fcae003b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcaec006fd0 tx=0x7fcaec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.413+0000 7fcaf57fa700 1 -- 172.21.15.27:0/1756719368 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcae802cc00 con 0x7fcaf8108800 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.623+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fcaf81033b0 con 0x7fcae0039050 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.623+0000 7fcaf57fa700 1 -- 172.21.15.27:0/1756719368 <== mgr.14120 v2:172.21.15.27:6800/3353153293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fcaf81033b0 con 0x7fcae0039050 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.626+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fcae0039050 msgr2=0x7fcae003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.626+0000 7fcafe4d7700 1 --2- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fcae0039050 0x7fcae003b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcaec006fd0 tx=0x7fcaec006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.626+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 msgr2=0x7fcaf819bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.626+0000 7fcafe4d7700 1 --2- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf819bc90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcae8011670 tx=0x7fcae8011b60 comp rx=0 tx=0).stop 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.626+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 shutdown_connections 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.626+0000 7fcafe4d7700 1 --2- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fcae0039050 0x7fcae003b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:22.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.626+0000 7fcafe4d7700 1 --2- 172.21.15.27:0/1756719368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf8108800 0x7fcaf819bc90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:22.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.627+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 >> 172.21.15.27:0/1756719368 conn(0x7fcaf81021d0 msgr2=0x7fcaf8102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:22.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.627+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 shutdown_connections 2024-04-25T01:15:22.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:21.627+0000 7fcafe4d7700 1 -- 172.21.15.27:0/1756719368 wait complete. 2024-04-25T01:15:22.202 INFO:teuthology.orchestra.run.smithi027.stdout:Generating ssh key... 2024-04-25T01:15:22.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-mon[29197]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: Generating public/private rsa key pair. 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: Your identification has been saved in /tmp/tmpnjf7wdgv/key. 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: Your public key has been saved in /tmp/tmpnjf7wdgv/key.pub. 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: The key fingerprint is: 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: SHA256:LvbvUUNvCLRlf7pc4hALOkyTDDZn3GX8Zw/x5OBODVI ceph-1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: The key's randomart image is: 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: +---[RSA 3072]----+ 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | +.o..o=.E | 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | . *.o.=o.+ .| 2024-04-25T01:15:23.201 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | = + o+.O.| 2024-04-25T01:15:23.202 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | o o + ==o*| 2024-04-25T01:15:23.202 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | S *o=+o| 2024-04-25T01:15:23.202 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | . . . *.+.| 2024-04-25T01:15:23.202 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | o . . + | 2024-04-25T01:15:23.202 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | . o . | 2024-04-25T01:15:23.202 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: | .oo | 2024-04-25T01:15:23.202 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: +----[SHA256]-----+ 2024-04-25T01:15:23.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.618+0000 7f74c9575700 1 Processor -- start 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.618+0000 7f74c9575700 1 -- start start 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c9575700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c410ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c9575700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f74c407bdd0 con 0x7f74c4108830 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c410ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c410ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50400/0 (socket says 172.21.15.27:50400) 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c2ffd700 1 -- 172.21.15.27:0/2808022188 learned_addr learned my addr 172.21.15.27:0/2808022188 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c2ffd700 1 -- 172.21.15.27:0/2808022188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c410b160 con 0x7f74c4108830 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c2ffd700 1 --2- 172.21.15.27:0/2808022188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c410ac20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f74b40098a0 tx=0x7f74b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6f94237111bad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c1ffb700 1 -- 172.21.15.27:0/2808022188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74b4004160 con 0x7f74c4108830 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c1ffb700 1 -- 172.21.15.27:0/2808022188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f74b4004300 con 0x7f74c4108830 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.619+0000 7f74c1ffb700 1 -- 172.21.15.27:0/2808022188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74b4011b50 con 0x7f74c4108830 2024-04-25T01:15:23.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.620+0000 7f74c9575700 1 -- 172.21.15.27:0/2808022188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 msgr2=0x7f74c410ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.620+0000 7f74c9575700 1 --2- 172.21.15.27:0/2808022188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c410ac20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f74b40098a0 tx=0x7f74b4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.620+0000 7f74c9575700 1 -- 172.21.15.27:0/2808022188 shutdown_connections 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.620+0000 7f74c9575700 1 --2- 172.21.15.27:0/2808022188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c410ac20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.620+0000 7f74c9575700 1 -- 172.21.15.27:0/2808022188 >> 172.21.15.27:0/2808022188 conn(0x7f74c41021e0 msgr2=0x7f74c4104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.620+0000 7f74c9575700 1 -- 172.21.15.27:0/2808022188 shutdown_connections 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.620+0000 7f74c9575700 1 -- 172.21.15.27:0/2808022188 wait complete. 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.621+0000 7f74c9575700 1 Processor -- start 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.621+0000 7f74c9575700 1 -- start start 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.621+0000 7f74c9575700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c419bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.621+0000 7f74c9575700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f74c410c120 con 0x7f74c4108830 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.621+0000 7f74c2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c419bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.621+0000 7f74c2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c419bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50416/0 (socket says 172.21.15.27:50416) 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.621+0000 7f74c2ffd700 1 -- 172.21.15.27:0/203008430 learned_addr learned my addr 172.21.15.27:0/203008430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.622+0000 7f74c2ffd700 1 -- 172.21.15.27:0/203008430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c419c1a0 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.622+0000 7f74c2ffd700 1 --2- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c419bc60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f74b4011670 tx=0x7f74b4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.622+0000 7f74abfff700 1 -- 172.21.15.27:0/203008430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74b40039c0 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.622+0000 7f74abfff700 1 -- 172.21.15.27:0/203008430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f74b4011e50 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.622+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c419c430 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.622+0000 7f74abfff700 1 -- 172.21.15.27:0/203008430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74b401a5c0 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.622+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c419f120 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.623+0000 7f74abfff700 1 -- 172.21.15.27:0/203008430 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f74b401a8c0 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.623+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c4063890 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.623+0000 7f74abfff700 1 --2- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f74ac039000 0x7f74ac03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.623+0000 7f74abfff700 1 -- 172.21.15.27:0/203008430 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f74b404d5b0 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.623+0000 7f74c27fc700 1 --2- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f74ac039000 0x7f74ac03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.624+0000 7f74c27fc700 1 --2- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f74ac039000 0x7f74ac03b4c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f74b8006fd0 tx=0x7f74b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.628+0000 7f74abfff700 1 -- 172.21.15.27:0/203008430 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f74b402d910 con 0x7f74c4108830 2024-04-25T01:15:23.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.838+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f74c4103350 con 0x7f74ac039000 2024-04-25T01:15:23.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.972+0000 7f74abfff700 1 -- 172.21.15.27:0/203008430 <== mgr.14120 v2:172.21.15.27:6800/3353153293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f74c4103350 con 0x7f74ac039000 2024-04-25T01:15:23.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f74ac039000 msgr2=0x7f74ac03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:23.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 --2- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f74ac039000 0x7f74ac03b4c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f74b8006fd0 tx=0x7f74b8006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:23.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 msgr2=0x7f74c419bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:23.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 --2- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c419bc60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f74b4011670 tx=0x7f74b4011b60 comp rx=0 tx=0).stop 2024-04-25T01:15:23.537 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 shutdown_connections 2024-04-25T01:15:23.538 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 --2- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f74ac039000 0x7f74ac03b4c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:23.538 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 --2- 172.21.15.27:0/203008430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c4108830 0x7f74c419bc60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:23.538 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 >> 172.21.15.27:0/203008430 conn(0x7f74c41021e0 msgr2=0x7f74c4102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:23.538 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 shutdown_connections 2024-04-25T01:15:23.538 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:22.975+0000 7f74c9575700 1 -- 172.21.15.27:0/203008430 wait complete. 2024-04-25T01:15:24.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:23 smithi027 ceph-mon[29197]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:24.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:23 smithi027 ceph-mon[29197]: Generating ssh key... 2024-04-25T01:15:24.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:23 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:24.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:23 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:24.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8c9n5alAJq5XI4/hLZkSZ/5JzC95QmVd+bmP60d/cnqfpCxZu8JlMfIPhTYH8qS+svozGlAz69wcPKWlZ0jUf4IyNbljQzh/98iGJOT7LaUL4li8UgVlyp/+4l9+McG2Y7UGOHiollFga7i/AO0tK6vNa0wU1YfO68WvUP1ro62hfo+hLDz74uzYbf2LKXI5EdNI0v5a4Li5VPbwMBQpsMRuD3g/QGPsJe8648T5R3sSvkvMOXkpooqNYCybQ0VvQ6m8zSKx8+vvZrRNT0g9UAIFKnd2gyCl2HZGZIFPzGcm54sR2OA0+VILT3tN4Jb1HHp+Che7nzqEZljqNC7Q4GYwsXhyU1S/xoXtf5WUxUTU9ka8hcZzEFDAlSxtX5r92kkxNILqFH+Kts3szTlj6zNSPPY9OKD6QsYGtdM4s+sibABQ4JpX3qqFk5dK0/Exffi/yt4nN/cbVVmKbghjf0nDmonUFQTRln+tkXaSdehB4nIQZubd62prc4zEAoAE= ceph-1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:15:24.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.963+0000 7ffa3cb71700 1 Processor -- start 2024-04-25T01:15:24.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.963+0000 7ffa3cb71700 1 -- start start 2024-04-25T01:15:24.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3cb71700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa3810ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:24.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3cb71700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3807be50 con 0x7ffa38108780 2024-04-25T01:15:24.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa3810ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:24.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa3810ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50420/0 (socket says 172.21.15.27:50420) 2024-04-25T01:15:24.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3659c700 1 -- 172.21.15.27:0/3249724598 learned_addr learned my addr 172.21.15.27:0/3249724598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3659c700 1 -- 172.21.15.27:0/3249724598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3810b0b0 con 0x7ffa38108780 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3659c700 1 --2- 172.21.15.27:0/3249724598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa3810ab70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009e80 tx=0x7ffa3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8843aeee8616f3c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3559a700 1 -- 172.21.15.27:0/3249724598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa30017070 con 0x7ffa38108780 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3559a700 1 -- 172.21.15.27:0/3249724598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ffa3000b8c0 con 0x7ffa38108780 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.964+0000 7ffa3559a700 1 -- 172.21.15.27:0/3249724598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa300045c0 con 0x7ffa38108780 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.965+0000 7ffa3cb71700 1 -- 172.21.15.27:0/3249724598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 msgr2=0x7ffa3810ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.965+0000 7ffa3cb71700 1 --2- 172.21.15.27:0/3249724598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa3810ab70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009e80 tx=0x7ffa3000b330 comp rx=0 tx=0).stop 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.965+0000 7ffa3cb71700 1 -- 172.21.15.27:0/3249724598 shutdown_connections 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.965+0000 7ffa3cb71700 1 --2- 172.21.15.27:0/3249724598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa3810ab70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.965+0000 7ffa3cb71700 1 -- 172.21.15.27:0/3249724598 >> 172.21.15.27:0/3249724598 conn(0x7ffa381021d0 msgr2=0x7ffa381045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.965+0000 7ffa3cb71700 1 -- 172.21.15.27:0/3249724598 shutdown_connections 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.965+0000 7ffa3cb71700 1 -- 172.21.15.27:0/3249724598 wait complete. 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.966+0000 7ffa3cb71700 1 Processor -- start 2024-04-25T01:15:24.731 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.966+0000 7ffa3cb71700 1 -- start start 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.966+0000 7ffa3cb71700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa38108000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.966+0000 7ffa3cb71700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3810c0e0 con 0x7ffa38108780 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.966+0000 7ffa3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa38108000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.966+0000 7ffa3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa38108000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50428/0 (socket says 172.21.15.27:50428) 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.966+0000 7ffa3659c700 1 -- 172.21.15.27:0/1824611569 learned_addr learned my addr 172.21.15.27:0/1824611569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.967+0000 7ffa3659c700 1 -- 172.21.15.27:0/1824611569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa38108540 con 0x7ffa38108780 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.967+0000 7ffa3659c700 1 --2- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa38108000 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffa30004350 tx=0x7ffa3000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.967+0000 7ffa2f7fe700 1 -- 172.21.15.27:0/1824611569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa30020020 con 0x7ffa38108780 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.967+0000 7ffa2f7fe700 1 -- 172.21.15.27:0/1824611569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ffa30003c10 con 0x7ffa38108780 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.967+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa38106730 con 0x7ffa38108780 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.967+0000 7ffa2f7fe700 1 -- 172.21.15.27:0/1824611569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa30017040 con 0x7ffa38108780 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.967+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa38106c70 con 0x7ffa38108780 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.968+0000 7ffa2f7fe700 1 -- 172.21.15.27:0/1824611569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7ffa30023020 con 0x7ffa38108780 2024-04-25T01:15:24.732 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.968+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa38063890 con 0x7ffa38108780 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.968+0000 7ffa2f7fe700 1 --2- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7ffa24039050 0x7ffa2403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.968+0000 7ffa2f7fe700 1 -- 172.21.15.27:0/1824611569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ffa3000f050 con 0x7ffa38108780 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.968+0000 7ffa35d9b700 1 --2- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7ffa24039050 0x7ffa2403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.969+0000 7ffa35d9b700 1 --2- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7ffa24039050 0x7ffa2403b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa20006fd0 tx=0x7ffa20006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:23.973+0000 7ffa2f7fe700 1 -- 172.21.15.27:0/1824611569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffa3002a020 con 0x7ffa38108780 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.183+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7ffa381033d0 con 0x7ffa24039050 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.183+0000 7ffa2f7fe700 1 -- 172.21.15.27:0/1824611569 <== mgr.14120 v2:172.21.15.27:6800/3353153293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7ffa381033d0 con 0x7ffa24039050 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.185+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7ffa24039050 msgr2=0x7ffa2403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.185+0000 7ffa3cb71700 1 --2- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7ffa24039050 0x7ffa2403b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa20006fd0 tx=0x7ffa20006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.185+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 msgr2=0x7ffa38108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.186+0000 7ffa3cb71700 1 --2- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa38108000 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffa30004350 tx=0x7ffa3000bd50 comp rx=0 tx=0).stop 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.186+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 shutdown_connections 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.186+0000 7ffa3cb71700 1 --2- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7ffa24039050 0x7ffa2403b510 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.186+0000 7ffa3cb71700 1 --2- 172.21.15.27:0/1824611569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa38108780 0x7ffa38108000 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:24.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.186+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 >> 172.21.15.27:0/1824611569 conn(0x7ffa381021d0 msgr2=0x7ffa38102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:24.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.186+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 shutdown_connections 2024-04-25T01:15:24.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:24.186+0000 7ffa3cb71700 1 -- 172.21.15.27:0/1824611569 wait complete. 2024-04-25T01:15:24.734 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:15:24.734 INFO:teuthology.orchestra.run.smithi027.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T01:15:24.734 INFO:teuthology.orchestra.run.smithi027.stdout:Adding host smithi027... 2024-04-25T01:15:25.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:25 smithi027 ceph-mon[29197]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:26.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:26 smithi027 ceph-mon[29197]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:26.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:26 smithi027 ceph-mon[29197]: Deploying cephadm binary to smithi027 2024-04-25T01:15:30.163 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Added host 'smithi027' with addr '172.21.15.27' 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.156+0000 7f0179707700 1 Processor -- start 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0179707700 1 -- start start 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0179707700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017410abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0179707700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f017407bdf0 con 0x7f01741087e0 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0172ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017410abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0172ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017410abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50436/0 (socket says 172.21.15.27:50436) 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0172ffd700 1 -- 172.21.15.27:0/179633438 learned_addr learned my addr 172.21.15.27:0/179633438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0172ffd700 1 -- 172.21.15.27:0/179633438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017410b110 con 0x7f01741087e0 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0172ffd700 1 --2- 172.21.15.27:0/179633438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017410abd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f01640098a0 tx=0x7f0164009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae771f9f63a9ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0171ffb700 1 -- 172.21.15.27:0/179633438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0164004160 con 0x7f01741087e0 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0171ffb700 1 -- 172.21.15.27:0/179633438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0164004300 con 0x7f01741087e0 2024-04-25T01:15:30.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.157+0000 7f0171ffb700 1 -- 172.21.15.27:0/179633438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0164011b50 con 0x7f01741087e0 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.158+0000 7f0179707700 1 -- 172.21.15.27:0/179633438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 msgr2=0x7f017410abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.158+0000 7f0179707700 1 --2- 172.21.15.27:0/179633438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017410abd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f01640098a0 tx=0x7f0164009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.158+0000 7f0179707700 1 -- 172.21.15.27:0/179633438 shutdown_connections 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.158+0000 7f0179707700 1 --2- 172.21.15.27:0/179633438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017410abd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.158+0000 7f0179707700 1 -- 172.21.15.27:0/179633438 >> 172.21.15.27:0/179633438 conn(0x7f01741021d0 msgr2=0x7f01741045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.158+0000 7f0179707700 1 -- 172.21.15.27:0/179633438 shutdown_connections 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.158+0000 7f0179707700 1 -- 172.21.15.27:0/179633438 wait complete. 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.159+0000 7f0179707700 1 Processor -- start 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.159+0000 7f0179707700 1 -- start start 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.159+0000 7f0179707700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.159+0000 7f0179707700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f017410c0e0 con 0x7f01741087e0 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.159+0000 7f0172ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.159+0000 7f0172ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017419bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50444/0 (socket says 172.21.15.27:50444) 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.159+0000 7f0172ffd700 1 -- 172.21.15.27:0/88876207 learned_addr learned my addr 172.21.15.27:0/88876207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:30.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.159+0000 7f0172ffd700 1 -- 172.21.15.27:0/88876207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017419c1c0 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.160+0000 7f0172ffd700 1 --2- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017419bc80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0164011b60 tx=0x7f0164011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.160+0000 7f016bfff700 1 -- 172.21.15.27:0/88876207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0164003b20 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.160+0000 7f016bfff700 1 -- 172.21.15.27:0/88876207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0164003cc0 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.160+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017419c450 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.160+0000 7f016bfff700 1 -- 172.21.15.27:0/88876207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0164024840 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.160+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f017419f140 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.161+0000 7f016bfff700 1 -- 172.21.15.27:0/88876207 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f0164022020 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.161+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0174063890 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.161+0000 7f016bfff700 1 --2- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f0160039050 0x7f016003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.161+0000 7f016bfff700 1 -- 172.21.15.27:0/88876207 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0164011e00 con 0x7f01741087e0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.161+0000 7f01727fc700 1 --2- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f0160039050 0x7f016003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.161+0000 7f01727fc700 1 --2- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f0160039050 0x7f016003b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f015c006fd0 tx=0x7f015c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:30.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.166+0000 7f016bfff700 1 -- 172.21.15.27:0/88876207 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f016400be30 con 0x7f01741087e0 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:25.375+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}) -- 0x7f01741033d0 con 0x7f0160039050 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.628+0000 7f016bfff700 1 -- 172.21.15.27:0/88876207 <== mgr.14120 v2:172.21.15.27:6800/3353153293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f01741033d0 con 0x7f0160039050 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.630+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f0160039050 msgr2=0x7f016003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.630+0000 7f0179707700 1 --2- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f0160039050 0x7f016003b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f015c006fd0 tx=0x7f015c006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.630+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 msgr2=0x7f017419bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.630+0000 7f0179707700 1 --2- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017419bc80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0164011b60 tx=0x7f0164011c40 comp rx=0 tx=0).stop 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.630+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 shutdown_connections 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.630+0000 7f0179707700 1 --2- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f0160039050 0x7f016003b510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.630+0000 7f0179707700 1 --2- 172.21.15.27:0/88876207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01741087e0 0x7f017419bc80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.630+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 >> 172.21.15.27:0/88876207 conn(0x7f01741021d0 msgr2=0x7f0174102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.631+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 shutdown_connections 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:29.631+0000 7f0179707700 1 -- 172.21.15.27:0/88876207 wait complete. 2024-04-25T01:15:30.167 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mon service... 2024-04-25T01:15:30.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:30 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:30.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:30 smithi027 ceph-mon[29197]: Added host smithi027 2024-04-25T01:15:30.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:30 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:15:31.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T01:15:31.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.671+0000 7f55617a6700 1 Processor -- start 2024-04-25T01:15:31.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.671+0000 7f55617a6700 1 -- start start 2024-04-25T01:15:31.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.672+0000 7f55617a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c1085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:31.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.672+0000 7f55617a6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f555c108ba0 con 0x7f555c1081b0 2024-04-25T01:15:31.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.672+0000 7f555affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c1085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.672+0000 7f555affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c1085d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54324/0 (socket says 172.21.15.27:54324) 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.672+0000 7f555affd700 1 -- 172.21.15.27:0/176097561 learned_addr learned my addr 172.21.15.27:0/176097561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.672+0000 7f555affd700 1 -- 172.21.15.27:0/176097561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555c106e10 con 0x7f555c1081b0 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.672+0000 7f555affd700 1 --2- 172.21.15.27:0/176097561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c1085d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f554c00bb40 tx=0x7f554c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ed4d0052ffd6a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f5559ffb700 1 -- 172.21.15.27:0/176097561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f554c00dbe0 con 0x7f555c1081b0 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f5559ffb700 1 -- 172.21.15.27:0/176097561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f554c0044f0 con 0x7f555c1081b0 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f5559ffb700 1 -- 172.21.15.27:0/176097561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f554c010470 con 0x7f555c1081b0 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f55617a6700 1 -- 172.21.15.27:0/176097561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 msgr2=0x7f555c1085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f55617a6700 1 --2- 172.21.15.27:0/176097561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c1085d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f554c00bb40 tx=0x7f554c00be50 comp rx=0 tx=0).stop 2024-04-25T01:15:31.459 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f55617a6700 1 -- 172.21.15.27:0/176097561 shutdown_connections 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f55617a6700 1 --2- 172.21.15.27:0/176097561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c1085d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f55617a6700 1 -- 172.21.15.27:0/176097561 >> 172.21.15.27:0/176097561 conn(0x7f555c101940 msgr2=0x7f555c103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f55617a6700 1 -- 172.21.15.27:0/176097561 shutdown_connections 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.673+0000 7f55617a6700 1 -- 172.21.15.27:0/176097561 wait complete. 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.674+0000 7f55617a6700 1 Processor -- start 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.674+0000 7f55617a6700 1 -- start start 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.674+0000 7f55617a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.674+0000 7f55617a6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f555c105d00 con 0x7f555c1081b0 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.674+0000 7f555affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.674+0000 7f555affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54334/0 (socket says 172.21.15.27:54334) 2024-04-25T01:15:31.460 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.674+0000 7f555affd700 1 -- 172.21.15.27:0/3305273488 learned_addr learned my addr 172.21.15.27:0/3305273488 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.675+0000 7f555affd700 1 -- 172.21.15.27:0/3305273488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555c199fc0 con 0x7f555c1081b0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.675+0000 7f555affd700 1 --2- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c199a80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f554c00b340 tx=0x7f554c0040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.675+0000 7f5553fff700 1 -- 172.21.15.27:0/3305273488 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f554c027070 con 0x7f555c1081b0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.675+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f555c19a250 con 0x7f555c1081b0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.675+0000 7f5553fff700 1 -- 172.21.15.27:0/3305273488 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f554c010be0 con 0x7f555c1081b0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.675+0000 7f5553fff700 1 -- 172.21.15.27:0/3305273488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f554c022c20 con 0x7f555c1081b0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.675+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f555c19cef0 con 0x7f555c1081b0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.676+0000 7f5553fff700 1 -- 172.21.15.27:0/3305273488 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f554c02a070 con 0x7f555c1081b0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.676+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f555c063890 con 0x7f555c1081b0 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.676+0000 7f5553fff700 1 --2- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5548039000 0x7f554803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:31.461 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.676+0000 7f5553fff700 1 -- 172.21.15.27:0/3305273488 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f554c01f070 con 0x7f555c1081b0 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.676+0000 7f555a7fc700 1 --2- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5548039000 0x7f554803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.677+0000 7f555a7fc700 1 --2- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5548039000 0x7f554803b4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5544006fd0 tx=0x7f5544006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.681+0000 7f5553fff700 1 -- 172.21.15.27:0/3305273488 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f554c015070 con 0x7f555c1081b0 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.911+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f555c062670 con 0x7f5548039000 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.916+0000 7f5553fff700 1 -- 172.21.15.27:0/3305273488 <== mgr.14120 v2:172.21.15.27:6800/3353153293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f555c062670 con 0x7f5548039000 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5548039000 msgr2=0x7f554803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 --2- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5548039000 0x7f554803b4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5544006fd0 tx=0x7f5544006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 msgr2=0x7f555c199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 --2- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c199a80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f554c00b340 tx=0x7f554c0040b0 comp rx=0 tx=0).stop 2024-04-25T01:15:31.462 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 shutdown_connections 2024-04-25T01:15:31.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 --2- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5548039000 0x7f554803b4c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:31.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 --2- 172.21.15.27:0/3305273488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c1081b0 0x7f555c199a80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:31.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 >> 172.21.15.27:0/3305273488 conn(0x7f555c101940 msgr2=0x7f555c1900a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:31.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 shutdown_connections 2024-04-25T01:15:31.463 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:30.918+0000 7f55617a6700 1 -- 172.21.15.27:0/3305273488 wait complete. 2024-04-25T01:15:31.463 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mgr service... 2024-04-25T01:15:32.070 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:31 smithi027 ceph-mon[29197]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:32.071 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:31 smithi027 ceph-mon[29197]: Saving service mon spec with placement count:5 2024-04-25T01:15:32.071 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:31 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.188+0000 7f5c84dbb700 1 Processor -- start 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.188+0000 7f5c84dbb700 1 -- start start 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.189+0000 7f5c84dbb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c8010abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.189+0000 7f5c84dbb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8007be10 con 0x7f5c801087c0 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.189+0000 7f5c7e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c8010abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.189+0000 7f5c7e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c8010abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54336/0 (socket says 172.21.15.27:54336) 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.189+0000 7f5c7e59c700 1 -- 172.21.15.27:0/3613611693 learned_addr learned my addr 172.21.15.27:0/3613611693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.189+0000 7f5c7e59c700 1 -- 172.21.15.27:0/3613611693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8010b0f0 con 0x7f5c801087c0 2024-04-25T01:15:33.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.189+0000 7f5c7e59c700 1 --2- 172.21.15.27:0/3613611693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c8010abb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c700098a0 tx=0x7f5c70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e97455ba05b0fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c7d59a700 1 -- 172.21.15.27:0/3613611693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c70004160 con 0x7f5c801087c0 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c7d59a700 1 -- 172.21.15.27:0/3613611693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5c70004300 con 0x7f5c801087c0 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c7d59a700 1 -- 172.21.15.27:0/3613611693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c70011b50 con 0x7f5c801087c0 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c84dbb700 1 -- 172.21.15.27:0/3613611693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 msgr2=0x7f5c8010abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c84dbb700 1 --2- 172.21.15.27:0/3613611693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c8010abb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c700098a0 tx=0x7f5c70009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c84dbb700 1 -- 172.21.15.27:0/3613611693 shutdown_connections 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c84dbb700 1 --2- 172.21.15.27:0/3613611693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c8010abb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c84dbb700 1 -- 172.21.15.27:0/3613611693 >> 172.21.15.27:0/3613611693 conn(0x7f5c801021d0 msgr2=0x7f5c801045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c84dbb700 1 -- 172.21.15.27:0/3613611693 shutdown_connections 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.190+0000 7f5c84dbb700 1 -- 172.21.15.27:0/3613611693 wait complete. 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.191+0000 7f5c84dbb700 1 Processor -- start 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.191+0000 7f5c84dbb700 1 -- start start 2024-04-25T01:15:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.191+0000 7f5c84dbb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c80108040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.191+0000 7f5c84dbb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8010c110 con 0x7f5c801087c0 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.191+0000 7f5c7e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c80108040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.191+0000 7f5c7e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c80108040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54340/0 (socket says 172.21.15.27:54340) 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.191+0000 7f5c7e59c700 1 -- 172.21.15.27:0/1931623210 learned_addr learned my addr 172.21.15.27:0/1931623210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.192+0000 7f5c7e59c700 1 -- 172.21.15.27:0/1931623210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c80108580 con 0x7f5c801087c0 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.192+0000 7f5c7e59c700 1 --2- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c80108040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5c70011670 tx=0x7f5c70011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.192+0000 7f5c777fe700 1 -- 172.21.15.27:0/1931623210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c700039c0 con 0x7f5c801087c0 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.192+0000 7f5c777fe700 1 -- 172.21.15.27:0/1931623210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5c70011e50 con 0x7f5c801087c0 2024-04-25T01:15:33.206 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.192+0000 7f5c777fe700 1 -- 172.21.15.27:0/1931623210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c7001a5c0 con 0x7f5c801087c0 2024-04-25T01:15:33.207 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.192+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c80106770 con 0x7f5c801087c0 2024-04-25T01:15:33.207 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.192+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c80106cb0 con 0x7f5c801087c0 2024-04-25T01:15:33.207 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.193+0000 7f5c777fe700 1 -- 172.21.15.27:0/1931623210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f5c70024430 con 0x7f5c801087c0 2024-04-25T01:15:33.207 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.193+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c80063890 con 0x7f5c801087c0 2024-04-25T01:15:33.207 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.193+0000 7f5c777fe700 1 --2- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5c6c0390b0 0x7f5c6c03b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:33.207 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.193+0000 7f5c777fe700 1 -- 172.21.15.27:0/1931623210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5c7004d3d0 con 0x7f5c801087c0 2024-04-25T01:15:33.207 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.194+0000 7f5c7dd9b700 1 --2- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5c6c0390b0 0x7f5c6c03b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:33.207 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.194+0000 7f5c7dd9b700 1 --2- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5c6c0390b0 0x7f5c6c03b570 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5c68006fd0 tx=0x7f5c68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.199+0000 7f5c777fe700 1 -- 172.21.15.27:0/1931623210 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5c7002d910 con 0x7f5c801087c0 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.418+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5c80107510 con 0x7f5c6c0390b0 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.425+0000 7f5c777fe700 1 -- 172.21.15.27:0/1931623210 <== mgr.14120 v2:172.21.15.27:6800/3353153293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5c80107510 con 0x7f5c6c0390b0 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5c6c0390b0 msgr2=0x7f5c6c03b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 --2- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5c6c0390b0 0x7f5c6c03b570 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5c68006fd0 tx=0x7f5c68006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 msgr2=0x7f5c80108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 --2- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c80108040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5c70011670 tx=0x7f5c70011b60 comp rx=0 tx=0).stop 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 shutdown_connections 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 --2- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f5c6c0390b0 0x7f5c6c03b570 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 --2- 172.21.15.27:0/1931623210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c801087c0 0x7f5c80108040 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 >> 172.21.15.27:0/1931623210 conn(0x7f5c801021d0 msgr2=0x7f5c80102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 shutdown_connections 2024-04-25T01:15:33.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:32.427+0000 7f5c84dbb700 1 -- 172.21.15.27:0/1931623210 wait complete. 2024-04-25T01:15:33.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:33 smithi027 ceph-mon[29197]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:33.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:33 smithi027 ceph-mon[29197]: Saving service mgr spec with placement count:2 2024-04-25T01:15:33.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:33 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:33.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:33 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:33.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:33 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:34.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.651+0000 7f0513498700 1 Processor -- start 2024-04-25T01:15:34.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.651+0000 7f0513498700 1 -- start start 2024-04-25T01:15:34.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.652+0000 7f0513498700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:34.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.652+0000 7f0513498700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f050c07be50 con 0x7f050c108760 2024-04-25T01:15:34.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.652+0000 7f0511234700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:34.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.652+0000 7f0511234700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54350/0 (socket says 172.21.15.27:54350) 2024-04-25T01:15:34.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.652+0000 7f0511234700 1 -- 172.21.15.27:0/3360841611 learned_addr learned my addr 172.21.15.27:0/3360841611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.652+0000 7f0511234700 1 -- 172.21.15.27:0/3360841611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c10b090 con 0x7f050c108760 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.652+0000 7f0511234700 1 --2- 172.21.15.27:0/3360841611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c10ab50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f05000098a0 tx=0x7f0500009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6440a18a0d5bca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f04fffff700 1 -- 172.21.15.27:0/3360841611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0500004160 con 0x7f050c108760 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f04fffff700 1 -- 172.21.15.27:0/3360841611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0500004300 con 0x7f050c108760 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f04fffff700 1 -- 172.21.15.27:0/3360841611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0500011b20 con 0x7f050c108760 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f0513498700 1 -- 172.21.15.27:0/3360841611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 msgr2=0x7f050c10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f0513498700 1 --2- 172.21.15.27:0/3360841611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c10ab50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f05000098a0 tx=0x7f0500009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f0513498700 1 -- 172.21.15.27:0/3360841611 shutdown_connections 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f0513498700 1 --2- 172.21.15.27:0/3360841611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c10ab50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f0513498700 1 -- 172.21.15.27:0/3360841611 >> 172.21.15.27:0/3360841611 conn(0x7f050c102170 msgr2=0x7f050c1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.653+0000 7f0513498700 1 -- 172.21.15.27:0/3360841611 shutdown_connections 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.654+0000 7f0513498700 1 -- 172.21.15.27:0/3360841611 wait complete. 2024-04-25T01:15:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.654+0000 7f0513498700 1 Processor -- start 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.654+0000 7f0513498700 1 -- start start 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.654+0000 7f0513498700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c19bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.654+0000 7f0513498700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f050c10c100 con 0x7f050c108760 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.654+0000 7f0511234700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c19bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.654+0000 7f0511234700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c19bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54358/0 (socket says 172.21.15.27:54358) 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.654+0000 7f0511234700 1 -- 172.21.15.27:0/31076209 learned_addr learned my addr 172.21.15.27:0/31076209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.655+0000 7f0511234700 1 -- 172.21.15.27:0/31076209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c19c160 con 0x7f050c108760 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.655+0000 7f0511234700 1 --2- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c19bc20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0500011b60 tx=0x7f0500011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.655+0000 7f04fe7fc700 1 -- 172.21.15.27:0/31076209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0500003b60 con 0x7f050c108760 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.655+0000 7f04fe7fc700 1 -- 172.21.15.27:0/31076209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f050001a910 con 0x7f050c108760 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.655+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050c19c3f0 con 0x7f050c108760 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.655+0000 7f04fe7fc700 1 -- 172.21.15.27:0/31076209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0500024930 con 0x7f050c108760 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.655+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f050c19c850 con 0x7f050c108760 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.656+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f050c063890 con 0x7f050c108760 2024-04-25T01:15:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.656+0000 7f04fe7fc700 1 -- 172.21.15.27:0/31076209 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f0500022020 con 0x7f050c108760 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.656+0000 7f04fe7fc700 1 --2- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f04f80390a0 0x7f04f803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.657+0000 7f04fe7fc700 1 -- 172.21.15.27:0/31076209 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f050004d390 con 0x7f050c108760 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.657+0000 7f0510a33700 1 --2- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f04f80390a0 0x7f04f803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.657+0000 7f0510a33700 1 --2- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f04f80390a0 0x7f04f803b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0508006fd0 tx=0x7f0508006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.661+0000 7f04fe7fc700 1 -- 172.21.15.27:0/31076209 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f050001ab30 con 0x7f050c108760 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.882+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f050c19f0e0 con 0x7f050c108760 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.887+0000 7f04fe7fc700 1 -- 172.21.15.27:0/31076209 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f050000bc70 con 0x7f050c108760 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.890+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f04f80390a0 msgr2=0x7f04f803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.890+0000 7f0513498700 1 --2- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f04f80390a0 0x7f04f803b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0508006fd0 tx=0x7f0508006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.890+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 msgr2=0x7f050c19bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.890+0000 7f0513498700 1 --2- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c19bc20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0500011b60 tx=0x7f0500011c40 comp rx=0 tx=0).stop 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.890+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 shutdown_connections 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.890+0000 7f0513498700 1 --2- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f04f80390a0 0x7f04f803b560 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:34.512 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.890+0000 7f0513498700 1 --2- 172.21.15.27:0/31076209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f050c108760 0x7f050c19bc20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:34.513 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.890+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 >> 172.21.15.27:0/31076209 conn(0x7f050c102170 msgr2=0x7f050c102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:34.513 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.891+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 shutdown_connections 2024-04-25T01:15:34.513 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:33.891+0000 7f0513498700 1 -- 172.21.15.27:0/31076209 wait complete. 2024-04-25T01:15:35.162 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:34 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/31076209' entity='client.admin' 2024-04-25T01:15:36.392 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.464+0000 7fdc34f76700 1 Processor -- start 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.464+0000 7fdc34f76700 1 -- start start 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.464+0000 7fdc34f76700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc3010abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.464+0000 7fdc34f76700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3007be10 con 0x7fdc301087c0 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.464+0000 7fdc2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc3010abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.464+0000 7fdc2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc3010abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54372/0 (socket says 172.21.15.27:54372) 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.464+0000 7fdc2e59c700 1 -- 172.21.15.27:0/1123234577 learned_addr learned my addr 172.21.15.27:0/1123234577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.465+0000 7fdc2e59c700 1 -- 172.21.15.27:0/1123234577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3010b0f0 con 0x7fdc301087c0 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.465+0000 7fdc2e59c700 1 --2- 172.21.15.27:0/1123234577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc3010abb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc200098a0 tx=0x7fdc20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c722d45d62c056a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.465+0000 7fdc2d59a700 1 -- 172.21.15.27:0/1123234577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc20004160 con 0x7fdc301087c0 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.465+0000 7fdc2d59a700 1 -- 172.21.15.27:0/1123234577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc20004300 con 0x7fdc301087c0 2024-04-25T01:15:36.393 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.465+0000 7fdc2d59a700 1 -- 172.21.15.27:0/1123234577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc20011b20 con 0x7fdc301087c0 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.466+0000 7fdc34f76700 1 -- 172.21.15.27:0/1123234577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 msgr2=0x7fdc3010abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.466+0000 7fdc34f76700 1 --2- 172.21.15.27:0/1123234577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc3010abb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc200098a0 tx=0x7fdc20009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.466+0000 7fdc34f76700 1 -- 172.21.15.27:0/1123234577 shutdown_connections 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.466+0000 7fdc34f76700 1 --2- 172.21.15.27:0/1123234577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc3010abb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.466+0000 7fdc34f76700 1 -- 172.21.15.27:0/1123234577 >> 172.21.15.27:0/1123234577 conn(0x7fdc301021d0 msgr2=0x7fdc301045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.466+0000 7fdc34f76700 1 -- 172.21.15.27:0/1123234577 shutdown_connections 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.466+0000 7fdc34f76700 1 -- 172.21.15.27:0/1123234577 wait complete. 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.467+0000 7fdc34f76700 1 Processor -- start 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.467+0000 7fdc34f76700 1 -- start start 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.467+0000 7fdc34f76700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc30108040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.467+0000 7fdc34f76700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3010c150 con 0x7fdc301087c0 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.467+0000 7fdc2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc30108040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.467+0000 7fdc2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc30108040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54376/0 (socket says 172.21.15.27:54376) 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.468+0000 7fdc2e59c700 1 -- 172.21.15.27:0/1453116823 learned_addr learned my addr 172.21.15.27:0/1453116823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.468+0000 7fdc2e59c700 1 -- 172.21.15.27:0/1453116823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc30108580 con 0x7fdc301087c0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.468+0000 7fdc2e59c700 1 --2- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc30108040 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdc200117e0 tx=0x7fdc20011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.468+0000 7fdc1f7fe700 1 -- 172.21.15.27:0/1453116823 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc20003bf0 con 0x7fdc301087c0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.468+0000 7fdc1f7fe700 1 -- 172.21.15.27:0/1453116823 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc2001a910 con 0x7fdc301087c0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.468+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc30106770 con 0x7fdc301087c0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.468+0000 7fdc1f7fe700 1 -- 172.21.15.27:0/1453116823 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc20024970 con 0x7fdc301087c0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.468+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc30106c30 con 0x7fdc301087c0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.469+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc30063890 con 0x7fdc301087c0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.470+0000 7fdc1f7fe700 1 -- 172.21.15.27:0/1453116823 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fdc20022020 con 0x7fdc301087c0 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.470+0000 7fdc1f7fe700 1 --2- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fdc18039050 0x7fdc1803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:36.395 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.470+0000 7fdc1f7fe700 1 -- 172.21.15.27:0/1453116823 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdc2004d3d0 con 0x7fdc301087c0 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.470+0000 7fdc2dd9b700 1 --2- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fdc18039050 0x7fdc1803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.471+0000 7fdc2dd9b700 1 --2- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fdc18039050 0x7fdc1803b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdc24006fd0 tx=0x7fdc24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.475+0000 7fdc1f7fe700 1 -- 172.21.15.27:0/1453116823 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdc20011e20 con 0x7fdc301087c0 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.692+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fdc30106f20 con 0x7fdc301087c0 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.695+0000 7fdc1f7fe700 1 -- 172.21.15.27:0/1453116823 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fdc30106f20 con 0x7fdc301087c0 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fdc18039050 msgr2=0x7fdc1803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 --2- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fdc18039050 0x7fdc1803b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdc24006fd0 tx=0x7fdc24006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 msgr2=0x7fdc30108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 --2- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc30108040 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdc200117e0 tx=0x7fdc20011c10 comp rx=0 tx=0).stop 2024-04-25T01:15:36.396 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 shutdown_connections 2024-04-25T01:15:36.397 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 --2- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7fdc18039050 0x7fdc1803b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:36.397 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 --2- 172.21.15.27:0/1453116823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc301087c0 0x7fdc30108040 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:36.397 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 >> 172.21.15.27:0/1453116823 conn(0x7fdc301021d0 msgr2=0x7fdc30102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:36.397 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 shutdown_connections 2024-04-25T01:15:36.397 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:35.699+0000 7fdc34f76700 1 -- 172.21.15.27:0/1453116823 wait complete. 2024-04-25T01:15:36.397 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling the dashboard module... 2024-04-25T01:15:36.642 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:36 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:36.643 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:36 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1453116823' entity='client.admin' 2024-04-25T01:15:38.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:38 smithi027 ceph-mon[29197]: from='mgr.14120 172.21.15.27:0/3733905356' entity='mgr.x' 2024-04-25T01:15:38.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:38 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2892820229' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T01:15:38.450 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:38 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: ignoring --setuser ceph since I am not root 2024-04-25T01:15:38.450 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:38 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: ignoring --setgroup ceph since I am not root 2024-04-25T01:15:38.450 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:38 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:38.333+0000 7f3d52a90200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:15:38.707 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:38 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:38.525+0000 7f3d52a90200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:15:38.707 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:38 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:38.631+0000 7f3d52a90200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:15:39.025 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:38 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:38.951+0000 7f3d52a90200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:15:39.025 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:39 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:39.039+0000 7f3d52a90200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:15:39.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:39 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2892820229' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T01:15:39.341 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:39 smithi027 ceph-mon[29197]: mgrmap e9: x(active, since 20s) 2024-04-25T01:15:39.349 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.885+0000 7f5638f12700 1 Processor -- start 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.885+0000 7f5638f12700 1 -- start start 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.886+0000 7f5638f12700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f5634108140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.886+0000 7f5638f12700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5634108710 con 0x7f5634107d20 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.886+0000 7f563259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f5634108140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.886+0000 7f563259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f5634108140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54390/0 (socket says 172.21.15.27:54390) 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.886+0000 7f563259c700 1 -- 172.21.15.27:0/2793589440 learned_addr learned my addr 172.21.15.27:0/2793589440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.886+0000 7f563259c700 1 -- 172.21.15.27:0/2793589440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5634108ed0 con 0x7f5634107d20 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.886+0000 7f563259c700 1 --2- 172.21.15.27:0/2793589440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f5634108140 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f56280098a0 tx=0x7f5628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ac8e93bed85c8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:39.350 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.887+0000 7f563159a700 1 -- 172.21.15.27:0/2793589440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5628004160 con 0x7f5634107d20 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.887+0000 7f563159a700 1 -- 172.21.15.27:0/2793589440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5628004300 con 0x7f5634107d20 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.887+0000 7f563159a700 1 -- 172.21.15.27:0/2793589440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5628011b20 con 0x7f5634107d20 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.887+0000 7f5638f12700 1 -- 172.21.15.27:0/2793589440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 msgr2=0x7f5634108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.887+0000 7f5638f12700 1 --2- 172.21.15.27:0/2793589440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f5634108140 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f56280098a0 tx=0x7f5628009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.887+0000 7f5638f12700 1 -- 172.21.15.27:0/2793589440 shutdown_connections 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.887+0000 7f5638f12700 1 --2- 172.21.15.27:0/2793589440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f5634108140 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.887+0000 7f5638f12700 1 -- 172.21.15.27:0/2793589440 >> 172.21.15.27:0/2793589440 conn(0x7f5634103320 msgr2=0x7f5634105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.888+0000 7f5638f12700 1 -- 172.21.15.27:0/2793589440 shutdown_connections 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.888+0000 7f5638f12700 1 -- 172.21.15.27:0/2793589440 wait complete. 2024-04-25T01:15:39.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.888+0000 7f5638f12700 1 Processor -- start 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.888+0000 7f5638f12700 1 -- start start 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f5638f12700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f563419bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f5638f12700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f563410c280 con 0x7f5634107d20 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f563259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f563419bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f563259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f563419bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54402/0 (socket says 172.21.15.27:54402) 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f563259c700 1 -- 172.21.15.27:0/2892820229 learned_addr learned my addr 172.21.15.27:0/2892820229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f563259c700 1 -- 172.21.15.27:0/2892820229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563419c180 con 0x7f5634107d20 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f563259c700 1 --2- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f563419bc40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f56280117e0 tx=0x7f5628011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f56237fe700 1 -- 172.21.15.27:0/2892820229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5628003bf0 con 0x7f5634107d20 2024-04-25T01:15:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f56237fe700 1 -- 172.21.15.27:0/2892820229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562801a910 con 0x7f5634107d20 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f56237fe700 1 -- 172.21.15.27:0/2892820229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56280249a0 con 0x7f5634107d20 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563419c410 con 0x7f5634107d20 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.889+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f563419c870 con 0x7f5634107d20 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.890+0000 7f56237fe700 1 -- 172.21.15.27:0/2892820229 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f5628022020 con 0x7f5634107d20 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.890+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5634063890 con 0x7f5634107d20 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.891+0000 7f56237fe700 1 --2- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f561c039000 0x7f561c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.891+0000 7f56237fe700 1 -- 172.21.15.27:0/2892820229 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f562804d650 con 0x7f5634107d20 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.891+0000 7f5631d9b700 1 --2- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f561c039000 0x7f561c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.891+0000 7f5631d9b700 1 --2- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f561c039000 0x7f561c03b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:36.896+0000 7f56237fe700 1 -- 172.21.15.27:0/2892820229 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5628011e20 con 0x7f5634107d20 2024-04-25T01:15:39.353 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:37.178+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f563419f2e0 con 0x7f5634107d20 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.151+0000 7f56237fe700 1 -- 172.21.15.27:0/2892820229 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f56280172b0 con 0x7f5634107d20 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.153+0000 7f56237fe700 1 -- 172.21.15.27:0/2892820229 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f562802d910 con 0x7f5634107d20 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.154+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f561c039000 msgr2=0x7f561c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.154+0000 7f5638f12700 1 --2- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f561c039000 0x7f561c03b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.154+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 msgr2=0x7f563419bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.154+0000 7f5638f12700 1 --2- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f563419bc40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f56280117e0 tx=0x7f5628011c10 comp rx=0 tx=0).stop 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.155+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 shutdown_connections 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.155+0000 7f5638f12700 1 --2- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f561c039000 0x7f561c03b4c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.155+0000 7f5638f12700 1 --2- 172.21.15.27:0/2892820229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5634107d20 0x7f563419bc40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:39.354 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.155+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 >> 172.21.15.27:0/2892820229 conn(0x7f5634103320 msgr2=0x7f5634103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:39.355 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.155+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 shutdown_connections 2024-04-25T01:15:39.355 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:38.155+0000 7f5638f12700 1 -- 172.21.15.27:0/2892820229 wait complete. 2024-04-25T01:15:40.298 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:40 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3141639320' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:15:40.658 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:40 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:40.312+0000 7f3d52a90200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:15:40.658 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:40 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:40.495+0000 7f3d52a90200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:15:40.659 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:40 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:40.673+0000 7f3d52a90200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:15:40.665 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:15:40.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T01:15:40.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:15:40.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:15:40.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:15:40.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:15:40.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.817+0000 7f82f8513700 1 Processor -- start 2024-04-25T01:15:40.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.817+0000 7f82f8513700 1 -- start start 2024-04-25T01:15:40.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.817+0000 7f82f8513700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f010ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.817+0000 7f82f8513700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82f007be30 con 0x7f82f01087a0 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.818+0000 7f82f62af700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f010ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.818+0000 7f82f62af700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f010ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55726/0 (socket says 172.21.15.27:55726) 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.818+0000 7f82f62af700 1 -- 172.21.15.27:0/3892379371 learned_addr learned my addr 172.21.15.27:0/3892379371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.818+0000 7f82f62af700 1 -- 172.21.15.27:0/3892379371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f010b0d0 con 0x7f82f01087a0 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.818+0000 7f82f62af700 1 --2- 172.21.15.27:0/3892379371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f010ab90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82e00098a0 tx=0x7f82e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b7913ab66497d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f52ad700 1 -- 172.21.15.27:0/3892379371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82e0004160 con 0x7f82f01087a0 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f52ad700 1 -- 172.21.15.27:0/3892379371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e0004300 con 0x7f82f01087a0 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f52ad700 1 -- 172.21.15.27:0/3892379371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82e0011b20 con 0x7f82f01087a0 2024-04-25T01:15:40.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f8513700 1 -- 172.21.15.27:0/3892379371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 msgr2=0x7f82f010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:40.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f8513700 1 --2- 172.21.15.27:0/3892379371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f010ab90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82e00098a0 tx=0x7f82e0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:40.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f8513700 1 -- 172.21.15.27:0/3892379371 shutdown_connections 2024-04-25T01:15:40.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f8513700 1 --2- 172.21.15.27:0/3892379371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f010ab90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:40.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f8513700 1 -- 172.21.15.27:0/3892379371 >> 172.21.15.27:0/3892379371 conn(0x7f82f01021d0 msgr2=0x7f82f01045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:40.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f8513700 1 -- 172.21.15.27:0/3892379371 shutdown_connections 2024-04-25T01:15:40.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.819+0000 7f82f8513700 1 -- 172.21.15.27:0/3892379371 wait complete. 2024-04-25T01:15:40.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.820+0000 7f82f8513700 1 Processor -- start 2024-04-25T01:15:40.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.820+0000 7f82f8513700 1 -- start start 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.820+0000 7f82f8513700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f0108020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.820+0000 7f82f8513700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82f010c130 con 0x7f82f01087a0 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.820+0000 7f82f62af700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f0108020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.820+0000 7f82f62af700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f0108020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55734/0 (socket says 172.21.15.27:55734) 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.820+0000 7f82f62af700 1 -- 172.21.15.27:0/3141639320 learned_addr learned my addr 172.21.15.27:0/3141639320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.821+0000 7f82f62af700 1 -- 172.21.15.27:0/3141639320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f0108560 con 0x7f82f01087a0 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.821+0000 7f82f62af700 1 --2- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f0108020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f82e00117e0 tx=0x7f82e0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.821+0000 7f82e77fe700 1 -- 172.21.15.27:0/3141639320 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82e0003bf0 con 0x7f82f01087a0 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.821+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f0106750 con 0x7f82f01087a0 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.821+0000 7f82e77fe700 1 -- 172.21.15.27:0/3141639320 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e001a910 con 0x7f82f01087a0 2024-04-25T01:15:40.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.821+0000 7f82e77fe700 1 -- 172.21.15.27:0/3141639320 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82e0024970 con 0x7f82f01087a0 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.821+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f0106c10 con 0x7f82f01087a0 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.822+0000 7f82e77fe700 1 -- 172.21.15.27:0/3141639320 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f82e0022020 con 0x7f82f01087a0 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.822+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f0063890 con 0x7f82f01087a0 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.822+0000 7f82e77fe700 1 --2- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f82dc039050 0x7f82dc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.822+0000 7f82e77fe700 1 -- 172.21.15.27:0/3141639320 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f82e004d360 con 0x7f82f01087a0 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.822+0000 7f82f5aae700 1 -- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f82dc039050 msgr2=0x7f82dc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3353153293 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.822+0000 7f82f5aae700 1 --2- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f82dc039050 0x7f82dc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:39.827+0000 7f82e77fe700 1 -- 172.21.15.27:0/3141639320 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f82e0011e20 con 0x7f82f01087a0 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.023+0000 7f82f5aae700 1 -- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f82dc039050 msgr2=0x7f82dc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3353153293 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.023+0000 7f82f5aae700 1 --2- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f82dc039050 0x7f82dc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.094+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f82f0107580 con 0x7f82f01087a0 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.094+0000 7f82e77fe700 1 -- 172.21.15.27:0/3141639320 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f82e000b3e0 con 0x7f82f01087a0 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f82dc039050 msgr2=0x7f82dc03b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:15:40.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 --2- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f82dc039050 0x7f82dc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 msgr2=0x7f82f0108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 --2- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f0108020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f82e00117e0 tx=0x7f82e0011c10 comp rx=0 tx=0).stop 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 shutdown_connections 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 --2- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f82dc039050 0x7f82dc03b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 --2- 172.21.15.27:0/3141639320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f01087a0 0x7f82f0108020 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 >> 172.21.15.27:0/3141639320 conn(0x7f82f01021d0 msgr2=0x7f82f0102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 shutdown_connections 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:40.098+0000 7f82f8513700 1 -- 172.21.15.27:0/3141639320 wait complete. 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-04-25T01:15:40.678 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 9... 2024-04-25T01:15:41.282 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:40 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:40.985+0000 7f3d52a90200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:15:41.282 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:41 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:41.084+0000 7f3d52a90200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:15:41.282 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:41 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:41.296+0000 7f3d52a90200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:15:41.700 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:41 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:41.497+0000 7f3d52a90200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:15:42.450 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:42 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:42.125+0000 7f3d52a90200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:15:42.858 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:42 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:42.565+0000 7f3d52a90200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:15:42.858 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:42 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:42.665+0000 7f3d52a90200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:15:42.859 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:42 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:42.873+0000 7f3d52a90200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:15:43.700 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:43 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:43.418+0000 7f3d52a90200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:15:43.700 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:43 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:43.514+0000 7f3d52a90200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:15:44.104 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:43 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:43.765+0000 7f3d52a90200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:15:44.104 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:43 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:43.904+0000 7f3d52a90200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:15:44.104 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:44 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:44.002+0000 7f3d52a90200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:15:44.450 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:44 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:44.122+0000 7f3d52a90200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:15:44.450 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:44 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:44.218+0000 7f3d52a90200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:15:45.326 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:45.063+0000 7f3d52a90200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:15:45.326 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:15:45.285+0000 7f3d52a90200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:15:45.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: Active manager daemon x restarted 2024-04-25T01:15:45.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: Activating manager daemon x 2024-04-25T01:15:45.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T01:15:45.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: mgrmap e10: x(active, starting, since 0.0033138s) 2024-04-25T01:15:45.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:15:45.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:15:45.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:15:45.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:15:45.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:15:45.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: Manager daemon x is now available 2024-04-25T01:15:45.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:15:45.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:15:46.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:15:46.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:46 smithi027 ceph-mon[29197]: mgrmap e11: x(active, since 1.00627s) 2024-04-25T01:15:47.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:15:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T01:15:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:15:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:15:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.109+0000 7f9af7cdf700 1 Processor -- start 2024-04-25T01:15:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.109+0000 7f9af7cdf700 1 -- start start 2024-04-25T01:15:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.109+0000 7f9af7cdf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af010ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.109+0000 7f9af7cdf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9af007be10 con 0x7f9af01087a0 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.109+0000 7f9af5a7b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af010ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.109+0000 7f9af5a7b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af010ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55746/0 (socket says 172.21.15.27:55746) 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.109+0000 7f9af5a7b700 1 -- 172.21.15.27:0/3017253276 learned_addr learned my addr 172.21.15.27:0/3017253276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.109+0000 7f9af5a7b700 1 -- 172.21.15.27:0/3017253276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af010b0d0 con 0x7f9af01087a0 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.110+0000 7f9af5a7b700 1 --2- 172.21.15.27:0/3017253276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af010ab90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0098a0 tx=0x7f9aec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e9be54d9f848de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.110+0000 7f9af4a79700 1 -- 172.21.15.27:0/3017253276 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec004160 con 0x7f9af01087a0 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.110+0000 7f9af4a79700 1 -- 172.21.15.27:0/3017253276 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aec004300 con 0x7f9af01087a0 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.110+0000 7f9af4a79700 1 -- 172.21.15.27:0/3017253276 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec011b20 con 0x7f9af01087a0 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.110+0000 7f9af7cdf700 1 -- 172.21.15.27:0/3017253276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 msgr2=0x7f9af010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.111+0000 7f9af7cdf700 1 --2- 172.21.15.27:0/3017253276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af010ab90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0098a0 tx=0x7f9aec009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.111+0000 7f9af7cdf700 1 -- 172.21.15.27:0/3017253276 shutdown_connections 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.111+0000 7f9af7cdf700 1 --2- 172.21.15.27:0/3017253276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af010ab90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:47.359 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.111+0000 7f9af7cdf700 1 -- 172.21.15.27:0/3017253276 >> 172.21.15.27:0/3017253276 conn(0x7f9af0102170 msgr2=0x7f9af01045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.111+0000 7f9af7cdf700 1 -- 172.21.15.27:0/3017253276 shutdown_connections 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.111+0000 7f9af7cdf700 1 -- 172.21.15.27:0/3017253276 wait complete. 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.111+0000 7f9af7cdf700 1 Processor -- start 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.111+0000 7f9af7cdf700 1 -- start start 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af7cdf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af7cdf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9af010c130 con 0x7f9af01087a0 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af5a7b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af5a7b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af019bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55752/0 (socket says 172.21.15.27:55752) 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af5a7b700 1 -- 172.21.15.27:0/638300335 learned_addr learned my addr 172.21.15.27:0/638300335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af5a7b700 1 -- 172.21.15.27:0/638300335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af019c160 con 0x7f9af01087a0 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af5a7b700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af019bc20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0116a0 tx=0x7f9aec011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec003a40 con 0x7f9af01087a0 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aec003be0 con 0x7f9af01087a0 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af019c3f0 con 0x7f9af01087a0 2024-04-25T01:15:47.360 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec01a750 con 0x7f9af01087a0 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.112+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af019c850 con 0x7f9af01087a0 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.113+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f9aec01aa70 con 0x7f9af01087a0 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.113+0000 7f9ae6ffd700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 0x7f9adc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.113+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 --> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9af0063890 con 0x7f9adc039050 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.113+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9aec04dab0 con 0x7f9af01087a0 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.113+0000 7f9af527a700 1 -- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 msgr2=0x7f9adc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3353153293 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.114+0000 7f9af527a700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 0x7f9adc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.315+0000 7f9af527a700 1 -- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 msgr2=0x7f9adc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3353153293 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.315+0000 7f9af527a700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 0x7f9adc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.716+0000 7f9af527a700 1 -- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 msgr2=0x7f9adc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3353153293 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:41.716+0000 7f9af527a700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 0x7f9adc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:42.517+0000 7f9af527a700 1 -- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 msgr2=0x7f9adc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3353153293 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:42.517+0000 7f9af527a700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 0x7f9adc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:15:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:44.118+0000 7f9af527a700 1 -- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 msgr2=0x7f9adc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/3353153293 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:44.118+0000 7f9af527a700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 0x7f9adc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:45.289+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f9aec02d910 con 0x7f9af01087a0 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:45.289+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 msgr2=0x7f9adc03b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:45.289+0000 7f9ae6ffd700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 0x7f9adc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.291+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f9aec04f800 con 0x7f9af01087a0 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.291+0000 7f9ae6ffd700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9adc03c920 0x7f9adc03ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.291+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9aec022e80 con 0x7f9adc03c920 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.292+0000 7f9af527a700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9adc03c920 0x7f9adc03ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.292+0000 7f9af527a700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9adc03c920 0x7f9adc03ed10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0003a10 tx=0x7f9ae00092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.293+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f9aec022e80 con 0x7f9adc03c920 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9af0103300 con 0x7f9adc03c920 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9ae6ffd700 1 -- 172.21.15.27:0/638300335 <== mgr.14152 v2:172.21.15.27:6800/4052240654 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9af0103300 con 0x7f9adc03c920 2024-04-25T01:15:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9adc03c920 msgr2=0x7f9adc03ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9adc03c920 0x7f9adc03ed10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0003a10 tx=0x7f9ae00092b0 comp rx=0 tx=0).stop 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 msgr2=0x7f9af019bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af019bc20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0116a0 tx=0x7f9aec011ba0 comp rx=0 tx=0).stop 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 shutdown_connections 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9adc03c920 0x7f9adc03ed10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af01087a0 0x7f9af019bc20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 --2- 172.21.15.27:0/638300335 >> [v2:172.21.15.27:6800/3353153293,v1:172.21.15.27:6801/3353153293] conn(0x7f9adc039050 0x7f9adc03b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.300+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 >> 172.21.15.27:0/638300335 conn(0x7f9af0102170 msgr2=0x7f9af0102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.301+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 shutdown_connections 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:46.301+0000 7f9af7cdf700 1 -- 172.21.15.27:0/638300335 wait complete. 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 9 is available 2024-04-25T01:15:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:Generating a dashboard self-signed certificate... 2024-04-25T01:15:47.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:47 smithi027 ceph-mon[29197]: [25/Apr/2024:01:15:46] ENGINE Bus STARTING 2024-04-25T01:15:47.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:47 smithi027 ceph-mon[29197]: [25/Apr/2024:01:15:46] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T01:15:47.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:47 smithi027 ceph-mon[29197]: [25/Apr/2024:01:15:46] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T01:15:47.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:47 smithi027 ceph-mon[29197]: [25/Apr/2024:01:15:46] ENGINE Bus STARTED 2024-04-25T01:15:48.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:48 smithi027 ceph-mon[29197]: mgrmap e12: x(active, since 2s) 2024-04-25T01:15:48.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:48 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:15:48.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:48 smithi027 ceph-mon[29197]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:48.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:48 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:15:48.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:48 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:15:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T01:15:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.805+0000 7ff6069df700 1 Processor -- start 2024-04-25T01:15:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.805+0000 7ff6069df700 1 -- start start 2024-04-25T01:15:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.805+0000 7ff6069df700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff600108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.805+0000 7ff6069df700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6001086e0 con 0x7ff600107cf0 2024-04-25T01:15:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.805+0000 7ff5fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff600108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.805+0000 7ff5fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff600108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59688/0 (socket says 172.21.15.27:59688) 2024-04-25T01:15:48.696 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.805+0000 7ff5fffff700 1 -- 172.21.15.27:0/3541083403 learned_addr learned my addr 172.21.15.27:0/3541083403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.806+0000 7ff5fffff700 1 -- 172.21.15.27:0/3541083403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff600108ef0 con 0x7ff600107cf0 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.806+0000 7ff5fffff700 1 --2- 172.21.15.27:0/3541083403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff600108110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00098a0 tx=0x7ff5f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1193f08448d7a77c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.806+0000 7ff5feffd700 1 -- 172.21.15.27:0/3541083403 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5f0004160 con 0x7ff600107cf0 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.806+0000 7ff5feffd700 1 -- 172.21.15.27:0/3541083403 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5f0004300 con 0x7ff600107cf0 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.806+0000 7ff5feffd700 1 -- 172.21.15.27:0/3541083403 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5f0011b20 con 0x7ff600107cf0 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 -- 172.21.15.27:0/3541083403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 msgr2=0x7ff600108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 --2- 172.21.15.27:0/3541083403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff600108110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00098a0 tx=0x7ff5f0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 -- 172.21.15.27:0/3541083403 shutdown_connections 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 --2- 172.21.15.27:0/3541083403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff600108110 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 -- 172.21.15.27:0/3541083403 >> 172.21.15.27:0/3541083403 conn(0x7ff6001032d0 msgr2=0x7ff6001056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 -- 172.21.15.27:0/3541083403 shutdown_connections 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 -- 172.21.15.27:0/3541083403 wait complete. 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 Processor -- start 2024-04-25T01:15:48.697 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.807+0000 7ff6069df700 1 -- start start 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.808+0000 7ff6069df700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff60007db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.808+0000 7ff6069df700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff60010c2a0 con 0x7ff600107cf0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.808+0000 7ff5fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff60007db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.808+0000 7ff5fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff60007db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59696/0 (socket says 172.21.15.27:59696) 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.808+0000 7ff5fffff700 1 -- 172.21.15.27:0/139211015 learned_addr learned my addr 172.21.15.27:0/139211015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.808+0000 7ff5fffff700 1 -- 172.21.15.27:0/139211015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60007e050 con 0x7ff600107cf0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.808+0000 7ff5fffff700 1 --2- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff60007db10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00117e0 tx=0x7ff5f0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.809+0000 7ff5fd7fa700 1 -- 172.21.15.27:0/139211015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5f0003bf0 con 0x7ff600107cf0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.809+0000 7ff5fd7fa700 1 -- 172.21.15.27:0/139211015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5f001a910 con 0x7ff600107cf0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.809+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60007c240 con 0x7ff600107cf0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.809+0000 7ff5fd7fa700 1 -- 172.21.15.27:0/139211015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5f0024970 con 0x7ff600107cf0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.809+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff60007c700 con 0x7ff600107cf0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.810+0000 7ff5fd7fa700 1 -- 172.21.15.27:0/139211015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7ff5f0022020 con 0x7ff600107cf0 2024-04-25T01:15:48.698 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.810+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff600063890 con 0x7ff600107cf0 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.810+0000 7ff5fd7fa700 1 --2- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff5e80390a0 0x7ff5e803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.810+0000 7ff5fd7fa700 1 -- 172.21.15.27:0/139211015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff5f004d910 con 0x7ff600107cf0 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.810+0000 7ff5ff7fe700 1 --2- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff5e80390a0 0x7ff5e803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.811+0000 7ff5ff7fe700 1 --2- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff5e80390a0 0x7ff5e803b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4006fd0 tx=0x7ff5f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:47.816+0000 7ff5fd7fa700 1 -- 172.21.15.27:0/139211015 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff5f001ab70 con 0x7ff600107cf0 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.034+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7ff60007d010 con 0x7ff5e80390a0 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.104+0000 7ff5fd7fa700 1 -- 172.21.15.27:0/139211015 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7ff60007d010 con 0x7ff5e80390a0 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.106+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff5e80390a0 msgr2=0x7ff5e803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.106+0000 7ff6069df700 1 --2- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff5e80390a0 0x7ff5e803b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4006fd0 tx=0x7ff5f4006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.106+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 msgr2=0x7ff60007db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.106+0000 7ff6069df700 1 --2- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff60007db10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00117e0 tx=0x7ff5f0011ba0 comp rx=0 tx=0).stop 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.107+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 shutdown_connections 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.107+0000 7ff6069df700 1 --2- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff5e80390a0 0x7ff5e803b560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:48.699 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.107+0000 7ff6069df700 1 --2- 172.21.15.27:0/139211015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff600107cf0 0x7ff60007db10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:48.700 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.107+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 >> 172.21.15.27:0/139211015 conn(0x7ff6001032d0 msgr2=0x7ff600103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:48.700 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.107+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 shutdown_connections 2024-04-25T01:15:48.700 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:48.107+0000 7ff6069df700 1 -- 172.21.15.27:0/139211015 wait complete. 2024-04-25T01:15:48.700 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial admin user... 2024-04-25T01:15:50.294 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$UqPzZH8itI4cDFcLT8YlwORlsnRFRDG44ySVYVFtcTbu9av395/7C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714007749, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T01:15:50.294 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42b1f6b700 1 Processor -- start 2024-04-25T01:15:50.294 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42b1f6b700 1 -- start start 2024-04-25T01:15:50.294 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42b1f6b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:50.294 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42b1f6b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f42ac1086e0 con 0x7f42ac107cf0 2024-04-25T01:15:50.294 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:50.294 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59706/0 (socket says 172.21.15.27:59706) 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42ab7fe700 1 -- 172.21.15.27:0/2111350733 learned_addr learned my addr 172.21.15.27:0/2111350733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42ab7fe700 1 -- 172.21.15.27:0/2111350733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac108ef0 con 0x7f42ac107cf0 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.163+0000 7f42ab7fe700 1 --2- 172.21.15.27:0/2111350733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac108110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f429c0098a0 tx=0x7f429c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466fd27c15099ab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42aa7fc700 1 -- 172.21.15.27:0/2111350733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f429c004160 con 0x7f42ac107cf0 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42aa7fc700 1 -- 172.21.15.27:0/2111350733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f429c004300 con 0x7f42ac107cf0 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42aa7fc700 1 -- 172.21.15.27:0/2111350733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f429c011b20 con 0x7f42ac107cf0 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42b1f6b700 1 -- 172.21.15.27:0/2111350733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 msgr2=0x7f42ac108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42b1f6b700 1 --2- 172.21.15.27:0/2111350733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac108110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f429c0098a0 tx=0x7f429c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42b1f6b700 1 -- 172.21.15.27:0/2111350733 shutdown_connections 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42b1f6b700 1 --2- 172.21.15.27:0/2111350733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac108110 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42b1f6b700 1 -- 172.21.15.27:0/2111350733 >> 172.21.15.27:0/2111350733 conn(0x7f42ac1032d0 msgr2=0x7f42ac1056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42b1f6b700 1 -- 172.21.15.27:0/2111350733 shutdown_connections 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.164+0000 7f42b1f6b700 1 -- 172.21.15.27:0/2111350733 wait complete. 2024-04-25T01:15:50.295 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.165+0000 7f42b1f6b700 1 Processor -- start 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.165+0000 7f42b1f6b700 1 -- start start 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.165+0000 7f42b1f6b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac19bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.165+0000 7f42b1f6b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f42ac10c2a0 con 0x7f42ac107cf0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.165+0000 7f42ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac19bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.165+0000 7f42ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac19bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59716/0 (socket says 172.21.15.27:59716) 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.165+0000 7f42ab7fe700 1 -- 172.21.15.27:0/509082371 learned_addr learned my addr 172.21.15.27:0/509082371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.166+0000 7f42ab7fe700 1 -- 172.21.15.27:0/509082371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac19c140 con 0x7f42ac107cf0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.166+0000 7f42ab7fe700 1 --2- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac19bc00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f429c0117e0 tx=0x7f429c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.166+0000 7f42a8ff9700 1 -- 172.21.15.27:0/509082371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f429c003bf0 con 0x7f42ac107cf0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.166+0000 7f42a8ff9700 1 -- 172.21.15.27:0/509082371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f429c01a910 con 0x7f42ac107cf0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.166+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42ac19c3d0 con 0x7f42ac107cf0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.166+0000 7f42a8ff9700 1 -- 172.21.15.27:0/509082371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f429c0249a0 con 0x7f42ac107cf0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.166+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42ac19c830 con 0x7f42ac107cf0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.167+0000 7f42a8ff9700 1 -- 172.21.15.27:0/509082371 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f429c022020 con 0x7f42ac107cf0 2024-04-25T01:15:50.296 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.167+0000 7f42a8ff9700 1 --2- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42940390b0 0x7f429403b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.167+0000 7f42a8ff9700 1 -- 172.21.15.27:0/509082371 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f429c04d430 con 0x7f42ac107cf0 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.167+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42ac063890 con 0x7f42ac107cf0 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.167+0000 7f42aaffd700 1 --2- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42940390b0 0x7f429403b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.167+0000 7f42aaffd700 1 --2- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42940390b0 0x7f429403b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f42a0006fd0 tx=0x7f42a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.172+0000 7f42a8ff9700 1 -- 172.21.15.27:0/509082371 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f429c011e20 con 0x7f42ac107cf0 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.452+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f42ac104480 con 0x7f42940390b0 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.698+0000 7f42a8ff9700 1 -- 172.21.15.27:0/509082371 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f42ac104480 con 0x7f42940390b0 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.700+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42940390b0 msgr2=0x7f429403b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.700+0000 7f42b1f6b700 1 --2- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42940390b0 0x7f429403b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f42a0006fd0 tx=0x7f42a0006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.700+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 msgr2=0x7f42ac19bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.700+0000 7f42b1f6b700 1 --2- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac19bc00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f429c0117e0 tx=0x7f429c011c10 comp rx=0 tx=0).stop 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.701+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 shutdown_connections 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.701+0000 7f42b1f6b700 1 --2- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42940390b0 0x7f429403b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.701+0000 7f42b1f6b700 1 --2- 172.21.15.27:0/509082371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac107cf0 0x7f42ac19bc00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:50.297 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.701+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 >> 172.21.15.27:0/509082371 conn(0x7f42ac1032d0 msgr2=0x7f42ac103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:50.298 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.701+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 shutdown_connections 2024-04-25T01:15:50.298 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:49.701+0000 7f42b1f6b700 1 -- 172.21.15.27:0/509082371 wait complete. 2024-04-25T01:15:50.298 INFO:teuthology.orchestra.run.smithi027.stdout:Fetching dashboard port number... 2024-04-25T01:15:50.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:50 smithi027 ceph-mon[29197]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:15:50.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:15:50.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:15:51.928 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:51 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/632959425' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T01:15:52.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T01:15:52.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.785+0000 7fa6496b3700 1 Processor -- start 2024-04-25T01:15:52.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.785+0000 7fa6496b3700 1 -- start start 2024-04-25T01:15:52.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.785+0000 7fa6496b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa644108610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:52.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.785+0000 7fa6496b3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa644108be0 con 0x7fa6441081f0 2024-04-25T01:15:52.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.785+0000 7fa642ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa644108610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:52.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.785+0000 7fa642ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa644108610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59728/0 (socket says 172.21.15.27:59728) 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.785+0000 7fa642ffd700 1 -- 172.21.15.27:0/745315109 learned_addr learned my addr 172.21.15.27:0/745315109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.785+0000 7fa642ffd700 1 -- 172.21.15.27:0/745315109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa644106e50 con 0x7fa6441081f0 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa642ffd700 1 --2- 172.21.15.27:0/745315109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa644108610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6340098a0 tx=0x7fa634009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc3d94cb2d04bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa641ffb700 1 -- 172.21.15.27:0/745315109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa634004160 con 0x7fa6441081f0 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa641ffb700 1 -- 172.21.15.27:0/745315109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa634004300 con 0x7fa6441081f0 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa641ffb700 1 -- 172.21.15.27:0/745315109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa634011b20 con 0x7fa6441081f0 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa6496b3700 1 -- 172.21.15.27:0/745315109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 msgr2=0x7fa644108610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa6496b3700 1 --2- 172.21.15.27:0/745315109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa644108610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6340098a0 tx=0x7fa634009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa6496b3700 1 -- 172.21.15.27:0/745315109 shutdown_connections 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa6496b3700 1 --2- 172.21.15.27:0/745315109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa644108610 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa6496b3700 1 -- 172.21.15.27:0/745315109 >> 172.21.15.27:0/745315109 conn(0x7fa644101940 msgr2=0x7fa644103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa6496b3700 1 -- 172.21.15.27:0/745315109 shutdown_connections 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.786+0000 7fa6496b3700 1 -- 172.21.15.27:0/745315109 wait complete. 2024-04-25T01:15:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.787+0000 7fa6496b3700 1 Processor -- start 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.787+0000 7fa6496b3700 1 -- start start 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.787+0000 7fa6496b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa64419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.787+0000 7fa6496b3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa64410b210 con 0x7fa6441081f0 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.787+0000 7fa642ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa64419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.787+0000 7fa642ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa64419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59734/0 (socket says 172.21.15.27:59734) 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.787+0000 7fa642ffd700 1 -- 172.21.15.27:0/632959425 learned_addr learned my addr 172.21.15.27:0/632959425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.788+0000 7fa642ffd700 1 -- 172.21.15.27:0/632959425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64419b060 con 0x7fa6441081f0 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.788+0000 7fa642ffd700 1 --2- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa64419ab20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6340117e0 tx=0x7fa634011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.788+0000 7fa62bfff700 1 -- 172.21.15.27:0/632959425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa634003bf0 con 0x7fa6441081f0 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.788+0000 7fa62bfff700 1 -- 172.21.15.27:0/632959425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63401a910 con 0x7fa6441081f0 2024-04-25T01:15:52.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.788+0000 7fa62bfff700 1 -- 172.21.15.27:0/632959425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa634024970 con 0x7fa6441081f0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.788+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64419b2f0 con 0x7fa6441081f0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.788+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64419b750 con 0x7fa6441081f0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.789+0000 7fa62bfff700 1 -- 172.21.15.27:0/632959425 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fa634022020 con 0x7fa6441081f0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.789+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa644063890 con 0x7fa6441081f0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.789+0000 7fa62bfff700 1 --2- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa62c039100 0x7fa62c03b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.789+0000 7fa62bfff700 1 -- 172.21.15.27:0/632959425 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa63404d4f0 con 0x7fa6441081f0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.789+0000 7fa6427fc700 1 --2- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa62c039100 0x7fa62c03b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.790+0000 7fa6427fc700 1 --2- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa62c039100 0x7fa62c03b5c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa638006fd0 tx=0x7fa638006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:50.794+0000 7fa62bfff700 1 -- 172.21.15.27:0/632959425 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa6340212c0 con 0x7fa6441081f0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.014+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fa64419e120 con 0x7fa6441081f0 2024-04-25T01:15:52.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.014+0000 7fa62bfff700 1 -- 172.21.15.27:0/632959425 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fa63401ab70 con 0x7fa6441081f0 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.016+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa62c039100 msgr2=0x7fa62c03b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 --2- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa62c039100 0x7fa62c03b5c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa638006fd0 tx=0x7fa638006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 msgr2=0x7fa64419ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 --2- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa64419ab20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6340117e0 tx=0x7fa634011ba0 comp rx=0 tx=0).stop 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 shutdown_connections 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 --2- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa62c039100 0x7fa62c03b5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 --2- 172.21.15.27:0/632959425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6441081f0 0x7fa64419ab20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 >> 172.21.15.27:0/632959425 conn(0x7fa644101940 msgr2=0x7fa6441023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 shutdown_connections 2024-04-25T01:15:52.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:51.017+0000 7fa6496b3700 1 -- 172.21.15.27:0/632959425 wait complete. 2024-04-25T01:15:52.568 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:15:52.568 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T01:15:52.577 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:15:52.577 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T01:15:52.577 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T01:15:52.577 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T01:15:52.579 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph Dashboard is now available at: 2024-04-25T01:15:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: URL: https://smithi027:8443/ 2024-04-25T01:15:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: User: admin 2024-04-25T01:15:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: Password: v1inv7i691 2024-04-25T01:15:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:52.580 INFO:teuthology.orchestra.run.smithi027.stdout:Saving cluster configuration to /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config directory 2024-04-25T01:15:52.580 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling autotune for osd_memory_target 2024-04-25T01:15:52.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:52 smithi027 ceph-mon[29197]: mgrmap e13: x(active, since 6s) 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.041+0000 7f019a7f8700 1 Processor -- start 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.041+0000 7f019a7f8700 1 -- start start 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.041+0000 7f019a7f8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f0194106a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.041+0000 7f019a7f8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f019407bdb0 con 0x7f0194106610 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.041+0000 7f0193fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f0194106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.041+0000 7f0193fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f0194106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59746/0 (socket says 172.21.15.27:59746) 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.041+0000 7f0193fff700 1 -- 172.21.15.27:0/2841237229 learned_addr learned my addr 172.21.15.27:0/2841237229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.041+0000 7f0193fff700 1 -- 172.21.15.27:0/2841237229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0194106f70 con 0x7f0194106610 2024-04-25T01:15:53.809 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f0193fff700 1 --2- 172.21.15.27:0/2841237229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f0194106a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f01840098a0 tx=0x7f0184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90634d3c8d6de53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f0192ffd700 1 -- 172.21.15.27:0/2841237229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0184004160 con 0x7f0194106610 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f0192ffd700 1 -- 172.21.15.27:0/2841237229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0184004300 con 0x7f0194106610 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f0192ffd700 1 -- 172.21.15.27:0/2841237229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0184011b20 con 0x7f0194106610 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f019a7f8700 1 -- 172.21.15.27:0/2841237229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 msgr2=0x7f0194106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f019a7f8700 1 --2- 172.21.15.27:0/2841237229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f0194106a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f01840098a0 tx=0x7f0184009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f019a7f8700 1 -- 172.21.15.27:0/2841237229 shutdown_connections 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f019a7f8700 1 --2- 172.21.15.27:0/2841237229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f0194106a30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.042+0000 7f019a7f8700 1 -- 172.21.15.27:0/2841237229 >> 172.21.15.27:0/2841237229 conn(0x7f01941021d0 msgr2=0x7f01941045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.043+0000 7f019a7f8700 1 -- 172.21.15.27:0/2841237229 shutdown_connections 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.043+0000 7f019a7f8700 1 -- 172.21.15.27:0/2841237229 wait complete. 2024-04-25T01:15:53.810 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.043+0000 7f019a7f8700 1 Processor -- start 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.043+0000 7f019a7f8700 1 -- start start 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f019a7f8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f019419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f019a7f8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0194108040 con 0x7f0194106610 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f0193fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f019419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f0193fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f019419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59760/0 (socket says 172.21.15.27:59760) 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f0193fff700 1 -- 172.21.15.27:0/1158373867 learned_addr learned my addr 172.21.15.27:0/1158373867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f0193fff700 1 -- 172.21.15.27:0/1158373867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019419e330 con 0x7f0194106610 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f0193fff700 1 --2- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f019419ddf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f01840117e0 tx=0x7f0184011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f01917fa700 1 -- 172.21.15.27:0/1158373867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0184003ba0 con 0x7f0194106610 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f01917fa700 1 -- 172.21.15.27:0/1158373867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f018401a910 con 0x7f0194106610 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f019419e5c0 con 0x7f0194106610 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f01917fa700 1 -- 172.21.15.27:0/1158373867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01840248f0 con 0x7f0194106610 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.044+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f019419ea20 con 0x7f0194106610 2024-04-25T01:15:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.045+0000 7f01917fa700 1 -- 172.21.15.27:0/1158373867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0184022020 con 0x7f0194106610 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.045+0000 7f01917fa700 1 --2- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f017c0391a0 0x7f017c03b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.046+0000 7f01917fa700 1 -- 172.21.15.27:0/1158373867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f018404d680 con 0x7f0194106610 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.046+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0194063890 con 0x7f0194106610 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.046+0000 7f01937fe700 1 --2- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f017c0391a0 0x7f017c03b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.046+0000 7f01937fe700 1 --2- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f017c0391a0 0x7f017c03b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0188006fd0 tx=0x7f0188006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.051+0000 7f01917fa700 1 -- 172.21.15.27:0/1158373867 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0184020c10 con 0x7f0194106610 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.255+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f0194062e40 con 0x7f0194106610 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.256+0000 7f01917fa700 1 -- 172.21.15.27:0/1158373867 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f0184021d80 con 0x7f0194106610 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f017c0391a0 msgr2=0x7f017c03b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 --2- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f017c0391a0 0x7f017c03b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0188006fd0 tx=0x7f0188006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 msgr2=0x7f019419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 --2- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f019419ddf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f01840117e0 tx=0x7f0184011ba0 comp rx=0 tx=0).stop 2024-04-25T01:15:53.812 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 shutdown_connections 2024-04-25T01:15:53.813 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 --2- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f017c0391a0 0x7f017c03b660 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:53.813 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 --2- 172.21.15.27:0/1158373867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0194106610 0x7f019419ddf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:53.813 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 >> 172.21.15.27:0/1158373867 conn(0x7f01941021d0 msgr2=0x7f0194194a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:53.813 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 shutdown_connections 2024-04-25T01:15:53.813 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:53.258+0000 7f019a7f8700 1 -- 172.21.15.27:0/1158373867 wait complete. 2024-04-25T01:15:55.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.223+0000 7f3e448d3700 1 Processor -- start 2024-04-25T01:15:55.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.223+0000 7f3e448d3700 1 -- start start 2024-04-25T01:15:55.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e448d3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4010abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:55.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e448d3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4007be10 con 0x7f3e401087c0 2024-04-25T01:15:55.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e3e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4010abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:55.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e3e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4010abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59774/0 (socket says 172.21.15.27:59774) 2024-04-25T01:15:55.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e3e59c700 1 -- 172.21.15.27:0/1821960030 learned_addr learned my addr 172.21.15.27:0/1821960030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e3e59c700 1 -- 172.21.15.27:0/1821960030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4010b0f0 con 0x7f3e401087c0 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e3e59c700 1 --2- 172.21.15.27:0/1821960030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4010abb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e300098a0 tx=0x7f3e30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611dc29f4d334ad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e3d59a700 1 -- 172.21.15.27:0/1821960030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e30004160 con 0x7f3e401087c0 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e3d59a700 1 -- 172.21.15.27:0/1821960030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e30004300 con 0x7f3e401087c0 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.224+0000 7f3e3d59a700 1 -- 172.21.15.27:0/1821960030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e30011b20 con 0x7f3e401087c0 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.225+0000 7f3e448d3700 1 -- 172.21.15.27:0/1821960030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 msgr2=0x7f3e4010abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.225+0000 7f3e448d3700 1 --2- 172.21.15.27:0/1821960030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4010abb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e300098a0 tx=0x7f3e30009bb0 comp rx=0 tx=0).stop 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.225+0000 7f3e448d3700 1 -- 172.21.15.27:0/1821960030 shutdown_connections 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.225+0000 7f3e448d3700 1 --2- 172.21.15.27:0/1821960030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4010abb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.225+0000 7f3e448d3700 1 -- 172.21.15.27:0/1821960030 >> 172.21.15.27:0/1821960030 conn(0x7f3e401021d0 msgr2=0x7f3e401045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.225+0000 7f3e448d3700 1 -- 172.21.15.27:0/1821960030 shutdown_connections 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.225+0000 7f3e448d3700 1 -- 172.21.15.27:0/1821960030 wait complete. 2024-04-25T01:15:55.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.226+0000 7f3e448d3700 1 Processor -- start 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.226+0000 7f3e448d3700 1 -- start start 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.226+0000 7f3e448d3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4019bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.226+0000 7f3e448d3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4010c150 con 0x7f3e401087c0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.226+0000 7f3e3e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4019bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.226+0000 7f3e3e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4019bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59786/0 (socket says 172.21.15.27:59786) 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.226+0000 7f3e3e59c700 1 -- 172.21.15.27:0/2540395560 learned_addr learned my addr 172.21.15.27:0/2540395560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.227+0000 7f3e3e59c700 1 -- 172.21.15.27:0/2540395560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4019c150 con 0x7f3e401087c0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.227+0000 7f3e3e59c700 1 --2- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4019bc10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3e300117e0 tx=0x7f3e30011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.227+0000 7f3e2f7fe700 1 -- 172.21.15.27:0/2540395560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e30003bf0 con 0x7f3e401087c0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.227+0000 7f3e2f7fe700 1 -- 172.21.15.27:0/2540395560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e3001a910 con 0x7f3e401087c0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.227+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4019c3e0 con 0x7f3e401087c0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.227+0000 7f3e2f7fe700 1 -- 172.21.15.27:0/2540395560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e30024970 con 0x7f3e401087c0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.227+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4019c840 con 0x7f3e401087c0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.228+0000 7f3e2f7fe700 1 -- 172.21.15.27:0/2540395560 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3e30022020 con 0x7f3e401087c0 2024-04-25T01:15:55.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.228+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e40063890 con 0x7f3e401087c0 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.228+0000 7f3e2f7fe700 1 --2- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e280390f0 0x7f3e2803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.228+0000 7f3e2f7fe700 1 -- 172.21.15.27:0/2540395560 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3e3004d730 con 0x7f3e401087c0 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.228+0000 7f3e3dd9b700 1 --2- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e280390f0 0x7f3e2803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.229+0000 7f3e3dd9b700 1 --2- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e280390f0 0x7f3e2803b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3e34006fd0 tx=0x7f3e34006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.233+0000 7f3e2f7fe700 1 -- 172.21.15.27:0/2540395560 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3e30011e20 con 0x7f3e401087c0 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.559+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f3e4019f210 con 0x7f3e401087c0 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.561+0000 7f3e2f7fe700 1 -- 172.21.15.27:0/2540395560 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f3e30019c70 con 0x7f3e401087c0 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e280390f0 msgr2=0x7f3e2803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 --2- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e280390f0 0x7f3e2803b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3e34006fd0 tx=0x7f3e34006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 msgr2=0x7f3e4019bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 --2- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4019bc10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3e300117e0 tx=0x7f3e30011c10 comp rx=0 tx=0).stop 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 shutdown_connections 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 --2- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e280390f0 0x7f3e2803b5b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:55.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 --2- 172.21.15.27:0/2540395560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e401087c0 0x7f3e4019bc10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 >> 172.21.15.27:0/2540395560 conn(0x7f3e401021d0 msgr2=0x7f3e40102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 shutdown_connections 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T01:15:54.564+0000 7f3e448d3700 1 -- 172.21.15.27:0/2540395560 wait complete. 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T01:15:55.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:55.143 INFO:teuthology.orchestra.run.smithi027.stdout: ceph telemetry on 2024-04-25T01:15:55.143 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:55.143 INFO:teuthology.orchestra.run.smithi027.stdout:For more information see: 2024-04-25T01:15:55.143 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:55.143 INFO:teuthology.orchestra.run.smithi027.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T01:15:55.143 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:15:55.143 INFO:teuthology.orchestra.run.smithi027.stdout:Bootstrap complete. 2024-04-25T01:15:55.178 INFO:tasks.cephadm:Fetching config... 2024-04-25T01:15:55.178 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:15:55.179 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T01:15:55.195 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T01:15:55.195 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:15:55.195 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T01:15:55.251 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T01:15:55.251 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:15:55.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T01:15:55.316 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T01:15:55.316 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:15:55.316 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T01:15:55.371 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T01:15:55.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8c9n5alAJq5XI4/hLZkSZ/5JzC95QmVd+bmP60d/cnqfpCxZu8JlMfIPhTYH8qS+svozGlAz69wcPKWlZ0jUf4IyNbljQzh/98iGJOT7LaUL4li8UgVlyp/+4l9+McG2Y7UGOHiollFga7i/AO0tK6vNa0wU1YfO68WvUP1ro62hfo+hLDz74uzYbf2LKXI5EdNI0v5a4Li5VPbwMBQpsMRuD3g/QGPsJe8648T5R3sSvkvMOXkpooqNYCybQ0VvQ6m8zSKx8+vvZrRNT0g9UAIFKnd2gyCl2HZGZIFPzGcm54sR2OA0+VILT3tN4Jb1HHp+Che7nzqEZljqNC7Q4GYwsXhyU1S/xoXtf5WUxUTU9ka8hcZzEFDAlSxtX5r92kkxNILqFH+Kts3szTlj6zNSPPY9OKD6QsYGtdM4s+sibABQ4JpX3qqFk5dK0/Exffi/yt4nN/cbVVmKbghjf0nDmonUFQTRln+tkXaSdehB4nIQZubd62prc4zEAoAE= ceph-1469ec2e-02a1-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:15:55.457 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8c9n5alAJq5XI4/hLZkSZ/5JzC95QmVd+bmP60d/cnqfpCxZu8JlMfIPhTYH8qS+svozGlAz69wcPKWlZ0jUf4IyNbljQzh/98iGJOT7LaUL4li8UgVlyp/+4l9+McG2Y7UGOHiollFga7i/AO0tK6vNa0wU1YfO68WvUP1ro62hfo+hLDz74uzYbf2LKXI5EdNI0v5a4Li5VPbwMBQpsMRuD3g/QGPsJe8648T5R3sSvkvMOXkpooqNYCybQ0VvQ6m8zSKx8+vvZrRNT0g9UAIFKnd2gyCl2HZGZIFPzGcm54sR2OA0+VILT3tN4Jb1HHp+Che7nzqEZljqNC7Q4GYwsXhyU1S/xoXtf5WUxUTU9ka8hcZzEFDAlSxtX5r92kkxNILqFH+Kts3szTlj6zNSPPY9OKD6QsYGtdM4s+sibABQ4JpX3qqFk5dK0/Exffi/yt4nN/cbVVmKbghjf0nDmonUFQTRln+tkXaSdehB4nIQZubd62prc4zEAoAE= ceph-1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:15:55.471 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8c9n5alAJq5XI4/hLZkSZ/5JzC95QmVd+bmP60d/cnqfpCxZu8JlMfIPhTYH8qS+svozGlAz69wcPKWlZ0jUf4IyNbljQzh/98iGJOT7LaUL4li8UgVlyp/+4l9+McG2Y7UGOHiollFga7i/AO0tK6vNa0wU1YfO68WvUP1ro62hfo+hLDz74uzYbf2LKXI5EdNI0v5a4Li5VPbwMBQpsMRuD3g/QGPsJe8648T5R3sSvkvMOXkpooqNYCybQ0VvQ6m8zSKx8+vvZrRNT0g9UAIFKnd2gyCl2HZGZIFPzGcm54sR2OA0+VILT3tN4Jb1HHp+Che7nzqEZljqNC7Q4GYwsXhyU1S/xoXtf5WUxUTU9ka8hcZzEFDAlSxtX5r92kkxNILqFH+Kts3szTlj6zNSPPY9OKD6QsYGtdM4s+sibABQ4JpX3qqFk5dK0/Exffi/yt4nN/cbVVmKbghjf0nDmonUFQTRln+tkXaSdehB4nIQZubd62prc4zEAoAE= ceph-1469ec2e-02a1-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:15:55.510 INFO:teuthology.orchestra.run.smithi139.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8c9n5alAJq5XI4/hLZkSZ/5JzC95QmVd+bmP60d/cnqfpCxZu8JlMfIPhTYH8qS+svozGlAz69wcPKWlZ0jUf4IyNbljQzh/98iGJOT7LaUL4li8UgVlyp/+4l9+McG2Y7UGOHiollFga7i/AO0tK6vNa0wU1YfO68WvUP1ro62hfo+hLDz74uzYbf2LKXI5EdNI0v5a4Li5VPbwMBQpsMRuD3g/QGPsJe8648T5R3sSvkvMOXkpooqNYCybQ0VvQ6m8zSKx8+vvZrRNT0g9UAIFKnd2gyCl2HZGZIFPzGcm54sR2OA0+VILT3tN4Jb1HHp+Che7nzqEZljqNC7Q4GYwsXhyU1S/xoXtf5WUxUTU9ka8hcZzEFDAlSxtX5r92kkxNILqFH+Kts3szTlj6zNSPPY9OKD6QsYGtdM4s+sibABQ4JpX3qqFk5dK0/Exffi/yt4nN/cbVVmKbghjf0nDmonUFQTRln+tkXaSdehB4nIQZubd62prc4zEAoAE= ceph-1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:15:55.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8c9n5alAJq5XI4/hLZkSZ/5JzC95QmVd+bmP60d/cnqfpCxZu8JlMfIPhTYH8qS+svozGlAz69wcPKWlZ0jUf4IyNbljQzh/98iGJOT7LaUL4li8UgVlyp/+4l9+McG2Y7UGOHiollFga7i/AO0tK6vNa0wU1YfO68WvUP1ro62hfo+hLDz74uzYbf2LKXI5EdNI0v5a4Li5VPbwMBQpsMRuD3g/QGPsJe8648T5R3sSvkvMOXkpooqNYCybQ0VvQ6m8zSKx8+vvZrRNT0g9UAIFKnd2gyCl2HZGZIFPzGcm54sR2OA0+VILT3tN4Jb1HHp+Che7nzqEZljqNC7Q4GYwsXhyU1S/xoXtf5WUxUTU9ka8hcZzEFDAlSxtX5r92kkxNILqFH+Kts3szTlj6zNSPPY9OKD6QsYGtdM4s+sibABQ4JpX3qqFk5dK0/Exffi/yt4nN/cbVVmKbghjf0nDmonUFQTRln+tkXaSdehB4nIQZubd62prc4zEAoAE= ceph-1469ec2e-02a1-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:15:55.563 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8c9n5alAJq5XI4/hLZkSZ/5JzC95QmVd+bmP60d/cnqfpCxZu8JlMfIPhTYH8qS+svozGlAz69wcPKWlZ0jUf4IyNbljQzh/98iGJOT7LaUL4li8UgVlyp/+4l9+McG2Y7UGOHiollFga7i/AO0tK6vNa0wU1YfO68WvUP1ro62hfo+hLDz74uzYbf2LKXI5EdNI0v5a4Li5VPbwMBQpsMRuD3g/QGPsJe8648T5R3sSvkvMOXkpooqNYCybQ0VvQ6m8zSKx8+vvZrRNT0g9UAIFKnd2gyCl2HZGZIFPzGcm54sR2OA0+VILT3tN4Jb1HHp+Che7nzqEZljqNC7Q4GYwsXhyU1S/xoXtf5WUxUTU9ka8hcZzEFDAlSxtX5r92kkxNILqFH+Kts3szTlj6zNSPPY9OKD6QsYGtdM4s+sibABQ4JpX3qqFk5dK0/Exffi/yt4nN/cbVVmKbghjf0nDmonUFQTRln+tkXaSdehB4nIQZubd62prc4zEAoAE= ceph-1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:15:55.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T01:15:55.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:55 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2540395560' entity='client.admin' 2024-04-25T01:15:55.817 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:15:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.700+0000 7f0e22912700 1 -- 172.21.15.27:0/2540928097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 msgr2=0x7f0e1c100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.700+0000 7f0e22912700 1 --2- 172.21.15.27:0/2540928097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 0x7f0e1c100fb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009910 tx=0x7f0e0c009c20 comp rx=0 tx=0).stop 2024-04-25T01:15:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.700+0000 7f0e22912700 1 -- 172.21.15.27:0/2540928097 shutdown_connections 2024-04-25T01:15:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.700+0000 7f0e22912700 1 --2- 172.21.15.27:0/2540928097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 0x7f0e1c100fb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.700+0000 7f0e22912700 1 -- 172.21.15.27:0/2540928097 >> 172.21.15.27:0/2540928097 conn(0x7f0e1c0fc130 msgr2=0x7f0e1c0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.700+0000 7f0e22912700 1 -- 172.21.15.27:0/2540928097 shutdown_connections 2024-04-25T01:15:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.701+0000 7f0e22912700 1 -- 172.21.15.27:0/2540928097 wait complete. 2024-04-25T01:15:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.701+0000 7f0e22912700 1 Processor -- start 2024-04-25T01:15:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.701+0000 7f0e22912700 1 -- start start 2024-04-25T01:15:58.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e22912700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 0x7f0e1c06a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:58.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e22912700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1c106c20 con 0x7f0e1c100b90 2024-04-25T01:15:58.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e1bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 0x7f0e1c06a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:58.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e1bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 0x7f0e1c06a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50794/0 (socket says 172.21.15.27:50794) 2024-04-25T01:15:58.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e1bfff700 1 -- 172.21.15.27:0/1806486291 learned_addr learned my addr 172.21.15.27:0/1806486291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:15:58.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e1bfff700 1 -- 172.21.15.27:0/1806486291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1c19a080 con 0x7f0e1c100b90 2024-04-25T01:15:58.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e1bfff700 1 --2- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 0x7f0e1c06a130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c006010 tx=0x7f0e0c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:58.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e197fa700 1 -- 172.21.15.27:0/1806486291 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e0c027070 con 0x7f0e1c100b90 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1c19a250 con 0x7f0e1c100b90 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e197fa700 1 -- 172.21.15.27:0/1806486291 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e0c00be40 con 0x7f0e1c100b90 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e197fa700 1 -- 172.21.15.27:0/1806486291 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e0c00f460 con 0x7f0e1c100b90 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.702+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e1c19a670 con 0x7f0e1c100b90 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.703+0000 7f0e197fa700 1 -- 172.21.15.27:0/1806486291 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0e0c02a070 con 0x7f0e1c100b90 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.703+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e1c1073a0 con 0x7f0e1c100b90 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.703+0000 7f0e197fa700 1 --2- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0e040390f0 0x7f0e0403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.703+0000 7f0e197fa700 1 -- 172.21.15.27:0/1806486291 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0e0c01e030 con 0x7f0e1c100b90 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.704+0000 7f0e1b7fe700 1 --2- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0e040390f0 0x7f0e0403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:15:58.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.704+0000 7f0e1b7fe700 1 --2- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0e040390f0 0x7f0e0403b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006fd0 tx=0x7f0e10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:15:58.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.709+0000 7f0e197fa700 1 -- 172.21.15.27:0/1806486291 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e0c013070 con 0x7f0e1c100b90 2024-04-25T01:15:58.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.918+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f0e1c02dcf0 con 0x7f0e1c100b90 2024-04-25T01:15:58.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.921+0000 7f0e197fa700 1 -- 172.21.15.27:0/1806486291 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f0e0c0310c0 con 0x7f0e1c100b90 2024-04-25T01:15:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0e040390f0 msgr2=0x7f0e0403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 --2- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0e040390f0 0x7f0e0403b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006fd0 tx=0x7f0e10006e40 comp rx=0 tx=0).stop 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 msgr2=0x7f0e1c06a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 --2- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 0x7f0e1c06a130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c006010 tx=0x7f0e0c005050 comp rx=0 tx=0).stop 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 shutdown_connections 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 --2- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0e040390f0 0x7f0e0403b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 --2- 172.21.15.27:0/1806486291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c100b90 0x7f0e1c06a130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 >> 172.21.15.27:0/1806486291 conn(0x7f0e1c0fc130 msgr2=0x7f0e1c0fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 shutdown_connections 2024-04-25T01:15:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:15:58.923+0000 7f0e22912700 1 -- 172.21.15.27:0/1806486291 wait complete. 2024-04-25T01:15:59.559 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T01:15:59.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T01:15:59.801 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:00.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:59 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1806486291' entity='client.admin' 2024-04-25T01:16:00.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:00.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:00.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:16:00.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:00.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:00.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:15:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:01.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.296+0000 7efcded7c700 1 -- 172.21.15.27:0/3774443102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 msgr2=0x7efcd8104bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:01.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.296+0000 7efcded7c700 1 --2- 172.21.15.27:0/3774443102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 0x7efcd8104bf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efccc009910 tx=0x7efccc009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:01.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.296+0000 7efcded7c700 1 -- 172.21.15.27:0/3774443102 shutdown_connections 2024-04-25T01:16:01.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.296+0000 7efcded7c700 1 --2- 172.21.15.27:0/3774443102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 0x7efcd8104bf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:01.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.296+0000 7efcded7c700 1 -- 172.21.15.27:0/3774443102 >> 172.21.15.27:0/3774443102 conn(0x7efcd80fc170 msgr2=0x7efcd80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:01.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.296+0000 7efcded7c700 1 -- 172.21.15.27:0/3774443102 shutdown_connections 2024-04-25T01:16:01.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.296+0000 7efcded7c700 1 -- 172.21.15.27:0/3774443102 wait complete. 2024-04-25T01:16:01.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.297+0000 7efcded7c700 1 Processor -- start 2024-04-25T01:16:01.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.297+0000 7efcded7c700 1 -- start start 2024-04-25T01:16:01.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.297+0000 7efcded7c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 0x7efcd8102080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:01.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.297+0000 7efcded7c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efcd8107d40 con 0x7efcd8102800 2024-04-25T01:16:01.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.297+0000 7efcdcb18700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 0x7efcd8102080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:01.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.297+0000 7efcdcb18700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 0x7efcd8102080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50814/0 (socket says 172.21.15.27:50814) 2024-04-25T01:16:01.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.297+0000 7efcdcb18700 1 -- 172.21.15.27:0/1845656605 learned_addr learned my addr 172.21.15.27:0/1845656605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:16:01.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.297+0000 7efcdcb18700 1 -- 172.21.15.27:0/1845656605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd81025c0 con 0x7efcd8102800 2024-04-25T01:16:01.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.298+0000 7efcdcb18700 1 --2- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 0x7efcd8102080 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efccc005b40 tx=0x7efccc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:01.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.298+0000 7efcd5ffb700 1 -- 172.21.15.27:0/1845656605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efccc027070 con 0x7efcd8102800 2024-04-25T01:16:01.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.298+0000 7efcd5ffb700 1 -- 172.21.15.27:0/1845656605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efccc00be40 con 0x7efcd8102800 2024-04-25T01:16:01.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.298+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd81007b0 con 0x7efcd8102800 2024-04-25T01:16:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.298+0000 7efcd5ffb700 1 -- 172.21.15.27:0/1845656605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efccc00f460 con 0x7efcd8102800 2024-04-25T01:16:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.298+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd8100cf0 con 0x7efcd8102800 2024-04-25T01:16:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.299+0000 7efcd5ffb700 1 -- 172.21.15.27:0/1845656605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7efccc02a070 con 0x7efcd8102800 2024-04-25T01:16:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.299+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd81084c0 con 0x7efcd8102800 2024-04-25T01:16:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.299+0000 7efcd5ffb700 1 --2- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efcc80390f0 0x7efcc803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.299+0000 7efcd5ffb700 1 -- 172.21.15.27:0/1845656605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efccc01e030 con 0x7efcd8102800 2024-04-25T01:16:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.299+0000 7efcd7fff700 1 --2- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efcc80390f0 0x7efcc803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.299+0000 7efcd7fff700 1 --2- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efcc80390f0 0x7efcc803b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efcc4006fd0 tx=0x7efcc4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.304+0000 7efcd5ffb700 1 -- 172.21.15.27:0/1845656605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efccc013070 con 0x7efcd8102800 2024-04-25T01:16:01.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.516+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7efcd81907c0 con 0x7efcc80390f0 2024-04-25T01:16:01.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.519+0000 7efcd5ffb700 1 -- 172.21.15.27:0/1845656605 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efcd81907c0 con 0x7efcc80390f0 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.521+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efcc80390f0 msgr2=0x7efcc803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.521+0000 7efcded7c700 1 --2- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efcc80390f0 0x7efcc803b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efcc4006fd0 tx=0x7efcc4006e40 comp rx=0 tx=0).stop 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.521+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 msgr2=0x7efcd8102080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.521+0000 7efcded7c700 1 --2- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 0x7efcd8102080 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efccc005b40 tx=0x7efccc005030 comp rx=0 tx=0).stop 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.522+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 shutdown_connections 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.522+0000 7efcded7c700 1 --2- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efcc80390f0 0x7efcc803b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.522+0000 7efcded7c700 1 --2- 172.21.15.27:0/1845656605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd8102800 0x7efcd8102080 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.522+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 >> 172.21.15.27:0/1845656605 conn(0x7efcd80fc170 msgr2=0x7efcd80fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.522+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 shutdown_connections 2024-04-25T01:16:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:01.522+0000 7efcded7c700 1 -- 172.21.15.27:0/1845656605 wait complete. 2024-04-25T01:16:02.222 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi139 2024-04-25T01:16:02.222 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:16:02.222 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:16:02.245 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:16:02.245 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:02.307 INFO:tasks.cephadm:Adding host smithi139 to orchestrator... 2024-04-25T01:16:02.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch host add smithi139 2024-04-25T01:16:02.558 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:02.768 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:02.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.964+0000 7fa89981d700 1 -- 172.21.15.27:0/3113593317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 msgr2=0x7fa894103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.964+0000 7fa89981d700 1 --2- 172.21.15.27:0/3113593317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 0x7fa894103220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa88400b0a0 tx=0x7fa88400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:16:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.964+0000 7fa89981d700 1 -- 172.21.15.27:0/3113593317 shutdown_connections 2024-04-25T01:16:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.964+0000 7fa89981d700 1 --2- 172.21.15.27:0/3113593317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 0x7fa894103220 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.964+0000 7fa89981d700 1 -- 172.21.15.27:0/3113593317 >> 172.21.15.27:0/3113593317 conn(0x7fa8940fc690 msgr2=0x7fa8940feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.964+0000 7fa89981d700 1 -- 172.21.15.27:0/3113593317 shutdown_connections 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.964+0000 7fa89981d700 1 -- 172.21.15.27:0/3113593317 wait complete. 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.964+0000 7fa89981d700 1 Processor -- start 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.965+0000 7fa89981d700 1 -- start start 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.965+0000 7fa89981d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 0x7fa894100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.965+0000 7fa89981d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa89410c310 con 0x7fa894102e40 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.965+0000 7fa892ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 0x7fa894100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.965+0000 7fa892ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 0x7fa894100aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50834/0 (socket says 172.21.15.27:50834) 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.965+0000 7fa892ffd700 1 -- 172.21.15.27:0/2500591565 learned_addr learned my addr 172.21.15.27:0/2500591565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.965+0000 7fa892ffd700 1 -- 172.21.15.27:0/2500591565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa894102990 con 0x7fa894102e40 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.965+0000 7fa892ffd700 1 --2- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 0x7fa894100aa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa8840062a0 tx=0x7fa88400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.966+0000 7fa89881b700 1 -- 172.21.15.27:0/2500591565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa884017050 con 0x7fa894102e40 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.966+0000 7fa89881b700 1 -- 172.21.15.27:0/2500591565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa884023070 con 0x7fa894102e40 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.966+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa894100fe0 con 0x7fa894102e40 2024-04-25T01:16:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.966+0000 7fa89881b700 1 -- 172.21.15.27:0/2500591565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa884004000 con 0x7fa894102e40 2024-04-25T01:16:03.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.966+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8941014a0 con 0x7fa894102e40 2024-04-25T01:16:03.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.966+0000 7fa89881b700 1 -- 172.21.15.27:0/2500591565 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fa8840042c0 con 0x7fa894102e40 2024-04-25T01:16:03.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.967+0000 7fa89881b700 1 --2- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8800390f0 0x7fa88003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:03.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.967+0000 7fa89881b700 1 -- 172.21.15.27:0/2500591565 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa88400f050 con 0x7fa894102e40 2024-04-25T01:16:03.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.967+0000 7fa8927fc700 1 --2- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8800390f0 0x7fa88003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:03.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.967+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa89406a980 con 0x7fa894102e40 2024-04-25T01:16:03.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.967+0000 7fa8927fc700 1 --2- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8800390f0 0x7fa88003b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa87c006fd0 tx=0x7fa87c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:03.972+0000 7fa89881b700 1 -- 172.21.15.27:0/2500591565 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa884026bd0 con 0x7fa894102e40 2024-04-25T01:16:04.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:04.181+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}) -- 0x7fa894101fd0 con 0x7fa8800390f0 2024-04-25T01:16:04.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:04 smithi027 ceph-mon[29197]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:05.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:05 smithi027 ceph-mon[29197]: Deploying cephadm binary to smithi139 2024-04-25T01:16:06.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:06 smithi027 ceph-mon[29197]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:08.411 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi139' with addr '172.21.15.139' 2024-04-25T01:16:08.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.426+0000 7fa89881b700 1 -- 172.21.15.27:0/2500591565 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa894101fd0 con 0x7fa8800390f0 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8800390f0 msgr2=0x7fa88003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 --2- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8800390f0 0x7fa88003b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa87c006fd0 tx=0x7fa87c006e40 comp rx=0 tx=0).stop 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 msgr2=0x7fa894100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 --2- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 0x7fa894100aa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa8840062a0 tx=0x7fa88400bdd0 comp rx=0 tx=0).stop 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 shutdown_connections 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 --2- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8800390f0 0x7fa88003b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 --2- 172.21.15.27:0/2500591565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa894102e40 0x7fa894100aa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 >> 172.21.15.27:0/2500591565 conn(0x7fa8940fc690 msgr2=0x7fa894103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 shutdown_connections 2024-04-25T01:16:08.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:08.429+0000 7fa89981d700 1 -- 172.21.15.27:0/2500591565 wait complete. 2024-04-25T01:16:08.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:08 smithi027 ceph-mon[29197]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:08.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:16:09.191 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:09 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:09 smithi027 ceph-mon[29197]: Added host smithi139 2024-04-25T01:16:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:09 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:09.620 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:09 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:10.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:10 smithi027 ceph-mon[29197]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:10.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:10 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.696+0000 7fb0921f3700 1 -- 172.21.15.27:0/1619022514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 msgr2=0x7fb08c1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.696+0000 7fb0921f3700 1 --2- 172.21.15.27:0/1619022514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 0x7fb08c1011c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009910 tx=0x7fb07c009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.696+0000 7fb0921f3700 1 -- 172.21.15.27:0/1619022514 shutdown_connections 2024-04-25T01:16:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.696+0000 7fb0921f3700 1 --2- 172.21.15.27:0/1619022514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 0x7fb08c1011c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.696+0000 7fb0921f3700 1 -- 172.21.15.27:0/1619022514 >> 172.21.15.27:0/1619022514 conn(0x7fb08c0fc690 msgr2=0x7fb08c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.696+0000 7fb0921f3700 1 -- 172.21.15.27:0/1619022514 shutdown_connections 2024-04-25T01:16:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.696+0000 7fb0921f3700 1 -- 172.21.15.27:0/1619022514 wait complete. 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.697+0000 7fb0921f3700 1 Processor -- start 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.697+0000 7fb0921f3700 1 -- start start 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.697+0000 7fb0921f3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 0x7fb08c19f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.697+0000 7fb0921f3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10b1b0 con 0x7fb08c100de0 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.697+0000 7fb08b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 0x7fb08c19f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.697+0000 7fb08b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 0x7fb08c19f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34276/0 (socket says 172.21.15.27:34276) 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.697+0000 7fb08b7fe700 1 -- 172.21.15.27:0/2287428135 learned_addr learned my addr 172.21.15.27:0/2287428135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.698+0000 7fb08b7fe700 1 -- 172.21.15.27:0/2287428135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08c19fbe0 con 0x7fb08c100de0 2024-04-25T01:16:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.698+0000 7fb08b7fe700 1 --2- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 0x7fb08c19f6a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb07c006010 tx=0x7fb07c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.698+0000 7fb088ff9700 1 -- 172.21.15.27:0/2287428135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb07c026070 con 0x7fb08c100de0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.698+0000 7fb088ff9700 1 -- 172.21.15.27:0/2287428135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb07c00bbd0 con 0x7fb08c100de0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.698+0000 7fb088ff9700 1 -- 172.21.15.27:0/2287428135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb07c003910 con 0x7fb08c100de0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.698+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08c19fed0 con 0x7fb08c100de0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.698+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb08c199940 con 0x7fb08c100de0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.699+0000 7fb088ff9700 1 -- 172.21.15.27:0/2287428135 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb07c029070 con 0x7fb08c100de0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.699+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb08c1022c0 con 0x7fb08c100de0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.699+0000 7fb088ff9700 1 --2- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb074039150 0x7fb07403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.699+0000 7fb088ff9700 1 -- 172.21.15.27:0/2287428135 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb07c01e070 con 0x7fb08c100de0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.699+0000 7fb08affd700 1 --2- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb074039150 0x7fb07403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.700+0000 7fb08affd700 1 --2- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb074039150 0x7fb07403b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb080006fd0 tx=0x7fb080006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.704+0000 7fb088ff9700 1 -- 172.21.15.27:0/2287428135 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb07c013070 con 0x7fb08c100de0 2024-04-25T01:16:10.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.916+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb08c19aca0 con 0x7fb074039150 2024-04-25T01:16:10.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.917+0000 7fb088ff9700 1 -- 172.21.15.27:0/2287428135 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fb08c19aca0 con 0x7fb074039150 2024-04-25T01:16:10.901 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:16:10.901 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}] 2024-04-25T01:16:10.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb074039150 msgr2=0x7fb07403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:10.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 --2- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb074039150 0x7fb07403b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb080006fd0 tx=0x7fb080006e40 comp rx=0 tx=0).stop 2024-04-25T01:16:10.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 msgr2=0x7fb08c19f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:10.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 --2- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 0x7fb08c19f6a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb07c006010 tx=0x7fb07c005030 comp rx=0 tx=0).stop 2024-04-25T01:16:10.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 shutdown_connections 2024-04-25T01:16:10.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 --2- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb074039150 0x7fb07403b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:10.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 --2- 172.21.15.27:0/2287428135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08c100de0 0x7fb08c19f6a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:10.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 >> 172.21.15.27:0/2287428135 conn(0x7fb08c0fc690 msgr2=0x7fb08c06a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:10.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 shutdown_connections 2024-04-25T01:16:10.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:10.919+0000 7fb0921f3700 1 -- 172.21.15.27:0/2287428135 wait complete. 2024-04-25T01:16:11.465 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-04-25T01:16:11.465 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:16:11.465 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:16:11.485 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:16:11.485 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:11.543 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-04-25T01:16:11.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch host add smithi153 2024-04-25T01:16:11.782 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:12.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:12 smithi027 ceph-mon[29197]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:16:12.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:12 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.313+0000 7f38d1fce700 1 -- 172.21.15.27:0/1288062570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 msgr2=0x7f38cc107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.313+0000 7f38d1fce700 1 --2- 172.21.15.27:0/1288062570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 0x7f38cc107060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009910 tx=0x7f38c0009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.313+0000 7f38d1fce700 1 -- 172.21.15.27:0/1288062570 shutdown_connections 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.313+0000 7f38d1fce700 1 --2- 172.21.15.27:0/1288062570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 0x7f38cc107060 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.313+0000 7f38d1fce700 1 -- 172.21.15.27:0/1288062570 >> 172.21.15.27:0/1288062570 conn(0x7f38cc0fbde0 msgr2=0x7f38cc0fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.313+0000 7f38d1fce700 1 -- 172.21.15.27:0/1288062570 shutdown_connections 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.313+0000 7f38d1fce700 1 -- 172.21.15.27:0/1288062570 wait complete. 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.314+0000 7f38d1fce700 1 Processor -- start 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.314+0000 7f38d1fce700 1 -- start start 2024-04-25T01:16:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.314+0000 7f38d1fce700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 0x7f38cc19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.314+0000 7f38d1fce700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc10c3e0 con 0x7f38cc106c80 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.314+0000 7f38cb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 0x7f38cc19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.314+0000 7f38cb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 0x7f38cc19d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34298/0 (socket says 172.21.15.27:34298) 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.314+0000 7f38cb7fe700 1 -- 172.21.15.27:0/3079890844 learned_addr learned my addr 172.21.15.27:0/3079890844 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.315+0000 7f38cb7fe700 1 -- 172.21.15.27:0/3079890844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc19d970 con 0x7f38cc106c80 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.315+0000 7f38cb7fe700 1 --2- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 0x7f38cc19d430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005850 tx=0x7f38c0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.315+0000 7f38c8ff9700 1 -- 172.21.15.27:0/3079890844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38c0026070 con 0x7f38cc106c80 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.315+0000 7f38c8ff9700 1 -- 172.21.15.27:0/3079890844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c000bc30 con 0x7f38cc106c80 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.315+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38cc19dc00 con 0x7f38cc106c80 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.315+0000 7f38c8ff9700 1 -- 172.21.15.27:0/3079890844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38c0003a20 con 0x7f38cc106c80 2024-04-25T01:16:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.315+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38cc1975f0 con 0x7f38cc106c80 2024-04-25T01:16:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.316+0000 7f38c8ff9700 1 -- 172.21.15.27:0/3079890844 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f38c0029070 con 0x7f38cc106c80 2024-04-25T01:16:13.303 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:13 smithi027 ceph-mon[29197]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.316+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38cc1033c0 con 0x7f38cc106c80 2024-04-25T01:16:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.316+0000 7f38c8ff9700 1 --2- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f38b40390f0 0x7f38b403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.316+0000 7f38c8ff9700 1 -- 172.21.15.27:0/3079890844 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f38c001e070 con 0x7f38cc106c80 2024-04-25T01:16:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.316+0000 7f38caffd700 1 --2- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f38b40390f0 0x7f38b403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.317+0000 7f38caffd700 1 --2- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f38b40390f0 0x7f38b403b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f38bc006fd0 tx=0x7f38bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.321+0000 7f38c8ff9700 1 -- 172.21.15.27:0/3079890844 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f38c0013070 con 0x7f38cc106c80 2024-04-25T01:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:13.531+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) -- 0x7f38cc198b20 con 0x7f38b40390f0 2024-04-25T01:16:15.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:15.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:15.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: Deploying cephadm binary to smithi153 2024-04-25T01:16:15.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:15.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:15.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:15.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:15.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:16:15.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:15.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:16.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:16 smithi027 ceph-mon[29197]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:16:16.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:16 smithi027 ceph-mon[29197]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:16.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:16 smithi027 ceph-mon[29197]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:17.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:17 smithi027 ceph-mon[29197]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.839+0000 7f38c8ff9700 1 -- 172.21.15.27:0/3079890844 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f38cc198b20 con 0x7f38b40390f0 2024-04-25T01:16:17.825 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.841+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f38b40390f0 msgr2=0x7f38b403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.841+0000 7f38d1fce700 1 --2- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f38b40390f0 0x7f38b403b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f38bc006fd0 tx=0x7f38bc006e40 comp rx=0 tx=0).stop 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.841+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 msgr2=0x7f38cc19d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.841+0000 7f38d1fce700 1 --2- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 0x7f38cc19d430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005850 tx=0x7f38c0005030 comp rx=0 tx=0).stop 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.841+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 shutdown_connections 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.842+0000 7f38d1fce700 1 --2- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f38b40390f0 0x7f38b403b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.842+0000 7f38d1fce700 1 --2- 172.21.15.27:0/3079890844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc106c80 0x7f38cc19d430 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.842+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 >> 172.21.15.27:0/3079890844 conn(0x7f38cc0fbde0 msgr2=0x7f38cc1097f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.842+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 shutdown_connections 2024-04-25T01:16:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:17.842+0000 7f38d1fce700 1 -- 172.21.15.27:0/3079890844 wait complete. 2024-04-25T01:16:18.408 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:16:18.650 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:18.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:18.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:16:18.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:18.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:18.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:18.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:18.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: Added host smithi153 2024-04-25T01:16:18.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:18.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:20.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 -- 172.21.15.27:0/1793690833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 msgr2=0x7fd468101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 --2- 172.21.15.27:0/1793690833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 0x7fd468101180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd464009910 tx=0x7fd464009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 -- 172.21.15.27:0/1793690833 shutdown_connections 2024-04-25T01:16:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 --2- 172.21.15.27:0/1793690833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 0x7fd468101180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 -- 172.21.15.27:0/1793690833 >> 172.21.15.27:0/1793690833 conn(0x7fd4680fc650 msgr2=0x7fd4680fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 -- 172.21.15.27:0/1793690833 shutdown_connections 2024-04-25T01:16:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 -- 172.21.15.27:0/1793690833 wait complete. 2024-04-25T01:16:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 Processor -- start 2024-04-25T01:16:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.254+0000 7fd470366700 1 -- start start 2024-04-25T01:16:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd470366700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 0x7fd468197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd470366700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd46810b170 con 0x7fd468100da0 2024-04-25T01:16:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd46e102700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 0x7fd468197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd46e102700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 0x7fd468197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48394/0 (socket says 172.21.15.27:48394) 2024-04-25T01:16:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd46e102700 1 -- 172.21.15.27:0/3120643152 learned_addr learned my addr 172.21.15.27:0/3120643152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:16:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd46e102700 1 -- 172.21.15.27:0/3120643152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd468197c80 con 0x7fd468100da0 2024-04-25T01:16:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd46e102700 1 --2- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 0x7fd468197740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd464005850 tx=0x7fd464005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd45f7fe700 1 -- 172.21.15.27:0/3120643152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd464026070 con 0x7fd468100da0 2024-04-25T01:16:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd468197f70 con 0x7fd468100da0 2024-04-25T01:16:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd45f7fe700 1 -- 172.21.15.27:0/3120643152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd46400bbd0 con 0x7fd468100da0 2024-04-25T01:16:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd45f7fe700 1 -- 172.21.15.27:0/3120643152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd464003970 con 0x7fd468100da0 2024-04-25T01:16:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.255+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd46819c010 con 0x7fd468100da0 2024-04-25T01:16:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.256+0000 7fd45f7fe700 1 -- 172.21.15.27:0/3120643152 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd464029070 con 0x7fd468100da0 2024-04-25T01:16:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.256+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd468102280 con 0x7fd468100da0 2024-04-25T01:16:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.256+0000 7fd45f7fe700 1 --2- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd4540390f0 0x7fd45403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.256+0000 7fd45f7fe700 1 -- 172.21.15.27:0/3120643152 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd46401e070 con 0x7fd468100da0 2024-04-25T01:16:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.257+0000 7fd46d901700 1 --2- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd4540390f0 0x7fd45403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.257+0000 7fd46d901700 1 --2- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd4540390f0 0x7fd45403b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd458006fd0 tx=0x7fd458006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.262+0000 7fd45f7fe700 1 -- 172.21.15.27:0/3120643152 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd464013070 con 0x7fd468100da0 2024-04-25T01:16:20.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.474+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd468198aa0 con 0x7fd4540390f0 2024-04-25T01:16:20.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.475+0000 7fd45f7fe700 1 -- 172.21.15.27:0/3120643152 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fd468198aa0 con 0x7fd4540390f0 2024-04-25T01:16:20.460 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:16:20.460 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.477+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd4540390f0 msgr2=0x7fd45403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.477+0000 7fd470366700 1 --2- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd4540390f0 0x7fd45403b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd458006fd0 tx=0x7fd458006e40 comp rx=0 tx=0).stop 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.477+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 msgr2=0x7fd468197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.477+0000 7fd470366700 1 --2- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 0x7fd468197740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd464005850 tx=0x7fd464005030 comp rx=0 tx=0).stop 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.477+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 shutdown_connections 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.477+0000 7fd470366700 1 --2- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd4540390f0 0x7fd45403b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.477+0000 7fd470366700 1 --2- 172.21.15.27:0/3120643152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd468100da0 0x7fd468197740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.477+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 >> 172.21.15.27:0/3120643152 conn(0x7fd4680fc650 msgr2=0x7fd46806a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.478+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 shutdown_connections 2024-04-25T01:16:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:20.478+0000 7fd470366700 1 -- 172.21.15.27:0/3120643152 wait complete. 2024-04-25T01:16:21.109 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T01:16:21.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T01:16:21.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:20 smithi027 ceph-mon[29197]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:21.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:21.352 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:22.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:21 smithi027 ceph-mon[29197]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:16:22.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.878+0000 7f2ff88cf700 1 -- 172.21.15.27:0/3193135372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 msgr2=0x7f2ff41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:22.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.878+0000 7f2ff88cf700 1 --2- 172.21.15.27:0/3193135372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 0x7f2ff41031d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009910 tx=0x7f2fe4009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:22.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.878+0000 7f2ff88cf700 1 -- 172.21.15.27:0/3193135372 shutdown_connections 2024-04-25T01:16:22.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.878+0000 7f2ff88cf700 1 --2- 172.21.15.27:0/3193135372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 0x7f2ff41031d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.878+0000 7f2ff88cf700 1 -- 172.21.15.27:0/3193135372 >> 172.21.15.27:0/3193135372 conn(0x7f2ff40fc640 msgr2=0x7f2ff40fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.878+0000 7f2ff88cf700 1 -- 172.21.15.27:0/3193135372 shutdown_connections 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.878+0000 7f2ff88cf700 1 -- 172.21.15.27:0/3193135372 wait complete. 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff88cf700 1 Processor -- start 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff88cf700 1 -- start start 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff88cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 0x7f2ff4100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff88cf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff410c2e0 con 0x7f2ff4102df0 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 0x7f2ff4100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 0x7f2ff4100a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48402/0 (socket says 172.21.15.27:48402) 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff259c700 1 -- 172.21.15.27:0/1181107411 learned_addr learned my addr 172.21.15.27:0/1181107411 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:16:22.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff259c700 1 -- 172.21.15.27:0/1181107411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff4102940 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.879+0000 7f2ff259c700 1 --2- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 0x7f2ff4100a50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4006010 tx=0x7f2fe4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.880+0000 7f2fe37fe700 1 -- 172.21.15.27:0/1181107411 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2fe4026070 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.880+0000 7f2fe37fe700 1 -- 172.21.15.27:0/1181107411 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fe400bc30 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.880+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff4100f90 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.880+0000 7f2fe37fe700 1 -- 172.21.15.27:0/1181107411 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2fe4003ab0 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.880+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff4101450 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.881+0000 7f2fe37fe700 1 -- 172.21.15.27:0/1181107411 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f2fe4029070 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.881+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff406a9a0 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.881+0000 7f2fe37fe700 1 --2- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2fdc0390f0 0x7f2fdc03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.881+0000 7f2fe37fe700 1 -- 172.21.15.27:0/1181107411 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2fe401e070 con 0x7f2ff4102df0 2024-04-25T01:16:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.881+0000 7f2ff1d9b700 1 --2- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2fdc0390f0 0x7f2fdc03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.881+0000 7f2ff1d9b700 1 --2- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2fdc0390f0 0x7f2fdc03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8006fd0 tx=0x7f2fe8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:22.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:22.886+0000 7f2fe37fe700 1 -- 172.21.15.27:0/1181107411 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2fe4013070 con 0x7f2ff4102df0 2024-04-25T01:16:22.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:22 smithi027 ceph-mon[29197]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.097+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f2ff4102020 con 0x7f2ff4102df0 2024-04-25T01:16:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.895+0000 7f2fe37fe700 1 -- 172.21.15.27:0/1181107411 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f2fe4030090 con 0x7f2ff4102df0 2024-04-25T01:16:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2fdc0390f0 msgr2=0x7f2fdc03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 --2- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2fdc0390f0 0x7f2fdc03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8006fd0 tx=0x7f2fe8006e40 comp rx=0 tx=0).stop 2024-04-25T01:16:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 msgr2=0x7f2ff4100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 --2- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 0x7f2ff4100a50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4006010 tx=0x7f2fe4005050 comp rx=0 tx=0).stop 2024-04-25T01:16:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 shutdown_connections 2024-04-25T01:16:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 --2- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2fdc0390f0 0x7f2fdc03b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 --2- 172.21.15.27:0/1181107411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ff4102df0 0x7f2ff4100a50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 >> 172.21.15.27:0/1181107411 conn(0x7f2ff40fc640 msgr2=0x7f2ff4103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 shutdown_connections 2024-04-25T01:16:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:23.898+0000 7f2ff88cf700 1 -- 172.21.15.27:0/1181107411 wait complete. 2024-04-25T01:16:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-04-25T01:16:24.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:23 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1181107411' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:16:24.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:24.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:24.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:24.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:24.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:16:24.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:24.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:24.493 INFO:tasks.cephadm:Adding mon.a on smithi027 2024-04-25T01:16:24.493 INFO:tasks.cephadm:Adding mon.b on smithi139 2024-04-25T01:16:24.493 INFO:tasks.cephadm:Adding mon.c on smithi153 2024-04-25T01:16:24.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c' 2024-04-25T01:16:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:25.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:25.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:16:25.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:25.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:25.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1181107411' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:16:25.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:25.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:16:25.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:25.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:25.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:26.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.194+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3844947559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 msgr2=0x7f36e0106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:26.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.194+0000 7f36e7a8a700 1 --2- 172.21.15.153:0/3844947559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 0x7f36e0106fd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f36d000b0a0 tx=0x7f36d000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.194+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3844947559 shutdown_connections 2024-04-25T01:16:26.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.194+0000 7f36e7a8a700 1 --2- 172.21.15.153:0/3844947559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 0x7f36e0106fd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.194+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3844947559 >> 172.21.15.153:0/3844947559 conn(0x7f36e00fc630 msgr2=0x7f36e00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.194+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3844947559 shutdown_connections 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.195+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3844947559 wait complete. 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.195+0000 7f36e7a8a700 1 Processor -- start 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.195+0000 7f36e7a8a700 1 -- start start 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.195+0000 7f36e7a8a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 0x7f36e01965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.195+0000 7f36e7a8a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f36e010b1b0 con 0x7f36e0106bf0 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.195+0000 7f36e5826700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 0x7f36e01965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.195+0000 7f36e5826700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 0x7f36e01965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.153:52388/0 (socket says 172.21.15.153:52388) 2024-04-25T01:16:26.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.195+0000 7f36e5826700 1 -- 172.21.15.153:0/3355930730 learned_addr learned my addr 172.21.15.153:0/3355930730 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:16:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.196+0000 7f36e5826700 1 -- 172.21.15.153:0/3355930730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e0196af0 con 0x7f36e0106bf0 2024-04-25T01:16:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.196+0000 7f36e5826700 1 --2- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 0x7f36e01965b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f36d00062a0 tx=0x7f36d0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.196+0000 7f36deffd700 1 -- 172.21.15.153:0/3355930730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d0017040 con 0x7f36e0106bf0 2024-04-25T01:16:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.196+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e0196d80 con 0x7f36e0106bf0 2024-04-25T01:16:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.196+0000 7f36deffd700 1 -- 172.21.15.153:0/3355930730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d0022070 con 0x7f36e0106bf0 2024-04-25T01:16:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.196+0000 7f36deffd700 1 -- 172.21.15.153:0/3355930730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d001d440 con 0x7f36e0106bf0 2024-04-25T01:16:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.196+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e019ae80 con 0x7f36e0106bf0 2024-04-25T01:16:26.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.197+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36e01021f0 con 0x7f36e0106bf0 2024-04-25T01:16:26.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.197+0000 7f36deffd700 1 -- 172.21.15.153:0/3355930730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f36d000bb90 con 0x7f36e0106bf0 2024-04-25T01:16:26.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.197+0000 7f36deffd700 1 --2- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36c80390f0 0x7f36c803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:26.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.197+0000 7f36deffd700 1 -- 172.21.15.153:0/3355930730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f36d000f050 con 0x7f36e0106bf0 2024-04-25T01:16:26.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.198+0000 7f36e5025700 1 --2- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36c80390f0 0x7f36c803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:26.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.198+0000 7f36e5025700 1 --2- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36c80390f0 0x7f36c803b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f36d4006fd0 tx=0x7f36d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:26.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.203+0000 7f36deffd700 1 -- 172.21.15.153:0/3355930730 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f36d0007e20 con 0x7f36e0106bf0 2024-04-25T01:16:26.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.410+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}) -- 0x7f36e0197910 con 0x7f36c80390f0 2024-04-25T01:16:26.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.417+0000 7f36deffd700 1 -- 172.21.15.153:0/3355930730 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f36e0197910 con 0x7f36c80390f0 2024-04-25T01:16:26.401 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-04-25T01:16:26.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36c80390f0 msgr2=0x7f36c803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 --2- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36c80390f0 0x7f36c803b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f36d4006fd0 tx=0x7f36d4006e40 comp rx=0 tx=0).stop 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 msgr2=0x7f36e01965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 --2- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 0x7f36e01965b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f36d00062a0 tx=0x7f36d0003ce0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 shutdown_connections 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 --2- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36c80390f0 0x7f36c803b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 --2- 172.21.15.153:0/3355930730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e0106bf0 0x7f36e01965b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 >> 172.21.15.153:0/3355930730 conn(0x7f36e00fc630 msgr2=0x7f36e00fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 shutdown_connections 2024-04-25T01:16:26.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:26.419+0000 7f36e7a8a700 1 -- 172.21.15.153:0/3355930730 wait complete. 2024-04-25T01:16:27.003 DEBUG:teuthology.orchestra.run.smithi139:mon.b> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T01:16:27.007 DEBUG:teuthology.orchestra.run.smithi153:mon.c> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T01:16:27.010 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:16:27.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:16:27.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:26 smithi027 ceph-mon[29197]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:27.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:27.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:27.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:27.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:27.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:27.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:27.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:28.154 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:28.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:27 smithi027 ceph-mon[29197]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:28.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:27 smithi027 ceph-mon[29197]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c;count:3 2024-04-25T01:16:28.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:27 smithi027 ceph-mon[29197]: Deploying daemon mon.c on smithi153 2024-04-25T01:16:29.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:28 smithi027 ceph-mon[29197]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:30.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.589+0000 7fb776951700 1 -- 172.21.15.153:0/745347710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 msgr2=0x7fb770102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.589+0000 7fb776951700 1 --2- 172.21.15.153:0/745347710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 0x7fb770102280 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb760009910 tx=0x7fb760009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.589+0000 7fb776951700 1 -- 172.21.15.153:0/745347710 shutdown_connections 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.589+0000 7fb776951700 1 --2- 172.21.15.153:0/745347710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 0x7fb770102280 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.589+0000 7fb776951700 1 -- 172.21.15.153:0/745347710 >> 172.21.15.153:0/745347710 conn(0x7fb7700fd770 msgr2=0x7fb7700ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.589+0000 7fb776951700 1 -- 172.21.15.153:0/745347710 shutdown_connections 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.589+0000 7fb776951700 1 -- 172.21.15.153:0/745347710 wait complete. 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.590+0000 7fb776951700 1 Processor -- start 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.590+0000 7fb776951700 1 -- start start 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.590+0000 7fb776951700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 0x7fb770074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.590+0000 7fb776951700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb77010c270 con 0x7fb770101ea0 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb76ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 0x7fb770074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb76ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 0x7fb770074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.153:43244/0 (socket says 172.21.15.153:43244) 2024-04-25T01:16:30.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb76ffff700 1 -- 172.21.15.153:0/108812020 learned_addr learned my addr 172.21.15.153:0/108812020 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:16:30.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb76ffff700 1 -- 172.21.15.153:0/108812020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb770076420 con 0x7fb770101ea0 2024-04-25T01:16:30.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb76ffff700 1 --2- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 0x7fb770074530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb760005850 tx=0x7fb760005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb76d7fa700 1 -- 172.21.15.153:0/108812020 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb760026070 con 0x7fb770101ea0 2024-04-25T01:16:30.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb770074ad0 con 0x7fb770101ea0 2024-04-25T01:16:30.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb76d7fa700 1 -- 172.21.15.153:0/108812020 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb76000bc30 con 0x7fb770101ea0 2024-04-25T01:16:30.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb770074f90 con 0x7fb770101ea0 2024-04-25T01:16:30.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.591+0000 7fb76d7fa700 1 -- 172.21.15.153:0/108812020 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb760003a10 con 0x7fb770101ea0 2024-04-25T01:16:30.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.592+0000 7fb76d7fa700 1 -- 172.21.15.153:0/108812020 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb760029070 con 0x7fb770101ea0 2024-04-25T01:16:30.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.592+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb770103380 con 0x7fb770101ea0 2024-04-25T01:16:30.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.592+0000 7fb76d7fa700 1 --2- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb7580390f0 0x7fb75803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.592+0000 7fb76d7fa700 1 -- 172.21.15.153:0/108812020 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb76001e070 con 0x7fb770101ea0 2024-04-25T01:16:30.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.593+0000 7fb76f7fe700 1 --2- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb7580390f0 0x7fb75803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.593+0000 7fb76f7fe700 1 --2- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb7580390f0 0x7fb75803b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb764006fd0 tx=0x7fb764006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.598+0000 7fb76d7fa700 1 -- 172.21.15.153:0/108812020 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb760013070 con 0x7fb770101ea0 2024-04-25T01:16:30.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.917+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb770075a60 con 0x7fb770101ea0 2024-04-25T01:16:30.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.918+0000 7fb76d7fa700 1 -- 172.21.15.153:0/108812020 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fb760028d80 con 0x7fb770101ea0 2024-04-25T01:16:30.902 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:16:30.902 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","modified":"2024-04-25T01:14:36.895688Z","created":"2024-04-25T01:14:36.895688Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T01:16:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb7580390f0 msgr2=0x7fb75803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 --2- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb7580390f0 0x7fb75803b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb764006fd0 tx=0x7fb764006e40 comp rx=0 tx=0).stop 2024-04-25T01:16:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 msgr2=0x7fb770074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 --2- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 0x7fb770074530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb760005850 tx=0x7fb760005030 comp rx=0 tx=0).stop 2024-04-25T01:16:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 shutdown_connections 2024-04-25T01:16:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 --2- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb7580390f0 0x7fb75803b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 --2- 172.21.15.153:0/108812020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb770101ea0 0x7fb770074530 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 >> 172.21.15.153:0/108812020 conn(0x7fb7700fd770 msgr2=0x7fb770103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 shutdown_connections 2024-04-25T01:16:30.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:30.921+0000 7fb776951700 1 -- 172.21.15.153:0/108812020 wait complete. 2024-04-25T01:16:30.906 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-25T01:16:31.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:30 smithi027 ceph-mon[29197]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.248 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 systemd[1]: Starting Ceph mon.c for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 podman[32603]: 2024-04-25 01:16:31.556870293 +0000 UTC m=+0.017970302 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 podman[32603]: 2024-04-25 01:16:31.658896994 +0000 UTC m=+0.119997033 container create 4c8282d0292ed28b7b19313bda77e22c9404acfcb02faf68f402193f4692236d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 podman[32603]: 2024-04-25 01:16:31.844335727 +0000 UTC m=+0.305435721 container init 4c8282d0292ed28b7b19313bda77e22c9404acfcb02faf68f402193f4692236d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9) 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 podman[32603]: 2024-04-25 01:16:31.847225232 +0000 UTC m=+0.308325229 container start 4c8282d0292ed28b7b19313bda77e22c9404acfcb02faf68f402193f4692236d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pidfile_write: ignore empty --pid-file 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: load: jerasure load: lrc 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:16:31.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Git sha 0 2024-04-25T01:16:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:16:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: DB SUMMARY 2024-04-25T01:16:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: DB Session ID: LXVLBH7I3JKF0D36YN3E 2024-04-25T01:16:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:16:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:16:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T01:16:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-25T01:16:31.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.env: 0x55fd8c8a4c60 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.info_log: 0x55fd8e5ddaa0 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:16:31.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.statistics: (nil) 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.use_fsync: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.db_log_dir: 2024-04-25T01:16:31.916 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.wal_dir: 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.write_buffer_manager: 0x55fd8e5b01e0 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:16:31.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.unordered_write: 0 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.row_cache: None 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.wal_filter: None 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:16:31.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.wal_compression: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:16:31.919 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_open_files: -1 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:16:31.920 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Compression algorithms supported: 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: kZSTD supported: 0 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:16:31.921 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.merge_operator: 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_filter: None 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:16:31.922 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fd8e5ddba0) 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_type: 0 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: data_block_index_type: 0 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_shortening: 1 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: checksum: 4 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: no_block_cache: 0 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache: 0x55fd8e5e1090 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_options: 2024-04-25T01:16:31.923 INFO:journalctl@ceph.mon.c.smithi153.stdout: capacity : 536870912 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: num_shard_bits : 4 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: strict_capacity_limit : 0 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_cache_compressed: (nil) 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: persistent_cache: (nil) 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_size: 4096 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_size_deviation: 10 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_restart_interval: 16 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: index_block_restart_interval: 1 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: metadata_block_size: 4096 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: partition_filters: 0 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: use_delta_encoding: 1 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: filter_policy: bloomfilter 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: whole_key_filtering: 1 2024-04-25T01:16:31.924 INFO:journalctl@ceph.mon.c.smithi153.stdout: verify_compression: 0 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout: format_version: 5 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout: enable_index_compression: 1 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout: block_align: 0 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout: max_auto_readahead_size: 262144 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout: prepopulate_block_cache: 0 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression: NoCompression 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:16:31.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.num_levels: 7 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:16:31.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:16:31.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:16:31.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:16:31.929 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.ttl: 2592000 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:16:31.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T01:16:31.931 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cb260565-acef-4656-9833-dbbb750b5d1e 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007791878227, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007791878768, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714007791, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cb260565-acef-4656-9833-dbbb750b5d1e", "db_session_id": "LXVLBH7I3JKF0D36YN3E", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007791878870, "job": 1, "event": "recovery_finished"} 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fd8e6d2000 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: DB pointer 0x55fd8e6bc000 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** DB Stats ** 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:16:31.932 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T01:16:31.933 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:16:31.934 INFO:journalctl@ceph.mon.c.smithi153.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout: Block cache BinnedLRUCache@0x55fd8e5e1090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: using public_addr v2:172.21.15.153:0/0 -> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: starting mon.c rank -1 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(???) e0 preinit fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).mds e1 new map 2024-04-25T01:16:31.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout: e1 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout: legacy client fscid: -1 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout: 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout: No filesystems configured 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:16:31.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Deploying cephadm binary to smithi153 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:31.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Added host smithi153 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:16:31.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1181107411' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:16:31.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1181107411' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:16:31.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c;count:3 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: Deploying daemon mon.c on smithi153 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.941 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:31.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 ceph-mon[32621]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:16:31.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 bash[32603]: 4c8282d0292ed28b7b19313bda77e22c9404acfcb02faf68f402193f4692236d 2024-04-25T01:16:31.942 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:31 smithi153 systemd[1]: Started Ceph mon.c for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:16:32.492 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:16:32.493 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:16:32.732 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:16:37.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:37.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:16:37.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: mon.a calling monitor election 2024-04-25T01:16:37.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:37.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: mon.c calling monitor election 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: monmap epoch 2 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: last_changed 2024-04-25T01:16:31.903244+0000 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: min_mon_release 19 (squid) 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: election_strategy: 1 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:37.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: fsmap 2024-04-25T01:16:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: mgrmap e13: x(active, since 51s) 2024-04-25T01:16:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: overall HEALTH_OK 2024-04-25T01:16:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:37.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.305+0000 7f150835e700 1 -- 172.21.15.153:0/1980463870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14ec0035c0 msgr2=0x7f14ec005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.305+0000 7f150835e700 1 --2- 172.21.15.153:0/1980463870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14ec0035c0 0x7f14ec005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1500100b90 tx=0x7f14fc00feb0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.305+0000 7f150835e700 1 -- 172.21.15.153:0/1980463870 shutdown_connections 2024-04-25T01:16:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.305+0000 7f150835e700 1 --2- 172.21.15.153:0/1980463870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14ec0035c0 0x7f14ec005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.305+0000 7f150835e700 1 -- 172.21.15.153:0/1980463870 >> 172.21.15.153:0/1980463870 conn(0x7f15000fc790 msgr2=0x7f15000febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.305+0000 7f150835e700 1 -- 172.21.15.153:0/1980463870 shutdown_connections 2024-04-25T01:16:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.305+0000 7f150835e700 1 -- 172.21.15.153:0/1980463870 wait complete. 2024-04-25T01:16:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f150835e700 1 Processor -- start 2024-04-25T01:16:37.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f150835e700 1 -- start start 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f150835e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 0x7f150019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f150835e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f150019db30 0x7f1500197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f150835e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1500108cf0 con 0x7f150019db30 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f150835e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1500108b70 con 0x7f14ec0035c0 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f15060fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 0x7f150019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f15060fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 0x7f150019d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33038/0 (socket says 172.21.15.153:33038) 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.306+0000 7f15060fa700 1 -- 172.21.15.153:0/2213882082 learned_addr learned my addr 172.21.15.153:0/2213882082 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15060fa700 1 -- 172.21.15.153:0/2213882082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 msgr2=0x7f150019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15060fa700 1 -- 172.21.15.153:0/2213882082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 msgr2=0x7f150019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15060fa700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 0x7f150019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:16:37.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15060fa700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 0x7f150019d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15058f9700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f150019db30 0x7f1500197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15058f9700 1 -- 172.21.15.153:0/2213882082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 msgr2=0x7f150019d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15058f9700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 0x7f150019d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15058f9700 1 -- 172.21.15.153:0/2213882082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150019e0b0 con 0x7f150019db30 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f15058f9700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f150019db30 0x7f1500197710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f14f000b4d0 tx=0x7f14f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.307+0000 7f14f77fe700 1 -- 172.21.15.153:0/2213882082 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f14f0011630 con 0x7f150019db30 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.308+0000 7f14f77fe700 1 -- 172.21.15.153:0/2213882082 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14f0011cb0 con 0x7f150019db30 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.308+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1500197cb0 con 0x7f150019db30 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.308+0000 7f14f77fe700 1 -- 172.21.15.153:0/2213882082 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f14f0010e60 con 0x7f150019db30 2024-04-25T01:16:37.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.308+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1500198240 con 0x7f150019db30 2024-04-25T01:16:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.309+0000 7f14f77fe700 1 -- 172.21.15.153:0/2213882082 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f14f0018020 con 0x7f150019db30 2024-04-25T01:16:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.309+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15001023a0 con 0x7f150019db30 2024-04-25T01:16:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.309+0000 7f14f77fe700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14e4038e40 0x7f14e403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.309+0000 7f14f77fe700 1 -- 172.21.15.153:0/2213882082 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f14f004bc80 con 0x7f150019db30 2024-04-25T01:16:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.309+0000 7f15060fa700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14e4038e40 0x7f14e403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.310+0000 7f15060fa700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14e4038e40 0x7f14e403b300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00fa60 tx=0x7f14fc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.315+0000 7f14f77fe700 1 -- 172.21.15.153:0/2213882082 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f14f001f020 con 0x7f150019db30 2024-04-25T01:16:37.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:37.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:16:37.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: mon.a calling monitor election 2024-04-25T01:16:37.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: mon.c calling monitor election 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: monmap epoch 2 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:37.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: last_changed 2024-04-25T01:16:31.903244+0000 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: min_mon_release 19 (squid) 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: election_strategy: 1 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: fsmap 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: mgrmap e13: x(active, since 51s) 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: overall HEALTH_OK 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:37.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:37.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.595+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f15001985b0 con 0x7f150019db30 2024-04-25T01:16:37.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.595+0000 7f14f77fe700 1 -- 172.21.15.153:0/2213882082 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f14f0018310 con 0x7f150019db30 2024-04-25T01:16:37.580 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:16:37.580 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","modified":"2024-04-25T01:16:31.903244Z","created":"2024-04-25T01:14:36.895688Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:16:37.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14e4038e40 msgr2=0x7f14e403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14e4038e40 0x7f14e403b300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00fa60 tx=0x7f14fc005960 comp rx=0 tx=0).stop 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f150019db30 msgr2=0x7f1500197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f150019db30 0x7f1500197710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f14f000b4d0 tx=0x7f14f000b890 comp rx=0 tx=0).stop 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 shutdown_connections 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14e4038e40 0x7f14e403b300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14ec0035c0 0x7f150019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 --2- 172.21.15.153:0/2213882082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f150019db30 0x7f1500197710 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 >> 172.21.15.153:0/2213882082 conn(0x7f15000fc790 msgr2=0x7f15001048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 shutdown_connections 2024-04-25T01:16:37.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:37.598+0000 7f150835e700 1 -- 172.21.15.153:0/2213882082 wait complete. 2024-04-25T01:16:37.583 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-04-25T01:16:38.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:37 smithi027 ceph-mon[29197]: Deploying daemon mon.b on smithi139 2024-04-25T01:16:38.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:37 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/2213882082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:16:38.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:37 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:38.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:37 smithi153 ceph-mon[32621]: Deploying daemon mon.b on smithi139 2024-04-25T01:16:38.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:37 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/2213882082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:16:38.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:37 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:39.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:38 smithi153 ceph-mon[32621]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:39.332 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:16:39.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:16:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:38 smithi027 ceph-mon[29197]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:39.572 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:16:40.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 podman[32741]: 2024-04-25 01:16:40.742642288 +0000 UTC m=+0.117802879 container create 4dc38a28c2a890fa97914af1b6b4eeddbecf67c0cec51c46506256b63bf1fe2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 -- 172.21.15.153:0/1005083936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 msgr2=0x7f7060102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 --2- 172.21.15.153:0/1005083936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 0x7f7060102390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7050009910 tx=0x7f7050009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 -- 172.21.15.153:0/1005083936 shutdown_connections 2024-04-25T01:16:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 --2- 172.21.15.153:0/1005083936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 0x7f7060102390 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 -- 172.21.15.153:0/1005083936 >> 172.21.15.153:0/1005083936 conn(0x7f70600fd840 msgr2=0x7f70600ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 -- 172.21.15.153:0/1005083936 shutdown_connections 2024-04-25T01:16:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 -- 172.21.15.153:0/1005083936 wait complete. 2024-04-25T01:16:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 Processor -- start 2024-04-25T01:16:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.942+0000 7f7066947700 1 -- start start 2024-04-25T01:16:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f7066947700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 0x7f7060074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f7066947700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 0x7f7060074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f7066947700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f706010c380 con 0x7f7060101fb0 2024-04-25T01:16:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f7066947700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f706010c200 con 0x7f7060074a70 2024-04-25T01:16:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 0x7f7060074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 0x7f7060074ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35896/0 (socket says 172.21.15.153:35896) 2024-04-25T01:16:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705f7fe700 1 -- 172.21.15.153:0/3781074938 learned_addr learned my addr 172.21.15.153:0/3781074938 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:16:40.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705ffff700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 0x7f7060074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705f7fe700 1 -- 172.21.15.153:0/3781074938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 msgr2=0x7f7060074ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T01:16:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705f7fe700 1 -- 172.21.15.153:0/3781074938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 msgr2=0x7f7060074ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:16:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705f7fe700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 0x7f7060074ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:16:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705f7fe700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 0x7f7060074ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:16:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705ffff700 1 -- 172.21.15.153:0/3781074938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 msgr2=0x7f7060074ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:16:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705ffff700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 0x7f7060074ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.943+0000 7f705ffff700 1 -- 172.21.15.153:0/3781074938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70600763b0 con 0x7f7060101fb0 2024-04-25T01:16:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.944+0000 7f705ffff700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 0x7f7060074530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7050014000 tx=0x7f7050005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.944+0000 7f705d7fa700 1 -- 172.21.15.153:0/3781074938 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7050005dc0 con 0x7f7060101fb0 2024-04-25T01:16:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.944+0000 7f705d7fa700 1 -- 172.21.15.153:0/3781074938 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f705000be40 con 0x7f7060101fb0 2024-04-25T01:16:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.944+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70601a3c90 con 0x7f7060101fb0 2024-04-25T01:16:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.944+0000 7f705d7fa700 1 -- 172.21.15.153:0/3781074938 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f705000f460 con 0x7f7060101fb0 2024-04-25T01:16:40.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.944+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70601a41a0 con 0x7f7060101fb0 2024-04-25T01:16:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.945+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7060103490 con 0x7f7060101fb0 2024-04-25T01:16:40.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.945+0000 7f705d7fa700 1 -- 172.21.15.153:0/3781074938 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f705002a070 con 0x7f7060101fb0 2024-04-25T01:16:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.945+0000 7f705d7fa700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7048038e90 0x7f704803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.945+0000 7f705d7fa700 1 -- 172.21.15.153:0/3781074938 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f705001e030 con 0x7f7060101fb0 2024-04-25T01:16:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.945+0000 7f705f7fe700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7048038e90 0x7f704803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.946+0000 7f705f7fe700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7048038e90 0x7f704803b350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7054009180 tx=0x7f7054008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:40.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:40.951+0000 7f705d7fa700 1 -- 172.21.15.153:0/3781074938 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7050013070 con 0x7f7060101fb0 2024-04-25T01:16:41.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:41.017+0000 7f705d7fa700 1 -- 172.21.15.153:0/3781074938 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70500047d0 con 0x7f7060101fb0 2024-04-25T01:16:41.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:41.227+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f70601a44c0 con 0x7f7060101fb0 2024-04-25T01:16:41.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:40 smithi153 ceph-mon[32621]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 podman[32741]: 2024-04-25 01:16:40.961439459 +0000 UTC m=+0.336600050 container init 4dc38a28c2a890fa97914af1b6b4eeddbecf67c0cec51c46506256b63bf1fe2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:16:41.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 podman[32741]: 2024-04-25 01:16:40.96430209 +0000 UTC m=+0.339462683 container start 4dc38a28c2a890fa97914af1b6b4eeddbecf67c0cec51c46506256b63bf1fe2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_CLEAN=True) 2024-04-25T01:16:41.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:16:41.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:16:41.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: pidfile_write: ignore empty --pid-file 2024-04-25T01:16:41.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: load: jerasure load: lrc 2024-04-25T01:16:41.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:16:41.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Git sha 0 2024-04-25T01:16:41.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:16:41.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: DB SUMMARY 2024-04-25T01:16:41.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: DB Session ID: OKFBOEHKN21KIRU1VA11 2024-04-25T01:16:41.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:16:41.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:16:41.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T01:16:41.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-25T01:16:41.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-25T01:16:41.300 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:16:41.300 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:16:41.300 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:16:41.300 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:16:41.300 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:16:41.300 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:16:41.301 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.env: 0x55af5aab8c60 2024-04-25T01:16:41.301 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:16:41.301 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.info_log: 0x55af5ba0baa0 2024-04-25T01:16:41.301 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:16:41.301 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.statistics: (nil) 2024-04-25T01:16:41.301 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.use_fsync: 0 2024-04-25T01:16:41.302 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:16:41.302 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:16:41.302 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:16:41.302 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:16:41.302 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:16:41.303 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:16:41.303 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:16:41.303 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:16:41.303 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:16:41.303 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:16:41.303 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:16:41.304 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.db_log_dir: 2024-04-25T01:16:41.304 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.wal_dir: 2024-04-25T01:16:41.304 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:16:41.304 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:16:41.304 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:16:41.304 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:16:41.305 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:16:41.305 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:16:41.305 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:16:41.305 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:16:41.305 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.write_buffer_manager: 0x55af5b9de1e0 2024-04-25T01:16:41.305 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:16:41.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:16:41.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:16:41.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:16:41.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:16:41.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:16:41.306 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:16:41.307 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:16:41.307 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.unordered_write: 0 2024-04-25T01:16:41.307 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:16:41.307 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:16:41.307 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:16:41.307 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:16:41.307 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.row_cache: None 2024-04-25T01:16:41.308 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.wal_filter: None 2024-04-25T01:16:41.310 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:16:41.311 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:16:41.311 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:16:41.311 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:16:41.311 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.wal_compression: 0 2024-04-25T01:16:41.311 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:16:41.311 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:16:41.312 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:16:41.312 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:16:41.312 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:16:41.312 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:16:41.312 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:16:41.312 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:16:41.313 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:16:41.313 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:16:41.313 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:16:41.313 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:16:41.313 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:16:41.313 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:16:41.314 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:16:41.314 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:16:41.314 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:16:41.314 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:16:41.314 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:16:41.314 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:16:41.315 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:16:41.315 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:16:41.315 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:16:41.315 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_open_files: -1 2024-04-25T01:16:41.315 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:16:41.315 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:16:41.316 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:16:41.316 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:16:41.316 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:16:41.316 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Compression algorithms supported: 2024-04-25T01:16:41.316 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:16:41.316 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: kZSTD supported: 0 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:16:41.317 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-25T01:16:41.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:16:41.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:16:41.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.merge_operator: 2024-04-25T01:16:41.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_filter: None 2024-04-25T01:16:41.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:16:41.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:16:41.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:16:41.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55af5ba0bba0) 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout: index_type: 0 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout: data_block_index_type: 0 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout: index_shortening: 1 2024-04-25T01:16:41.319 INFO:journalctl@ceph.mon.b.smithi139.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:16:41.320 INFO:journalctl@ceph.mon.b.smithi139.stdout: checksum: 4 2024-04-25T01:16:41.320 INFO:journalctl@ceph.mon.b.smithi139.stdout: no_block_cache: 0 2024-04-25T01:16:41.320 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_cache: 0x55af5ba0f090 2024-04-25T01:16:41.320 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:16:41.320 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_cache_options: 2024-04-25T01:16:41.320 INFO:journalctl@ceph.mon.b.smithi139.stdout: capacity : 536870912 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: num_shard_bits : 4 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: strict_capacity_limit : 0 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_cache_compressed: (nil) 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: persistent_cache: (nil) 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_size: 4096 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_size_deviation: 10 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_restart_interval: 16 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: index_block_restart_interval: 1 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: metadata_block_size: 4096 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: partition_filters: 0 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: use_delta_encoding: 1 2024-04-25T01:16:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout: filter_policy: bloomfilter 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: whole_key_filtering: 1 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: verify_compression: 0 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: format_version: 5 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: enable_index_compression: 1 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_align: 0 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: max_auto_readahead_size: 262144 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: prepopulate_block_cache: 0 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:16:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression: NoCompression 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.num_levels: 7 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:16:41.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:16:41.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:16:41.325 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:16:41.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:16:41.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.ttl: 2592000 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:16:41.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 77c04cb4-6ca2-49b3-bfc7-ab2e57ed7f64 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007800994844, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007800995255, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714007800, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "77c04cb4-6ca2-49b3-bfc7-ab2e57ed7f64", "db_session_id": "OKFBOEHKN21KIRU1VA11", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007800995341, "job": 1, "event": "recovery_finished"} 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55af5bb00000 2024-04-25T01:16:41.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: DB pointer 0x55af5baea000 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** DB Stats ** 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** Compaction Stats [default] ** 2024-04-25T01:16:41.330 INFO:journalctl@ceph.mon.b.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** Compaction Stats [default] ** 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:16:41.331 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: Block cache BinnedLRUCache@0x55af5ba0f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: using public_addr v2:172.21.15.139:0/0 -> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: starting mon.b rank -1 at public addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] at bind addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:41.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:40 smithi139 ceph-mon[32760]: mon.b@-1(???) e0 preinit fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: e1 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: legacy client fscid: -1 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: No filesystems configured 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:16:41.333 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Deploying cephadm binary to smithi153 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.334 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Added host smithi153 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1181107411' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:41.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1181107411' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:41.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c;count:3 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Deploying daemon mon.c on smithi153 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.a calling monitor election 2024-04-25T01:16:41.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.c calling monitor election 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: monmap epoch 2 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: last_changed 2024-04-25T01:16:31.903244+0000 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:41.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: min_mon_release 19 (squid) 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: election_strategy: 1 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: fsmap 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mgrmap e13: x(active, since 51s) 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: overall HEALTH_OK 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: Deploying daemon mon.b on smithi139 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/2213882082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:41.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:41.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 ceph-mon[32760]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:16:41.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 bash[32741]: 4dc38a28c2a890fa97914af1b6b4eeddbecf67c0cec51c46506256b63bf1fe2e 2024-04-25T01:16:41.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:41 smithi139 systemd[1]: Started Ceph mon.b for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:16:41.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:40 smithi027 ceph-mon[29197]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:46.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.026+0000 7f705d7fa700 1 -- 172.21.15.153:0/3781074938 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f7050031020 con 0x7f7060101fb0 2024-04-25T01:16:46.010 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-25T01:16:46.010 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":3,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","modified":"2024-04-25T01:16:41.016790Z","created":"2024-04-25T01:14:36.895688Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:16:46.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7048038e90 msgr2=0x7f704803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:46.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7048038e90 0x7f704803b350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7054009180 tx=0x7f7054008040 comp rx=0 tx=0).stop 2024-04-25T01:16:46.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 msgr2=0x7f7060074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:46.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 0x7f7060074530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7050014000 tx=0x7f7050005030 comp rx=0 tx=0).stop 2024-04-25T01:16:46.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 shutdown_connections 2024-04-25T01:16:46.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7048038e90 0x7f704803b350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:46.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7060101fb0 0x7f7060074530 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:46.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 --2- 172.21.15.153:0/3781074938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7060074a70 0x7f7060074ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:46.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.029+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 >> 172.21.15.153:0/3781074938 conn(0x7f70600fd840 msgr2=0x7f7060103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:46.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.030+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 shutdown_connections 2024-04-25T01:16:46.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:46.030+0000 7f7066947700 1 -- 172.21.15.153:0/3781074938 wait complete. 2024-04-25T01:16:46.014 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 3 2024-04-25T01:16:46.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:16:46.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: mon.a calling monitor election 2024-04-25T01:16:46.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: mon.c calling monitor election 2024-04-25T01:16:46.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:46.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/3781074938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:16:46.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: monmap epoch 3 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: last_changed 2024-04-25T01:16:41.016790+0000 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: min_mon_release 19 (squid) 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: election_strategy: 1 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:46.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: fsmap 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: mgrmap e13: x(active, since 60s) 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: overall HEALTH_OK 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:46.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: mon.a calling monitor election 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: mon.c calling monitor election 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/3781074938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: monmap epoch 3 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: last_changed 2024-04-25T01:16:41.016790+0000 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: min_mon_release 19 (squid) 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: election_strategy: 1 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:46.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T01:16:46.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: fsmap 2024-04-25T01:16:46.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:46.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: mgrmap e13: x(active, since 60s) 2024-04-25T01:16:46.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: overall HEALTH_OK 2024-04-25T01:16:46.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:46.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:46.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:46.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:46.607 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T01:16:46.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T01:16:46.863 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:47.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:47 smithi153 ceph-mon[32621]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:47.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:47.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:47.311 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:47 smithi027 ceph-mon[29197]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:47.311 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:47.311 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:48.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: mon.a calling monitor election 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: mon.c calling monitor election 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/3781074938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:48.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: monmap epoch 3 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: last_changed 2024-04-25T01:16:41.016790+0000 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: min_mon_release 19 (squid) 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: election_strategy: 1 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: fsmap 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: mgrmap e13: x(active, since 60s) 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: overall HEALTH_OK 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:48.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:48.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:48.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:48.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:48.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:48.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:48.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.575+0000 7fee0faeb700 1 -- 172.21.15.27:0/1364535252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 msgr2=0x7fee08101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.575+0000 7fee0faeb700 1 --2- 172.21.15.27:0/1364535252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 0x7fee08101180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fee04009910 tx=0x7fee04009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.575+0000 7fee0faeb700 1 -- 172.21.15.27:0/1364535252 shutdown_connections 2024-04-25T01:16:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.575+0000 7fee0faeb700 1 --2- 172.21.15.27:0/1364535252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 0x7fee08101180 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.575+0000 7fee0faeb700 1 -- 172.21.15.27:0/1364535252 >> 172.21.15.27:0/1364535252 conn(0x7fee080fc630 msgr2=0x7fee080fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.576+0000 7fee0faeb700 1 -- 172.21.15.27:0/1364535252 shutdown_connections 2024-04-25T01:16:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.576+0000 7fee0faeb700 1 -- 172.21.15.27:0/1364535252 wait complete. 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.576+0000 7fee0faeb700 1 Processor -- start 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.576+0000 7fee0faeb700 1 -- start start 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0faeb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 0x7fee08198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0faeb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08198d40 0x7fee081991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0faeb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee081a1870 0x7fee081a3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0faeb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee0810b170 con 0x7fee08100da0 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0faeb700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee0810aff0 con 0x7fee081a1870 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0faeb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee0810b2f0 con 0x7fee08198d40 2024-04-25T01:16:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d887700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 0x7fee08198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d887700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 0x7fee08198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44852/0 (socket says 172.21.15.27:44852) 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d887700 1 -- 172.21.15.27:0/478309493 learned_addr learned my addr 172.21.15.27:0/478309493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0e088700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee081a1870 0x7fee081a3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d086700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08198d40 0x7fee081991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d887700 1 -- 172.21.15.27:0/478309493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08198d40 msgr2=0x7fee081991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d887700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08198d40 0x7fee081991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d887700 1 -- 172.21.15.27:0/478309493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee081a1870 msgr2=0x7fee081a3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d887700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee081a1870 0x7fee081a3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d887700 1 -- 172.21.15.27:0/478309493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee081a44a0 con 0x7fee08100da0 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0e088700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee081a1870 0x7fee081a3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.577+0000 7fee0d086700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08198d40 0x7fee081991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.578+0000 7fee0d887700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 0x7fee08198800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee04005850 tx=0x7fee04005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.578+0000 7fedfeffd700 1 -- 172.21.15.27:0/478309493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04026070 con 0x7fee08100da0 2024-04-25T01:16:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.578+0000 7fedfeffd700 1 -- 172.21.15.27:0/478309493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0400b6f0 con 0x7fee08100da0 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.578+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee081a4790 con 0x7fee08100da0 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.578+0000 7fedfeffd700 1 -- 172.21.15.27:0/478309493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04003a00 con 0x7fee08100da0 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.578+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee081a4cd0 con 0x7fee08100da0 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.579+0000 7fedfeffd700 1 -- 172.21.15.27:0/478309493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fee04029070 con 0x7fee08100da0 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.579+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee08063890 con 0x7fee08100da0 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.579+0000 7fedfeffd700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fedf4039580 0x7fedf403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.579+0000 7fedfeffd700 1 -- 172.21.15.27:0/478309493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fee0404ce10 con 0x7fee08100da0 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.579+0000 7fee0d086700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fedf4039580 0x7fedf403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.580+0000 7fee0d086700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fedf4039580 0x7fedf403ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009910 tx=0x7fedf8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:48.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.584+0000 7fedfeffd700 1 -- 172.21.15.27:0/478309493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fee04013070 con 0x7fee08100da0 2024-04-25T01:16:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.800+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fee081a4f30 con 0x7fee08100da0 2024-04-25T01:16:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.801+0000 7fedfeffd700 1 -- 172.21.15.27:0/478309493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fee04030070 con 0x7fee08100da0 2024-04-25T01:16:48.785 INFO:teuthology.orchestra.run.smithi027.stdout:# minimal ceph.conf for 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:48.785 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-04-25T01:16:48.785 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:48.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon_host = [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-04-25T01:16:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fedf4039580 msgr2=0x7fedf403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fedf4039580 0x7fedf403ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009910 tx=0x7fedf8008040 comp rx=0 tx=0).stop 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 msgr2=0x7fee08198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 0x7fee08198800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee04005850 tx=0x7fee04005030 comp rx=0 tx=0).stop 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 shutdown_connections 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fedf4039580 0x7fedf403ba40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee081a1870 0x7fee081a3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee08100da0 0x7fee08198800 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 --2- 172.21.15.27:0/478309493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08198d40 0x7fee081991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 >> 172.21.15.27:0/478309493 conn(0x7fee080fc630 msgr2=0x7fee08102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 shutdown_connections 2024-04-25T01:16:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:48.803+0000 7fee0faeb700 1 -- 172.21.15.27:0/478309493 wait complete. 2024-04-25T01:16:49.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: mon.b calling monitor election 2024-04-25T01:16:49.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:49.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T01:16:49.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:16:49.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:16:49.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: mon.b calling monitor election 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: mon.c calling monitor election 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: mon.a calling monitor election 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: monmap epoch 3 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: last_changed 2024-04-25T01:16:41.016790+0000 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: min_mon_release 19 (squid) 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: election_strategy: 1 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:49.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: fsmap 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: mgrmap e13: x(active, since 62s) 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: overall HEALTH_OK 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:16:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: Reconfiguring daemon mon.a on smithi027 2024-04-25T01:16:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/478309493' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:49.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: mon.b calling monitor election 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: mon.b calling monitor election 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: mon.c calling monitor election 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: mon.a calling monitor election 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: monmap epoch 3 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: last_changed 2024-04-25T01:16:41.016790+0000 2024-04-25T01:16:49.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: min_mon_release 19 (squid) 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: election_strategy: 1 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: fsmap 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: mgrmap e13: x(active, since 62s) 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: overall HEALTH_OK 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.298 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: Reconfiguring daemon mon.a on smithi027 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/478309493' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:49.299 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:49.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: mon.b calling monitor election 2024-04-25T01:16:49.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:49.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T01:16:49.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:16:49.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:16:49.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: mon.b calling monitor election 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: mon.c calling monitor election 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: mon.a calling monitor election 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: monmap epoch 3 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: last_changed 2024-04-25T01:16:41.016790+0000 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: created 2024-04-25T01:14:36.895688+0000 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: min_mon_release 19 (squid) 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: election_strategy: 1 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: 2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: fsmap 2024-04-25T01:16:49.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: mgrmap e13: x(active, since 62s) 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: overall HEALTH_OK 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:49.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:49.386 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: Reconfiguring daemon mon.a on smithi027 2024-04-25T01:16:49.386 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/478309493' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:49.386 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:16:49.411 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T01:16:49.411 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:16:49.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:16:49.439 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:16:49.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:49.504 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:16:49.505 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:16:49.537 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:16:49.537 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:49.604 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:16:49.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:16:49.637 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:16:49.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:49.706 INFO:tasks.cephadm:Adding mgr.x on smithi027 2024-04-25T01:16:49.706 INFO:tasks.cephadm:Adding mgr.y on smithi139 2024-04-25T01:16:49.706 INFO:tasks.cephadm:Adding mgr.z on smithi153 2024-04-25T01:16:49.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi027=x;smithi139=y;smithi153=z' 2024-04-25T01:16:49.981 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:16:50.422 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:16:50 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:16:50.017+0000 7f3d0ff36700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T01:16:51.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:51 smithi153 ceph-mon[32621]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:51.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:51.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:51.257 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:51.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:51 smithi139 ceph-mon[32760]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:51.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:51.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:51.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:51.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:51 smithi027 ceph-mon[29197]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:51.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:51.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:51.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:51.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:51.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.482+0000 7f014391d700 1 -- 172.21.15.153:0/2117629035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c101dc0 msgr2=0x7f013c1021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.482+0000 7f014391d700 1 --2- 172.21.15.153:0/2117629035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c101dc0 0x7f013c1021e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0138009910 tx=0x7f0138009c20 comp rx=0 tx=0).stop 2024-04-25T01:16:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.482+0000 7f014391d700 1 -- 172.21.15.153:0/2117629035 shutdown_connections 2024-04-25T01:16:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.482+0000 7f014391d700 1 --2- 172.21.15.153:0/2117629035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c101dc0 0x7f013c1021e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.482+0000 7f014391d700 1 -- 172.21.15.153:0/2117629035 >> 172.21.15.153:0/2117629035 conn(0x7f013c0fd360 msgr2=0x7f013c0ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.482+0000 7f014391d700 1 -- 172.21.15.153:0/2117629035 shutdown_connections 2024-04-25T01:16:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.482+0000 7f014391d700 1 -- 172.21.15.153:0/2117629035 wait complete. 2024-04-25T01:16:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.482+0000 7f014391d700 1 Processor -- start 2024-04-25T01:16:51.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f014391d700 1 -- start start 2024-04-25T01:16:51.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f014391d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c101dc0 0x7f013c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f014391d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f013c197b20 0x7f013c19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f014391d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c197fa0 0x7f013c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f014391d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f013c107e00 con 0x7f013c197fa0 2024-04-25T01:16:51.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f014391d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f013c107c80 con 0x7f013c101dc0 2024-04-25T01:16:51.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f014391d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f013c107f80 con 0x7f013c197b20 2024-04-25T01:16:51.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c101dc0 0x7f013c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f0141eba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c197fa0 0x7f013c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c101dc0 0x7f013c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.153:40186/0 (socket says 172.21.15.153:40186) 2024-04-25T01:16:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 -- 172.21.15.153:0/1604385822 learned_addr learned my addr 172.21.15.153:0/1604385822 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:16:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f0140eb8700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f013c197b20 0x7f013c19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 -- 172.21.15.153:0/1604385822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f013c197b20 msgr2=0x7f013c19cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f013c197b20 0x7f013c19cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 -- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c197fa0 msgr2=0x7f013c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c197fa0 0x7f013c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 -- 172.21.15.153:0/1604385822 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c19f9f0 con 0x7f013c101dc0 2024-04-25T01:16:51.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f0141eba700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c197fa0 0x7f013c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:51.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.483+0000 7f01416b9700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c101dc0 0x7f013c1975e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0138000c00 tx=0x7f0138005000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:51.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.484+0000 7f01327fc700 1 -- 172.21.15.153:0/1604385822 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138027070 con 0x7f013c101dc0 2024-04-25T01:16:51.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.484+0000 7f01327fc700 1 -- 172.21.15.153:0/1604385822 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0138022470 con 0x7f013c101dc0 2024-04-25T01:16:51.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.484+0000 7f01327fc700 1 -- 172.21.15.153:0/1604385822 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013800f6c0 con 0x7f013c101dc0 2024-04-25T01:16:51.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.484+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013c19fc80 con 0x7f013c101dc0 2024-04-25T01:16:51.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.484+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013c1a0110 con 0x7f013c101dc0 2024-04-25T01:16:51.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.485+0000 7f01327fc700 1 -- 172.21.15.153:0/1604385822 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f013802a070 con 0x7f013c101dc0 2024-04-25T01:16:51.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.485+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013c063890 con 0x7f013c101dc0 2024-04-25T01:16:51.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.485+0000 7f01327fc700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0128039580 0x7f012803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.485+0000 7f01327fc700 1 -- 172.21.15.153:0/1604385822 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f013804d320 con 0x7f013c101dc0 2024-04-25T01:16:51.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.485+0000 7f0140eb8700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0128039580 0x7f012803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.485+0000 7f0140eb8700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0128039580 0x7f012803ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f012c009ad0 tx=0x7f012c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:51.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.490+0000 7f01327fc700 1 -- 172.21.15.153:0/1604385822 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0138013070 con 0x7f013c101dc0 2024-04-25T01:16:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.697+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi139=y;smithi153=z", "target": ["mon-mgr", ""]}) -- 0x7f013c062700 con 0x7f0128039580 2024-04-25T01:16:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.703+0000 7f01327fc700 1 -- 172.21.15.153:0/1604385822 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f013c062700 con 0x7f0128039580 2024-04-25T01:16:51.688 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mgr update... 2024-04-25T01:16:51.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.705+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0128039580 msgr2=0x7f012803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.705+0000 7f014391d700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0128039580 0x7f012803ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f012c009ad0 tx=0x7f012c009210 comp rx=0 tx=0).stop 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c101dc0 msgr2=0x7f013c1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c101dc0 0x7f013c1975e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0138000c00 tx=0x7f0138005000 comp rx=0 tx=0).stop 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 shutdown_connections 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0128039580 0x7f012803ba40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f013c197fa0 0x7f013c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c101dc0 0x7f013c1975e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 --2- 172.21.15.153:0/1604385822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f013c197b20 0x7f013c19cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 >> 172.21.15.153:0/1604385822 conn(0x7f013c0fd360 msgr2=0x7f013c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 shutdown_connections 2024-04-25T01:16:51.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:16:51.706+0000 7f014391d700 1 -- 172.21.15.153:0/1604385822 wait complete. 2024-04-25T01:16:52.092 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:52 smithi139 ceph-mon[32760]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:16:52.093 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:52 smithi139 ceph-mon[32760]: Reconfiguring daemon mon.b on smithi139 2024-04-25T01:16:52.093 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:52 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.093 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:52 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.093 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:52 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.093 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:52 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:52.093 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:52 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:52.093 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:52 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:52.230 DEBUG:teuthology.orchestra.run.smithi139:mgr.y> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T01:16:52.232 DEBUG:teuthology.orchestra.run.smithi153:mgr.z> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T01:16:52.238 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T01:16:52.238 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:16:52.238 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:16:52.258 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:16:52.258 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-04-25T01:16:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:16:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:52.317 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-25T01:16:52.317 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:52.317 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:52.317 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 01:15:58.011354114 +0000 2024-04-25T01:16:52.317 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 01:11:00.911771765 +0000 2024-04-25T01:16:52.317 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 01:11:00.911771765 +0000 2024-04-25T01:16:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 01:11:00.911771765 +0000 2024-04-25T01:16:52.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:16:52.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:52 smithi027 ceph-mon[29197]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:16:52.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:52 smithi027 ceph-mon[29197]: Reconfiguring daemon mon.b on smithi139 2024-04-25T01:16:52.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:52 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:52 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:52 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:52 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:52.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:52 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:52.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:52 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T01:16:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T01:16:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000219334 s, 2.3 MB/s 2024-04-25T01:16:52.387 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:16:52.446 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-04-25T01:16:52.492 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:52 smithi153 ceph-mon[32621]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:16:52.492 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:52 smithi153 ceph-mon[32621]: Reconfiguring daemon mon.b on smithi139 2024-04-25T01:16:52.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:52 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:52 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:52 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:52.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:52 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:16:52.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:52 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:16:52.493 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:52 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:16:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:52.504 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T01:16:52.504 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:52.504 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:52.504 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 01:15:58.046353215 +0000 2024-04-25T01:16:52.504 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 01:11:01.259762864 +0000 2024-04-25T01:16:52.504 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 01:11:01.259762864 +0000 2024-04-25T01:16:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 01:11:01.259762864 +0000 2024-04-25T01:16:52.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:16:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T01:16:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T01:16:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000203444 s, 2.5 MB/s 2024-04-25T01:16:52.571 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:16:52.629 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-04-25T01:16:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:16:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:52.685 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T01:16:52.685 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:52.685 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:52.685 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 01:15:58.075352470 +0000 2024-04-25T01:16:52.685 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 01:11:01.606753987 +0000 2024-04-25T01:16:52.686 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 01:11:01.606753987 +0000 2024-04-25T01:16:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 01:11:01.606753987 +0000 2024-04-25T01:16:52.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:16:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T01:16:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T01:16:52.750 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000199644 s, 2.6 MB/s 2024-04-25T01:16:52.751 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:16:52.808 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-04-25T01:16:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:16:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:52.865 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-25T01:16:52.865 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:52.865 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:52.865 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 01:15:58.102351776 +0000 2024-04-25T01:16:52.865 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 01:11:01.967744753 +0000 2024-04-25T01:16:52.865 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 01:11:01.967744753 +0000 2024-04-25T01:16:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 01:11:01.967744753 +0000 2024-04-25T01:16:52.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:16:52.928 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T01:16:52.928 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T01:16:52.928 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000172146 s, 3.0 MB/s 2024-04-25T01:16:52.929 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:16:52.986 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-04-25T01:16:52.986 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:16:53.005 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:16:53.005 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-04-25 01:16:40.429289246 +0000 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-04-25 01:11:16.403359254 +0000 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-04-25 01:11:16.403359254 +0000 2024-04-25T01:16:53.064 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-04-25 01:11:16.403359254 +0000 2024-04-25T01:16:53.064 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:16:53.128 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-04-25T01:16:53.129 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-04-25T01:16:53.129 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000263207 s, 1.9 MB/s 2024-04-25T01:16:53.130 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:16:53.187 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-04-25 01:16:40.429289246 +0000 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-04-25 01:11:16.746352255 +0000 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-04-25 01:11:16.746352255 +0000 2024-04-25T01:16:53.243 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-04-25 01:11:16.746352255 +0000 2024-04-25T01:16:53.244 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:16:53.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:53 smithi139 ceph-mon[32760]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:53.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:53 smithi139 ceph-mon[32760]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi139=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:53.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:53 smithi139 ceph-mon[32760]: Saving service mgr spec with placement smithi027=x;smithi139=y;smithi153=z;count:3 2024-04-25T01:16:53.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:53 smithi139 ceph-mon[32760]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:16:53.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:53 smithi139 ceph-mon[32760]: Reconfiguring daemon mon.c on smithi153 2024-04-25T01:16:53.307 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-04-25T01:16:53.307 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-04-25T01:16:53.307 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000257528 s, 2.0 MB/s 2024-04-25T01:16:53.308 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:16:53.365 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-04-25 01:16:40.429289246 +0000 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-04-25 01:11:17.084345359 +0000 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-04-25 01:11:17.084345359 +0000 2024-04-25T01:16:53.421 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-04-25 01:11:17.084345359 +0000 2024-04-25T01:16:53.422 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:16:53.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:53 smithi027 ceph-mon[29197]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:53.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:53 smithi027 ceph-mon[29197]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi139=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:53.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:53 smithi027 ceph-mon[29197]: Saving service mgr spec with placement smithi027=x;smithi139=y;smithi153=z;count:3 2024-04-25T01:16:53.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:53 smithi027 ceph-mon[29197]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:16:53.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:53 smithi027 ceph-mon[29197]: Reconfiguring daemon mon.c on smithi153 2024-04-25T01:16:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-04-25T01:16:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-04-25T01:16:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000195568 s, 2.6 MB/s 2024-04-25T01:16:53.485 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:16:53.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:53 smithi153 ceph-mon[32621]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:53.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:53 smithi153 ceph-mon[32621]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi139=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:16:53.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:53 smithi153 ceph-mon[32621]: Saving service mgr spec with placement smithi027=x;smithi139=y;smithi153=z;count:3 2024-04-25T01:16:53.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:53 smithi153 ceph-mon[32621]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:16:53.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:53 smithi153 ceph-mon[32621]: Reconfiguring daemon mon.c on smithi153 2024-04-25T01:16:53.542 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-04-25 01:16:40.430289226 +0000 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-04-25 01:11:17.416338584 +0000 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-04-25 01:11:17.416338584 +0000 2024-04-25T01:16:53.597 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-04-25 01:11:17.416338584 +0000 2024-04-25T01:16:53.598 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:16:53.659 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-04-25T01:16:53.659 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-04-25T01:16:53.659 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000223537 s, 2.3 MB/s 2024-04-25T01:16:53.660 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:16:53.716 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-25T01:16:53.716 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:16:53.732 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:16:53.733 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-04-25T01:16:53.789 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:16:53.789 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:53.789 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T01:16:53.789 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:53.789 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:53.789 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-25 01:16:30.974358218 +0000 2024-04-25T01:16:53.790 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-25 01:11:31.186379871 +0000 2024-04-25T01:16:53.790 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-25 01:11:31.186379871 +0000 2024-04-25T01:16:53.790 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-25 01:11:31.186379871 +0000 2024-04-25T01:16:53.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:16:53.854 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-25T01:16:53.854 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-25T01:16:53.854 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000248287 s, 2.1 MB/s 2024-04-25T01:16:53.855 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:16:53.912 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-04-25T01:16:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:16:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:53.967 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T01:16:53.967 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:53.967 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:53.967 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-25 01:16:30.974358218 +0000 2024-04-25T01:16:53.967 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-25 01:11:31.530373740 +0000 2024-04-25T01:16:53.967 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-25 01:11:31.530373740 +0000 2024-04-25T01:16:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-25 01:11:31.530373740 +0000 2024-04-25T01:16:53.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:16:54.030 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-25T01:16:54.030 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-25T01:16:54.030 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000188846 s, 2.7 MB/s 2024-04-25T01:16:54.032 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:16:54.089 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-04-25T01:16:54.144 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:16:54.144 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:54.144 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T01:16:54.144 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:54.145 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:54.145 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-25 01:16:30.975358201 +0000 2024-04-25T01:16:54.145 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-25 01:11:31.877367554 +0000 2024-04-25T01:16:54.145 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-25 01:11:31.877367554 +0000 2024-04-25T01:16:54.145 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-25 01:11:31.877367554 +0000 2024-04-25T01:16:54.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:16:54.207 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-25T01:16:54.207 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-25T01:16:54.208 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000190077 s, 2.7 MB/s 2024-04-25T01:16:54.209 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:16:54.265 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-04-25T01:16:54.320 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:16:54.320 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:16:54.320 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T01:16:54.320 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:16:54.320 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:16:54.320 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-25 01:16:30.975358201 +0000 2024-04-25T01:16:54.321 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-25 01:11:32.216361512 +0000 2024-04-25T01:16:54.321 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-25 01:11:32.216361512 +0000 2024-04-25T01:16:54.321 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-25 01:11:32.216361512 +0000 2024-04-25T01:16:54.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:16:54.383 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-25T01:16:54.383 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-25T01:16:54.383 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.00019487 s, 2.6 MB/s 2024-04-25T01:16:54.385 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:16:54.442 INFO:tasks.cephadm:Deploying osd.0 on smithi027 with /dev/vg_nvme/lv_4... 2024-04-25T01:16:54.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:16:54.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:16:54.531 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:16:54.531 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:54.531 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:54 smithi153 ceph-mon[32621]: Deploying daemon mgr.z on smithi153 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:16:54.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:16:54.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:54.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:54 smithi139 ceph-mon[32760]: Deploying daemon mgr.z on smithi153 2024-04-25T01:16:54.683 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:54.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:16:54.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:54.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:54.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:16:54.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:54.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:16:54.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:16:54.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:16:54.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:54.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:54 smithi027 ceph-mon[29197]: Deploying daemon mgr.z on smithi153 2024-04-25T01:16:56.030 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:55 smithi153 systemd[1]: Starting Ceph mgr.z for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:16:56.302 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 podman[34455]: 2024-04-25 01:16:56.014637589 +0000 UTC m=+0.018190722 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:16:56.302 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 podman[34455]: 2024-04-25 01:16:56.140632512 +0000 UTC m=+0.144185628 container create f231c531f574c160c41e68ee25cecf7ac1cb5cc7aca806a4a55823610ab95c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T01:16:56.302 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 podman[34455]: 2024-04-25 01:16:56.317797114 +0000 UTC m=+0.321350229 container init f231c531f574c160c41e68ee25cecf7ac1cb5cc7aca806a4a55823610ab95c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-25T01:16:56.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:56 smithi027 ceph-mon[29197]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:56.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:56 smithi153 ceph-mon[32621]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:56.646 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 podman[34455]: 2024-04-25 01:16:56.320964803 +0000 UTC m=+0.324517921 container start f231c531f574c160c41e68ee25cecf7ac1cb5cc7aca806a4a55823610ab95c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T01:16:56.647 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 ceph-mgr[34474]: -- 172.21.15.153:0/241364203 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563cc01ee5a0 con 0x563cc01c3000 2024-04-25T01:16:56.647 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 bash[34455]: f231c531f574c160c41e68ee25cecf7ac1cb5cc7aca806a4a55823610ab95c3e 2024-04-25T01:16:56.647 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 systemd[1]: Started Ceph mgr.z for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:16:56.647 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:56.470+0000 7f4bb67a5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:16:56.647 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:56.661+0000 7f4bb67a5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:16:56.742 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:16:56.742 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:16:56.742 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T01:16:56.743 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T01:16:56.743 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111571 s, 940 MB/s 2024-04-25T01:16:56.743 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T01:16:56.798 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:56 smithi139 ceph-mon[32760]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:57.029 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:56 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:56.763+0000 7f4bb67a5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:16:57.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_4 2024-04-25T01:16:57.486 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:57.072+0000 7f4bb67a5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:16:57.487 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:57.155+0000 7f4bb67a5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:16:57.658 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:16:57.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:16:57.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:16:57.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:16:57.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:57.700 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:57 smithi027 ceph-mon[29197]: Deploying daemon mgr.y on smithi139 2024-04-25T01:16:57.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:16:57.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:16:57.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:16:57.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:57.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:57 smithi153 ceph-mon[32621]: Deploying daemon mgr.y on smithi139 2024-04-25T01:16:57.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.798 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.798 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:16:57.798 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:16:57.798 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:16:57.798 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:16:57.798 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:16:57.798 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:57 smithi139 ceph-mon[32760]: Deploying daemon mgr.y on smithi139 2024-04-25T01:16:58.705 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:16:58 smithi153 ceph-mon[32621]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:58.706 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:58.377+0000 7f4bb67a5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:16:58.706 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:58.546+0000 7f4bb67a5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:16:58.706 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:58.721+0000 7f4bb67a5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:16:58.746 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:16:58 smithi139 ceph-mon[32760]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:58.799 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:16:58 smithi027 ceph-mon[29197]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:16:59.015 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:16:59 smithi139 ceph-mgr[34147]: -- 172.21.15.139:0/1858721758 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557fd83545a0 con 0x557fd8328800 2024-04-25T01:16:59.279 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:59.005+0000 7f4bb67a5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:16:59.280 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:59.096+0000 7f4bb67a5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:16:59.296 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:16:59 smithi139 bash[34128]: 12df09f15ded9efe8bc0638105350d97d2ce7e9c58abc3cff67197040a56c6ee 2024-04-25T01:16:59.296 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:16:59 smithi139 systemd[1]: Started Ceph mgr.y for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:16:59.296 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:16:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:16:59.131+0000 7fdd0acf0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:16:59.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.361+0000 7fd0c0337700 1 -- 172.21.15.27:0/3080158130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8103490 msgr2=0x7fd0b81038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.361+0000 7fd0c0337700 1 --2- 172.21.15.27:0/3080158130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8103490 0x7fd0b81038f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80099c0 tx=0x7fd0a8009cd0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 -- 172.21.15.27:0/3080158130 shutdown_connections 2024-04-25T01:16:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 --2- 172.21.15.27:0/3080158130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0b8103e30 0x7fd0b810a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 --2- 172.21.15.27:0/3080158130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0b8102290 0x7fd0b81026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 --2- 172.21.15.27:0/3080158130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8103490 0x7fd0b81038f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 -- 172.21.15.27:0/3080158130 >> 172.21.15.27:0/3080158130 conn(0x7fd0b80fd830 msgr2=0x7fd0b80ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 -- 172.21.15.27:0/3080158130 shutdown_connections 2024-04-25T01:16:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 -- 172.21.15.27:0/3080158130 wait complete. 2024-04-25T01:16:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 Processor -- start 2024-04-25T01:16:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.362+0000 7fd0c0337700 1 -- start start 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0c0337700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8102290 0x7fd0b8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0c0337700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0b8103490 0x7fd0b81989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0c0337700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0b8103e30 0x7fd0b819fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0c0337700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b810de50 con 0x7fd0b8102290 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0c0337700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b810dcd0 con 0x7fd0b8103e30 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0c0337700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b810dfd0 con 0x7fd0b8103490 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8102290 0x7fd0b8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8102290 0x7fd0b8198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35366/0 (socket says 172.21.15.27:35366) 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 -- 172.21.15.27:0/350119678 learned_addr learned my addr 172.21.15.27:0/350119678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0bd8d2700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0b8103490 0x7fd0b81989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be8d4700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0b8103e30 0x7fd0b819fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 -- 172.21.15.27:0/350119678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0b8103490 msgr2=0x7fd0b81989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0b8103490 0x7fd0b81989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 -- 172.21.15.27:0/350119678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0b8103e30 msgr2=0x7fd0b819fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0b8103e30 0x7fd0b819fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 -- 172.21.15.27:0/350119678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b81a00f0 con 0x7fd0b8102290 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0bd8d2700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0b8103490 0x7fd0b81989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0be0d3700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8102290 0x7fd0b8198490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400d9b0 tx=0x7fd0b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4016070 con 0x7fd0b8102290 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0b400be30 con 0x7fd0b8102290 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.363+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b81a03e0 con 0x7fd0b8102290 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.364+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b400f3c0 con 0x7fd0b8102290 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.364+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b81a0970 con 0x7fd0b8102290 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.365+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd0b4019070 con 0x7fd0b8102290 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.365+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b8068f20 con 0x7fd0b8102290 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.365+0000 7fd0af7fe700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd0a4039330 0x7fd0a403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.365+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd0b404c1a0 con 0x7fd0b8102290 2024-04-25T01:16:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.365+0000 7fd0bd8d2700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd0a4039330 0x7fd0a403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.366+0000 7fd0bd8d2700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd0a4039330 0x7fd0a403b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8000c00 tx=0x7fd0a8003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.370+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd0b4020020 con 0x7fd0b8102290 2024-04-25T01:16:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:16:59.586+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd0b8062670 con 0x7fd0a4039330 2024-04-25T01:16:59.622 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:16:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:16:59.321+0000 7fdd0acf0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:16:59.623 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:16:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:16:59.433+0000 7fdd0acf0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:16:59.779 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:59.302+0000 7f4bb67a5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:16:59.780 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:16:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:16:59.502+0000 7f4bb67a5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:17:00.046 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:16:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:16:59.778+0000 7fdd0acf0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:17:00.046 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:16:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:16:59.885+0000 7fdd0acf0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:17:00.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:00.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:00.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:00.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:00.529 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:00.138+0000 7f4bb67a5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:17:00.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:00.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:00.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:00.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:00.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:00.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:00.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:00.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:00.547 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:00.861 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:00.571+0000 7f4bb67a5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:17:00.861 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:00.673+0000 7f4bb67a5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:17:00.862 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:00.877+0000 7f4bb67a5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:17:01.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:01.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:01.369 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:01.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:01.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:17:01.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:17:01.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:01.408 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:01.424+0000 7f4bb67a5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:17:01.409 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:01.409 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:01.409 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.409 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.410 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.410 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.410 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.410 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:01.410 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:01.411 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.411 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:17:01.411 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:17:01.411 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:01.423 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:01.105+0000 7fdd0acf0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:17:01.423 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:01.270+0000 7fdd0acf0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:01.423 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:01.438+0000 7fdd0acf0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:17:01.423 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:01.423 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:01.423 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.423 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:17:01.424 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:01.744 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:01.515+0000 7f4bb67a5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:17:01.990 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:01.717+0000 7fdd0acf0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:17:01.990 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:01.806+0000 7fdd0acf0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:17:01.990 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:02.005+0000 7fdd0acf0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:17:02.029 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:01.759+0000 7f4bb67a5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:17:02.030 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:01.894+0000 7f4bb67a5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:17:02.030 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:01.990+0000 7f4bb67a5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:17:02.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:02 smithi139 ceph-mon[32760]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:17:02.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:02 smithi139 ceph-mon[32760]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T01:17:02.296 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:02.194+0000 7fdd0acf0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:17:02.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:02 smithi027 ceph-mon[29197]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:17:02.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:02 smithi027 ceph-mon[29197]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T01:17:02.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:02 smithi153 ceph-mon[32621]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:17:02.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:02 smithi153 ceph-mon[32621]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T01:17:02.530 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:02 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:02.106+0000 7f4bb67a5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:02.530 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:02 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:02.198+0000 7f4bb67a5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:17:03.046 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:02.787+0000 7fdd0acf0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:17:03.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:03 smithi153 ceph-mon[32621]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:03.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3883636921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88cb2314-6b03-44ba-b2ec-e38198da56b8"}]: dispatch 2024-04-25T01:17:03.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3883636921' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88cb2314-6b03-44ba-b2ec-e38198da56b8"}]': finished 2024-04-25T01:17:03.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:03 smithi153 ceph-mon[32621]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:17:03.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:03 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:03.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/241413816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:03.281 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:03.033+0000 7f4bb67a5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:17:03.281 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:17:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:17:03.251+0000 7f4bb67a5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:17:03.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:03 smithi027 ceph-mon[29197]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:03.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3883636921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88cb2314-6b03-44ba-b2ec-e38198da56b8"}]: dispatch 2024-04-25T01:17:03.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3883636921' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88cb2314-6b03-44ba-b2ec-e38198da56b8"}]': finished 2024-04-25T01:17:03.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:03 smithi027 ceph-mon[29197]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:17:03.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:03 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:03.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/241413816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:03.486 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:03.208+0000 7fdd0acf0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:17:03.486 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:03.302+0000 7fdd0acf0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:17:03.486 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:03.501+0000 7fdd0acf0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:17:03.487 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-mon[32760]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:03.487 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3883636921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88cb2314-6b03-44ba-b2ec-e38198da56b8"}]: dispatch 2024-04-25T01:17:03.487 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3883636921' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88cb2314-6b03-44ba-b2ec-e38198da56b8"}]': finished 2024-04-25T01:17:03.487 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-mon[32760]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:17:03.487 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:03.487 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/241413816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:04.152+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7fd0b400fa80 con 0x7fd0b8102290 2024-04-25T01:17:04.296 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:04.046+0000 7fdd0acf0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:17:04.296 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:04.137+0000 7fdd0acf0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:17:04.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-mon[32760]: Standby manager daemon z started 2024-04-25T01:17:04.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:17:04.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:17:04.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:17:04.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:17:04.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:04 smithi027 ceph-mon[29197]: Standby manager daemon z started 2024-04-25T01:17:04.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:04 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:17:04.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:04 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:17:04.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:04 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:17:04.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:04 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:17:04.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:04 smithi153 ceph-mon[32621]: Standby manager daemon z started 2024-04-25T01:17:04.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:04 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:17:04.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:04 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:17:04.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:04 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:17:04.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:04 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.153:0/1474183776' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:17:04.702 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:04.378+0000 7fdd0acf0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:17:04.703 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:04.509+0000 7fdd0acf0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:17:04.703 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:04.603+0000 7fdd0acf0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:17:04.703 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:04.717+0000 7fdd0acf0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:05.046 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:04.809+0000 7fdd0acf0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:17:05.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:05 smithi027 ceph-mon[29197]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:05.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:05 smithi027 ceph-mon[29197]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:17:05.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:17:05.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:05 smithi153 ceph-mon[32621]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:05.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:05 smithi153 ceph-mon[32621]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:17:05.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:17:05.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:05 smithi139 ceph-mon[32760]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:05.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:05 smithi139 ceph-mon[32760]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:17:05.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:17:06.046 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:05 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:05.612+0000 7fdd0acf0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:17:06.046 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:17:05 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:17:05.821+0000 7fdd0acf0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:17:06.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:06.165+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fd0b400e350 con 0x7fd0b8102290 2024-04-25T01:17:06.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:06 smithi027 ceph-mon[29197]: Standby manager daemon y started 2024-04-25T01:17:06.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:06 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:17:06.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:06 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:17:06.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:06 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:17:06.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:06 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:17:06.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:06 smithi153 ceph-mon[32621]: Standby manager daemon y started 2024-04-25T01:17:06.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:06 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:17:06.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:06 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:17:06.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:06 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:17:06.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:06 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:17:06.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:06 smithi139 ceph-mon[32760]: Standby manager daemon y started 2024-04-25T01:17:06.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:06 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:17:06.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:06 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:17:06.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:06 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:17:06.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:06 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.139:0/576503499' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:17:07.415 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:07 smithi027 ceph-mon[29197]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:07.415 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:07 smithi027 ceph-mon[29197]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T01:17:07.415 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:17:07.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:07 smithi153 ceph-mon[32621]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:07.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:07 smithi153 ceph-mon[32621]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T01:17:07.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:17:07.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:07 smithi139 ceph-mon[32760]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:07.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:07 smithi139 ceph-mon[32760]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T01:17:07.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:17:09.063 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:08 smithi027 ceph-mon[29197]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:09.063 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:09.064 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:09.064 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:09.064 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:09.064 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:09.064 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:09.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:08 smithi153 ceph-mon[32621]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:09.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:09.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:09.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:09.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:09.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:09.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:09.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:08 smithi139 ceph-mon[32760]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:09.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:09.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:09.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:09.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:09.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:09.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:11.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:10 smithi027 ceph-mon[29197]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:11.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:10 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:17:11.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:10 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:11.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:10 smithi153 ceph-mon[32621]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:11.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:10 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:17:11.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:10 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:11.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:10 smithi139 ceph-mon[32760]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:11.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:10 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:17:11.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:10 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:12.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:11 smithi153 ceph-mon[32621]: Deploying daemon osd.0 on smithi027 2024-04-25T01:17:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:11 smithi139 ceph-mon[32760]: Deploying daemon osd.0 on smithi027 2024-04-25T01:17:12.302 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:11 smithi027 ceph-mon[29197]: Deploying daemon osd.0 on smithi027 2024-04-25T01:17:13.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:12 smithi153 ceph-mon[32621]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:13.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:12 smithi139 ceph-mon[32760]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:13.425 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:12 smithi027 ceph-mon[29197]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:15.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:14 smithi153 ceph-mon[32621]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:15.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:14 smithi139 ceph-mon[32760]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:15.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:14 smithi027 ceph-mon[29197]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:17.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:16 smithi027 ceph-mon[29197]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:17.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:17.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:17.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:17.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:16 smithi153 ceph-mon[32621]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:17.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:17.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:17.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:16 smithi139 ceph-mon[32760]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:19.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:18 smithi027 ceph-mon[29197]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:19.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:18 smithi153 ceph-mon[32621]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:19.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:18 smithi139 ceph-mon[32760]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:19.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.997+0000 7fd0af7fe700 1 -- 172.21.15.27:0/350119678 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd0b8062670 con 0x7fd0a4039330 2024-04-25T01:17:19.981 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 0 on host 'smithi027' 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd0a4039330 msgr2=0x7fd0a403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd0a4039330 0x7fd0a403b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8000c00 tx=0x7fd0a8003960 comp rx=0 tx=0).stop 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8102290 msgr2=0x7fd0b8198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8102290 0x7fd0b8198490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400d9b0 tx=0x7fd0b400dcc0 comp rx=0 tx=0).stop 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 shutdown_connections 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd0a4039330 0x7fd0a403b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0b8103e30 0x7fd0b819fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8102290 0x7fd0b8198490 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 --2- 172.21.15.27:0/350119678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0b8103490 0x7fd0b81989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 >> 172.21.15.27:0/350119678 conn(0x7fd0b80fd830 msgr2=0x7fd0b8106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:19.999+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 shutdown_connections 2024-04-25T01:17:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:17:20.000+0000 7fd0c0337700 1 -- 172.21.15.27:0/350119678 wait complete. 2024-04-25T01:17:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:20.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:20.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:20.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:20.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:20.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:20.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:20.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:20.644 DEBUG:teuthology.orchestra.run.smithi027:osd.0> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T01:17:20.646 INFO:tasks.cephadm:Deploying osd.1 on smithi139 with /dev/vg_nvme/lv_4... 2024-04-25T01:17:20.646 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:17:20.892 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:17:21.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:21 smithi027 ceph-mon[29197]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:21.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:21 smithi027 ceph-mon[29197]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:17:21.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:21 smithi153 ceph-mon[32621]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:21.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:21 smithi153 ceph-mon[32621]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:17:21.281 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:21 smithi139 ceph-mon[32760]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:21.281 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:21 smithi139 ceph-mon[32760]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:17:22.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:22 smithi027 ceph-mon[29197]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:17:22.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:22 smithi027 ceph-mon[29197]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:17:22.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:22 smithi027 ceph-mon[29197]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:17:22.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:22 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:22.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:22 smithi139 ceph-mon[32760]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:17:22.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:22 smithi139 ceph-mon[32760]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:17:22.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:22 smithi139 ceph-mon[32760]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:17:22.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:22 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:22.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:22 smithi153 ceph-mon[32621]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:17:22.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:22 smithi153 ceph-mon[32621]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:17:22.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:22 smithi153 ceph-mon[32621]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:17:22.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:22 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:22.450 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:17:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0[43531]: 2024-04-25T01:17:22.044+0000 7f92d161f700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T01:17:22.450 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:17:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0[43531]: 2024-04-25T01:17:22.048+0000 7f92cd412700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:17:22.625 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:17:22.625 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:17:22.625 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-04-25T01:17:22.625 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-04-25T01:17:22.625 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122867 s, 853 MB/s 2024-04-25T01:17:22.626 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-04-25T01:17:23.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:23 smithi027 ceph-mon[29197]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:23.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:23 smithi027 ceph-mon[29197]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:17:23.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:23 smithi027 ceph-mon[29197]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:17:23.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:23.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:23 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:23.200 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_4 2024-04-25T01:17:23.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:23 smithi153 ceph-mon[32621]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:23.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:23 smithi153 ceph-mon[32621]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:17:23.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:23 smithi153 ceph-mon[32621]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:17:23.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:23 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:23.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:23 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:23.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:23 smithi139 ceph-mon[32760]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:23.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:23 smithi139 ceph-mon[32760]: from='osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:17:23.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:23 smithi139 ceph-mon[32760]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:17:23.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:23 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:23.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:23 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:23.443 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:17:24.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:17:24.425 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:17:24.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] boot 2024-04-25T01:17:24.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:17:24.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:17:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:24.427 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:17:24.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] boot 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:24.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:17:24.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: osd.0 [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] boot 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:24.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:24.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 -- 172.21.15.139:0/2780459932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6730102d40 msgr2=0x7f67301095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 --2- 172.21.15.139:0/2780459932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6730102d40 0x7f67301095f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f672000b0d0 tx=0x7f672000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 -- 172.21.15.139:0/2780459932 shutdown_connections 2024-04-25T01:17:24.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 --2- 172.21.15.139:0/2780459932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6730102d40 0x7f67301095f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 --2- 172.21.15.139:0/2780459932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67301011a0 0x7f67301015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 --2- 172.21.15.139:0/2780459932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67301023a0 0x7f6730102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 -- 172.21.15.139:0/2780459932 >> 172.21.15.139:0/2780459932 conn(0x7f67300fc760 msgr2=0x7f67300feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:24.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 -- 172.21.15.139:0/2780459932 shutdown_connections 2024-04-25T01:17:24.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 -- 172.21.15.139:0/2780459932 wait complete. 2024-04-25T01:17:24.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.934+0000 7f673582d700 1 Processor -- start 2024-04-25T01:17:24.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f673582d700 1 -- start start 2024-04-25T01:17:24.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f673582d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67301011a0 0x7f673006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f673582d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67301023a0 0x7f673019d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f673582d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6730102d40 0x7f673019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f673582d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f673010ce60 con 0x7f6730102d40 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f673582d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f673010cce0 con 0x7f67301011a0 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f673582d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f673010cfe0 con 0x7f67301023a0 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67301011a0 0x7f673006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67301011a0 0x7f673006b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:35778/0 (socket says 172.21.15.139:35778) 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 -- 172.21.15.139:0/982649089 learned_addr learned my addr 172.21.15.139:0/982649089 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 -- 172.21.15.139:0/982649089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67301023a0 msgr2=0x7f673019d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672e7fc700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67301023a0 0x7f673019d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67301023a0 0x7f673019d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 -- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6730102d40 msgr2=0x7f673019f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672f7fe700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6730102d40 0x7f673019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6730102d40 0x7f673019f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 -- 172.21.15.139:0/982649089 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67301a01e0 con 0x7f67301011a0 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f672effd700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67301011a0 0x7f673006b4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f672800c8f0 tx=0x7f672800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.935+0000 7f673482b700 1 -- 172.21.15.139:0/982649089 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728004220 con 0x7f67301011a0 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.936+0000 7f673482b700 1 -- 172.21.15.139:0/982649089 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728007de0 con 0x7f67301011a0 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.936+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67301a04d0 con 0x7f67301011a0 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.936+0000 7f673482b700 1 -- 172.21.15.139:0/982649089 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728003de0 con 0x7f67301011a0 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.936+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67301a0a40 con 0x7f67301011a0 2024-04-25T01:17:24.922 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.937+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6730068f20 con 0x7f67301011a0 2024-04-25T01:17:24.923 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.938+0000 7f673482b700 1 -- 172.21.15.139:0/982649089 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f6728007900 con 0x7f67301011a0 2024-04-25T01:17:24.923 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.938+0000 7f673482b700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67180a2b90 0x7f67180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.923 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.938+0000 7f673482b700 1 -- 172.21.15.139:0/982649089 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f67280cd250 con 0x7f67301011a0 2024-04-25T01:17:24.923 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.938+0000 7f672e7fc700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67180a2b90 0x7f67180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.923 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.939+0000 7f672e7fc700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67180a2b90 0x7f67180a5050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6724006fd0 tx=0x7f6724008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.926 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:24.943+0000 7f673482b700 1 -- 172.21.15.139:0/982649089 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6728014030 con 0x7f67301011a0 2024-04-25T01:17:25.133 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:25.149+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6730062670 con 0x7f67180a2b90 2024-04-25T01:17:25.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:25 smithi139 ceph-mon[32760]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:25.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:25 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi027 2024-04-25T01:17:25.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:25 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T01:17:25.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:25 smithi139 ceph-mon[32760]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:17:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:25 smithi027 ceph-mon[29197]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:25 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi027 2024-04-25T01:17:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:25 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T01:17:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:25 smithi027 ceph-mon[29197]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:17:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:25 smithi153 ceph-mon[32621]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:25.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:25 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi027 2024-04-25T01:17:25.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:25 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T01:17:25.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:25 smithi153 ceph-mon[32621]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:17:26.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:26 smithi139 ceph-mon[32760]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:26.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:26 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:26.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:26 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:26.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:26 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:26.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:26 smithi153 ceph-mon[32621]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:26.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:26 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:26.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:26 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:26.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:26 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:26.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:26 smithi027 ceph-mon[29197]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:26.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:26.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:26.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:27.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:27 smithi139 ceph-mon[32760]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:27.442 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:27 smithi153 ceph-mon[32621]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:27.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:27 smithi027 ceph-mon[29197]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:28.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:28 smithi139 ceph-mon[32760]: from='client.? 172.21.15.139:0/997625036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]: dispatch 2024-04-25T01:17:28.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:28 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]: dispatch 2024-04-25T01:17:28.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:28 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]': finished 2024-04-25T01:17:28.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:28 smithi139 ceph-mon[32760]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:17:28.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:28 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:28.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:28 smithi027 ceph-mon[29197]: from='client.? 172.21.15.139:0/997625036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]: dispatch 2024-04-25T01:17:28.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:28 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]: dispatch 2024-04-25T01:17:28.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:28 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]': finished 2024-04-25T01:17:28.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:28 smithi027 ceph-mon[29197]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:17:28.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:28 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:28.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:28 smithi153 ceph-mon[32621]: from='client.? 172.21.15.139:0/997625036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]: dispatch 2024-04-25T01:17:28.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:28 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]: dispatch 2024-04-25T01:17:28.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:28 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "817954a3-7495-44a0-9983-511339d330f4"}]': finished 2024-04-25T01:17:28.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:28 smithi153 ceph-mon[32621]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:17:28.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:28 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:29.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:29 smithi027 ceph-mon[29197]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:29.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:29 smithi027 ceph-mon[29197]: from='client.? 172.21.15.139:0/2135004554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:29.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:29 smithi153 ceph-mon[32621]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:29.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:29 smithi153 ceph-mon[32621]: from='client.? 172.21.15.139:0/2135004554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:29.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:29 smithi139 ceph-mon[32760]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:29.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:29 smithi139 ceph-mon[32760]: from='client.? 172.21.15.139:0/2135004554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:31.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:31 smithi139 ceph-mon[32760]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:31.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:31 smithi027 ceph-mon[29197]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:31.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:31 smithi153 ceph-mon[32621]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:33.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:33 smithi139 ceph-mon[32760]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:33.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:33 smithi027 ceph-mon[29197]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:33.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:33 smithi153 ceph-mon[32621]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:35.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:35 smithi139 ceph-mon[32760]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:35.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:17:35.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:35.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:35 smithi027 ceph-mon[29197]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:35.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:17:35.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:35.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:35 smithi153 ceph-mon[32621]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:35.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:17:35.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:36.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:36 smithi139 ceph-mon[32760]: Deploying daemon osd.1 on smithi139 2024-04-25T01:17:36.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:36 smithi027 ceph-mon[29197]: Deploying daemon osd.1 on smithi139 2024-04-25T01:17:36.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:36 smithi153 ceph-mon[32621]: Deploying daemon osd.1 on smithi139 2024-04-25T01:17:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:37 smithi139 ceph-mon[32760]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:37.350 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:37 smithi027 ceph-mon[29197]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:37.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:37 smithi153 ceph-mon[32621]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:39.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:39 smithi139 ceph-mon[32760]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:39 smithi027 ceph-mon[29197]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:39 smithi153 ceph-mon[32621]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:41.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:40 smithi027 ceph-mon[29197]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:41.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:41.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:41.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:41.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:40 smithi153 ceph-mon[32621]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:41.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:41.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:41.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:41.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:40 smithi139 ceph-mon[32760]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:41.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:41.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:41.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:43.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:42 smithi027 ceph-mon[29197]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:43.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:43.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:43.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:42 smithi153 ceph-mon[32621]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:43.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:43.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:43.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:42 smithi139 ceph-mon[32760]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:43.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:43.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:43.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:43.649 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 1 on host 'smithi139' 2024-04-25T01:17:43.649 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.665+0000 7f673482b700 1 -- 172.21.15.139:0/982649089 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6730062670 con 0x7f67180a2b90 2024-04-25T01:17:43.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67180a2b90 msgr2=0x7f67180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67180a2b90 0x7f67180a5050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6724006fd0 tx=0x7f6724008040 comp rx=0 tx=0).stop 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67301011a0 msgr2=0x7f673006b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67301011a0 0x7f673006b4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f672800c8f0 tx=0x7f672800ccb0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 shutdown_connections 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67180a2b90 0x7f67180a5050 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6730102d40 0x7f673019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67301011a0 0x7f673006b4e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 --2- 172.21.15.139:0/982649089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67301023a0 0x7f673019d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 >> 172.21.15.139:0/982649089 conn(0x7f67300fc760 msgr2=0x7f67301056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 shutdown_connections 2024-04-25T01:17:43.652 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:17:43.668+0000 7f673582d700 1 -- 172.21.15.139:0/982649089 wait complete. 2024-04-25T01:17:44.264 DEBUG:teuthology.orchestra.run.smithi139:osd.1> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T01:17:44.266 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_4... 2024-04-25T01:17:44.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:17:44.525 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:17:44.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:44 smithi139 ceph-mon[32760]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:44.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:44.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:44.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:44.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:44 smithi139 ceph-mon[32760]: from='osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:17:44.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:44 smithi139 ceph-mon[32760]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:17:44.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:44 smithi153 ceph-mon[32621]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:44.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:44.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:44.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:44.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:44 smithi153 ceph-mon[32621]: from='osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:17:44.928 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:44 smithi153 ceph-mon[32621]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:17:44.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:44 smithi027 ceph-mon[29197]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:44.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:44.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:44.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:44.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:44 smithi027 ceph-mon[29197]: from='osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:17:44.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:44 smithi027 ceph-mon[29197]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:17:45.667 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:45 smithi139 ceph-mon[32760]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:17:45.668 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:45 smithi139 ceph-mon[32760]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:17:45.668 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:45 smithi139 ceph-mon[32760]: from='osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:17:45.668 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:45.668 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:45 smithi139 ceph-mon[32760]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:17:45.668 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:17:45 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1[37370]: 2024-04-25T01:17:45.633+0000 7ffbbe326700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T01:17:45.668 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:17:45 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1[37370]: 2024-04-25T01:17:45.636+0000 7ffbba119700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:17:45.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:45 smithi027 ceph-mon[29197]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:17:45.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:45 smithi027 ceph-mon[29197]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:17:45.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:45 smithi027 ceph-mon[29197]: from='osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:17:45.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:45.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:45 smithi027 ceph-mon[29197]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:17:45.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:45 smithi153 ceph-mon[32621]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:17:45.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:45 smithi153 ceph-mon[32621]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:17:45.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:45 smithi153 ceph-mon[32621]: from='osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:17:45.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:45.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:45 smithi153 ceph-mon[32621]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:17:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:17:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:17:46.309 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-25T01:17:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-25T01:17:46.310 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115008 s, 912 MB/s 2024-04-25T01:17:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-25T01:17:46.776 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:46 smithi139 ceph-mon[32760]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:46.777 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:46 smithi139 ceph-mon[32760]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:17:46.777 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:46 smithi139 ceph-mon[32760]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:17:46.777 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:46.777 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:46.777 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:46.887 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:46 smithi153 ceph-mon[32621]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:46.887 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:46 smithi153 ceph-mon[32621]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:17:46.887 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:46 smithi153 ceph-mon[32621]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:17:46.887 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:46.887 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:46.887 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:46.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-04-25T01:17:46.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:46 smithi027 ceph-mon[29197]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:46.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:46 smithi027 ceph-mon[29197]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:17:46.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:46 smithi027 ceph-mon[29197]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:17:46.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:46.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:46.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:47.157 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:17:47.934 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:17:47.934 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:17:47.934 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] boot 2024-04-25T01:17:47.934 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi139 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi139 to 17088M 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:47.935 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] boot 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi139 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:47.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:17:47.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi139 to 17088M 2024-04-25T01:17:47.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:47.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:47.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:47.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:48.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:17:48.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:17:48.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: osd.1 [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] boot 2024-04-25T01:17:48.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:17:48.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:17:48.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi139 2024-04-25T01:17:48.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:48.047 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:48.047 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:17:48.047 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi139 to 17088M 2024-04-25T01:17:48.047 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:48.047 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:48.047 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:17:48.047 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 -- 172.21.15.153:0/579047296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e41011b0 msgr2=0x7ff9e41015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 --2- 172.21.15.153:0/579047296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e41011b0 0x7ff9e41015d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009990 tx=0x7ff9d4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 -- 172.21.15.153:0/579047296 shutdown_connections 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 --2- 172.21.15.153:0/579047296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e4102d50 0x7ff9e4109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 --2- 172.21.15.153:0/579047296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e41011b0 0x7ff9e41015d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 --2- 172.21.15.153:0/579047296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e41023b0 0x7ff9e4102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 -- 172.21.15.153:0/579047296 >> 172.21.15.153:0/579047296 conn(0x7ff9e40fc790 msgr2=0x7ff9e40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 -- 172.21.15.153:0/579047296 shutdown_connections 2024-04-25T01:17:48.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.682+0000 7ff9eb905700 1 -- 172.21.15.153:0/579047296 wait complete. 2024-04-25T01:17:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.683+0000 7ff9eb905700 1 Processor -- start 2024-04-25T01:17:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.683+0000 7ff9eb905700 1 -- start start 2024-04-25T01:17:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.683+0000 7ff9eb905700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e41011b0 0x7ff9e4197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.683+0000 7ff9eb905700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e41023b0 0x7ff9e41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.683+0000 7ff9eb905700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e4102d50 0x7ff9e419ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9eb905700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e410ce40 con 0x7ff9e41011b0 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9eb905700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e410ccc0 con 0x7ff9e41023b0 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9eb905700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e410cfc0 con 0x7ff9e4102d50 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e4102d50 0x7ff9e419ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e4102d50 0x7ff9e419ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39606/0 (socket says 172.21.15.153:39606) 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 -- 172.21.15.153:0/2323079672 learned_addr learned my addr 172.21.15.153:0/2323079672 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e96a1700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e41011b0 0x7ff9e4197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e8ea0700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e41023b0 0x7ff9e41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 -- 172.21.15.153:0/2323079672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e41023b0 msgr2=0x7ff9e41979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e41023b0 0x7ff9e41979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 -- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e41011b0 msgr2=0x7ff9e4197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e41011b0 0x7ff9e4197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 -- 172.21.15.153:0/2323079672 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e419f0f0 con 0x7ff9e4102d50 2024-04-25T01:17:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9e9ea2700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e4102d50 0x7ff9e419ea50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc007f10 tx=0x7ff9dc00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9da7fc700 1 -- 172.21.15.153:0/2323079672 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc004bb0 con 0x7ff9e4102d50 2024-04-25T01:17:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9da7fc700 1 -- 172.21.15.153:0/2323079672 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9dc004d50 con 0x7ff9e4102d50 2024-04-25T01:17:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e419f3e0 con 0x7ff9e4102d50 2024-04-25T01:17:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9da7fc700 1 -- 172.21.15.153:0/2323079672 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc005a60 con 0x7ff9e4102d50 2024-04-25T01:17:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.684+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e419f9a0 con 0x7ff9e4102d50 2024-04-25T01:17:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.685+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e4068f20 con 0x7ff9e4102d50 2024-04-25T01:17:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.686+0000 7ff9da7fc700 1 -- 172.21.15.153:0/2323079672 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7ff9dc01a020 con 0x7ff9e4102d50 2024-04-25T01:17:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.687+0000 7ff9da7fc700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff9d00a2b90 0x7ff9d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.687+0000 7ff9da7fc700 1 -- 172.21.15.153:0/2323079672 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7ff9dc0ccf80 con 0x7ff9e4102d50 2024-04-25T01:17:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.687+0000 7ff9e96a1700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff9d00a2b90 0x7ff9d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.687+0000 7ff9e96a1700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff9d00a2b90 0x7ff9d00a5050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4003890 tx=0x7ff9d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:48.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.692+0000 7ff9da7fc700 1 -- 172.21.15.153:0/2323079672 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff9dc010070 con 0x7ff9e4102d50 2024-04-25T01:17:48.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:48 smithi153 ceph-mon[32621]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:48.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:48 smithi153 ceph-mon[32621]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:17:48.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:17:48.896+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff9e4062670 con 0x7ff9d00a2b90 2024-04-25T01:17:48.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:48 smithi027 ceph-mon[29197]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:48.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:48 smithi027 ceph-mon[29197]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:17:49.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:48 smithi139 ceph-mon[32760]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:17:49.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:48 smithi139 ceph-mon[32760]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:17:49.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:49 smithi153 ceph-mon[32621]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:49.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:49.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:49.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:49 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:49.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:49 smithi027 ceph-mon[29197]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:49.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:49.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:49.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:49 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:50.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:49 smithi139 ceph-mon[32760]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:50.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:17:50.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:17:50.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:49 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:50.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:50 smithi027 ceph-mon[29197]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:51.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:50 smithi153 ceph-mon[32621]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:51.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:50 smithi139 ceph-mon[32760]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:51.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:51 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/819214337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]: dispatch 2024-04-25T01:17:51.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:51 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]: dispatch 2024-04-25T01:17:51.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:51 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]': finished 2024-04-25T01:17:51.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:51 smithi027 ceph-mon[29197]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:17:51.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:17:52.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:51 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/819214337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]: dispatch 2024-04-25T01:17:52.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:51 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]: dispatch 2024-04-25T01:17:52.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:51 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]': finished 2024-04-25T01:17:52.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:51 smithi153 ceph-mon[32621]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:17:52.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:17:52.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:51 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/819214337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]: dispatch 2024-04-25T01:17:52.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:51 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]: dispatch 2024-04-25T01:17:52.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:51 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbaef244-9159-4e52-912a-0c7d4857f6fd"}]': finished 2024-04-25T01:17:52.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:51 smithi139 ceph-mon[32760]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:17:52.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:17:52.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:52 smithi027 ceph-mon[29197]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:52.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:52 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/2818290962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:53.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:52 smithi153 ceph-mon[32621]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:53.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:52 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/2818290962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:53.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:52 smithi139 ceph-mon[32760]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:53.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:52 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/2818290962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:17:54.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:54 smithi027 ceph-mon[29197]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:55.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:54 smithi153 ceph-mon[32621]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:55.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:54 smithi139 ceph-mon[32760]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:56.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:56 smithi027 ceph-mon[29197]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:57.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:56 smithi153 ceph-mon[32621]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:57.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:56 smithi139 ceph-mon[32760]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:58.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:58 smithi027 ceph-mon[29197]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:58.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:17:58.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:58 smithi153 ceph-mon[32621]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:17:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:58 smithi139 ceph-mon[32760]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:17:59.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:17:59.046 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:59.910 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:17:59 smithi153 ceph-mon[32621]: Deploying daemon osd.2 on smithi153 2024-04-25T01:17:59.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:17:59 smithi027 ceph-mon[29197]: Deploying daemon osd.2 on smithi153 2024-04-25T01:18:00.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:17:59 smithi139 ceph-mon[32760]: Deploying daemon osd.2 on smithi153 2024-04-25T01:18:00.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:00 smithi153 ceph-mon[32621]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:00.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:00 smithi027 ceph-mon[29197]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:01.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:00 smithi139 ceph-mon[32760]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:02 smithi027 ceph-mon[29197]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:03.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:02 smithi153 ceph-mon[32621]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:03.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:02 smithi139 ceph-mon[32760]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:05.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:04 smithi027 ceph-mon[29197]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:05.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:04 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:05.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:04 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:05.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:04 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:05.249 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:04 smithi153 ceph-mon[32621]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:05.250 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:04 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:05.250 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:04 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:05.250 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:04 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:05.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:04 smithi139 ceph-mon[32760]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:05.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:04 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:05.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:04 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:05.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:04 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:07.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:07 smithi153 ceph-mon[32621]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:07.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:07.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:07.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:07.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:07.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:07.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.461+0000 7ff9da7fc700 1 -- 172.21.15.153:0/2323079672 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff9e4062670 con 0x7ff9d00a2b90 2024-04-25T01:18:07.445 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 2 on host 'smithi153' 2024-04-25T01:18:07.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff9d00a2b90 msgr2=0x7ff9d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff9d00a2b90 0x7ff9d00a5050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4003890 tx=0x7ff9d4005fd0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e4102d50 msgr2=0x7ff9e419ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e4102d50 0x7ff9e419ea50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc007f10 tx=0x7ff9dc00d3b0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 shutdown_connections 2024-04-25T01:18:07.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff9d00a2b90 0x7ff9d00a5050 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9e4102d50 0x7ff9e419ea50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e41011b0 0x7ff9e4197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 --2- 172.21.15.153:0/2323079672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9e41023b0 0x7ff9e41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 >> 172.21.15.153:0/2323079672 conn(0x7ff9e40fc790 msgr2=0x7ff9e41056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:07.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 shutdown_connections 2024-04-25T01:18:07.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:18:07.463+0000 7ff9eb905700 1 -- 172.21.15.153:0/2323079672 wait complete. 2024-04-25T01:18:07.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:07 smithi027 ceph-mon[29197]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:07.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:07.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:07.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:07.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:07.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:07.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:07 smithi139 ceph-mon[32760]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:07.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:07.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:07.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:07.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:07.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:08.071 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T01:18:08.073 INFO:tasks.cephadm:Deploying osd.3 on smithi027 with /dev/vg_nvme/lv_3... 2024-04-25T01:18:08.073 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:18:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:08.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:08 smithi027 ceph-mon[29197]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:08.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:08.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:08.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:08.641 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:08 smithi027 ceph-mon[29197]: from='osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:18:08.642 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:08 smithi027 ceph-mon[29197]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:18:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:08 smithi153 ceph-mon[32621]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:08 smithi153 ceph-mon[32621]: from='osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:18:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:08 smithi153 ceph-mon[32621]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:18:08.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:08 smithi139 ceph-mon[32760]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:08.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:08.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:08.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:08.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:08 smithi139 ceph-mon[32760]: from='osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:18:08.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:08 smithi139 ceph-mon[32760]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:18:09.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:09 smithi153 ceph-mon[32621]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:18:09.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:09 smithi153 ceph-mon[32621]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:18:09.468 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:09 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:09.469 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:09 smithi153 ceph-mon[32621]: from='osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:18:09.469 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:09 smithi153 ceph-mon[32621]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:18:09.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:09 smithi027 ceph-mon[29197]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:18:09.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:09 smithi027 ceph-mon[29197]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:18:09.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:09 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:09.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:09 smithi027 ceph-mon[29197]: from='osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:18:09.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:09 smithi027 ceph-mon[29197]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:18:09.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:09 smithi139 ceph-mon[32760]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:18:09.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:09 smithi139 ceph-mon[32760]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:18:09.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:09 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:09.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:09 smithi139 ceph-mon[32760]: from='osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:18:09.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:09 smithi139 ceph-mon[32760]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:18:10.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:09 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2[37762]: 2024-04-25T01:18:09.918+0000 7fa36fa1f700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T01:18:10.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:09 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2[37762]: 2024-04-25T01:18:09.921+0000 7fa36780a700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:18:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:18:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:18:10.397 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T01:18:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T01:18:10.397 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112116 s, 935 MB/s 2024-04-25T01:18:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T01:18:10.649 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:10 smithi153 ceph-mon[32621]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:10.649 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:10 smithi153 ceph-mon[32621]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:18:10.649 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:10 smithi153 ceph-mon[32621]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:18:10.649 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:10 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:10.649 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:10 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:10.649 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:10 smithi153 ceph-mon[32621]: from='osd.2 ' entity='osd.2' 2024-04-25T01:18:10.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:10 smithi027 ceph-mon[29197]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:10.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:10 smithi027 ceph-mon[29197]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:18:10.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:10 smithi027 ceph-mon[29197]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:18:10.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:10 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:10.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:10 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:10.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:10 smithi027 ceph-mon[29197]: from='osd.2 ' entity='osd.2' 2024-04-25T01:18:10.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:10 smithi139 ceph-mon[32760]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:10.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:10 smithi139 ceph-mon[32760]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:18:10.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:10 smithi139 ceph-mon[32760]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:18:10.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:10 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:10.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:10 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:10.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:10 smithi139 ceph-mon[32760]: from='osd.2 ' entity='osd.2' 2024-04-25T01:18:11.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_3 2024-04-25T01:18:11.416 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] boot 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi153 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:12.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-25T01:18:12.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:12.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:12.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:18:12.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:18:12.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:12.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] boot 2024-04-25T01:18:12.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi153 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:12.280 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:18:12.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: osd.2 [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] boot 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi153 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:12.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:13.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:12 smithi027 ceph-mon[29197]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:13.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:12 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:18:13.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:12 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:18:13.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:12 smithi027 ceph-mon[29197]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:18:13.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:12 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:18:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 -- 172.21.15.27:0/3012018466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101190 msgr2=0x7f9a801015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3012018466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101190 0x7f9a801015b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a700099c0 tx=0x7f9a70009cd0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 -- 172.21.15.27:0/3012018466 shutdown_connections 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3012018466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a80102d30 0x7f9a801095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3012018466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80101190 0x7f9a801015b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3012018466 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a80102390 0x7f9a801027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 -- 172.21.15.27:0/3012018466 >> 172.21.15.27:0/3012018466 conn(0x7f9a800fc730 msgr2=0x7f9a800feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 -- 172.21.15.27:0/3012018466 shutdown_connections 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.237+0000 7f9a8549e700 1 -- 172.21.15.27:0/3012018466 wait complete. 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.238+0000 7f9a8549e700 1 Processor -- start 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.238+0000 7f9a8549e700 1 -- start start 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.238+0000 7f9a8549e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a80101190 0x7f9a801051f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.238+0000 7f9a8549e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80102390 0x7f9a80103840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.238+0000 7f9a8549e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a80102d30 0x7f9a80103d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.238+0000 7f9a8549e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8010cd70 con 0x7f9a80102390 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.238+0000 7f9a8549e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8010cbf0 con 0x7f9a80101190 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.238+0000 7f9a8549e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8010cef0 con 0x7f9a80102d30 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a80101190 0x7f9a801051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a80101190 0x7f9a801051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:39820/0 (socket says 172.21.15.27:39820) 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 -- 172.21.15.27:0/3085715928 learned_addr learned my addr 172.21.15.27:0/3085715928 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7e7fc700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80102390 0x7f9a80103840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7f7fe700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a80102d30 0x7f9a80103d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 -- 172.21.15.27:0/3085715928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a80102d30 msgr2=0x7f9a80103d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a80102d30 0x7f9a80103d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 -- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80102390 msgr2=0x7f9a80103840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80102390 0x7f9a80103840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 -- 172.21.15.27:0/3085715928 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a80104630 con 0x7f9a80101190 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a7effd700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a80101190 0x7f9a801051f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000fa80 tx=0x7f9a7000fab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a67fff700 1 -- 172.21.15.27:0/3085715928 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70027070 con 0x7f9a80101190 2024-04-25T01:18:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a67fff700 1 -- 172.21.15.27:0/3085715928 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7000fd10 con 0x7f9a80101190 2024-04-25T01:18:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a801a25d0 con 0x7f9a80101190 2024-04-25T01:18:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a67fff700 1 -- 172.21.15.27:0/3085715928 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a700173f0 con 0x7f9a80101190 2024-04-25T01:18:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.239+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a801a2a80 con 0x7f9a80101190 2024-04-25T01:18:13.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.240+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a80068f20 con 0x7f9a80101190 2024-04-25T01:18:13.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.242+0000 7f9a67fff700 1 -- 172.21.15.27:0/3085715928 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f9a7002a070 con 0x7f9a80101190 2024-04-25T01:18:13.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.242+0000 7f9a67fff700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9a680a2b90 0x7f9a680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.242+0000 7f9a7e7fc700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9a680a2b90 0x7f9a680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.242+0000 7f9a67fff700 1 -- 172.21.15.27:0/3085715928 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f9a700ce9b0 con 0x7f9a80101190 2024-04-25T01:18:13.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.243+0000 7f9a7e7fc700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9a680a2b90 0x7f9a680a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800f440 tx=0x7f9a780079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.246+0000 7f9a67fff700 1 -- 172.21.15.27:0/3085715928 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a70031080 con 0x7f9a80101190 2024-04-25T01:18:13.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:12 smithi153 ceph-mon[32621]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:13.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:12 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:18:13.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:12 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:18:13.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:12 smithi153 ceph-mon[32621]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:18:13.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:12 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:18:13.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:12 smithi139 ceph-mon[32760]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:18:13.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:12 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:18:13.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:12 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:18:13.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:12 smithi139 ceph-mon[32760]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:18:13.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:12 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:18:13.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:13.464+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f9a8006a620 con 0x7f9a680a2b90 2024-04-25T01:18:13.698 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:18:13 smithi027 sudo[48304]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:18:13.699 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:18:13 smithi027 sudo[48304]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:13.699 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:18:13 smithi027 sudo[48304]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:13.699 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:18:13 smithi027 sudo[48304]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:14.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:14.075+0000 7f9a67fff700 1 -- 172.21.15.27:0/3085715928 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9a70098dc0 con 0x7f9a80101190 2024-04-25T01:18:14.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:14 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:18:14.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:14 smithi139 ceph-mon[32760]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:18:14.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:14 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:18:14.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:14 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:18:14.320 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:14 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:14.346 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:18:14 smithi027 sudo[48363]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:18:14.346 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:18:14 smithi027 sudo[48363]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:14.347 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:18:14 smithi027 sudo[48363]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:14.347 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:18:14 smithi027 sudo[48363]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:14.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:14 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:18:14.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:14 smithi027 ceph-mon[29197]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:18:14.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:14 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:18:14.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:14 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:18:14.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:14 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:14.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:14 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:18:14.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:14 smithi153 ceph-mon[32621]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:18:14.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:14 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:18:14.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:14 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:18:14.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:14 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:14.572 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:18:14 smithi139 sudo[41052]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:18:14.572 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:18:14 smithi139 sudo[41052]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:14.572 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:18:14 smithi139 sudo[41052]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:14.572 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:18:14 smithi139 sudo[41052]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:15.045 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:18:14 smithi139 sudo[41055]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:18:15.046 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:18:14 smithi139 sudo[41055]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:15.046 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:18:14 smithi139 sudo[41055]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:15.046 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:18:14 smithi139 sudo[41055]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:15.100 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:14 smithi153 sudo[41375]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:18:15.100 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:14 smithi153 sudo[41375]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:15.100 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:14 smithi153 sudo[41375]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:15.100 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:14 smithi153 sudo[41375]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:15.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:15 smithi153 ceph-mon[32621]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:15.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:15 smithi153 ceph-mon[32621]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:15.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:15 smithi153 ceph-mon[32621]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:18:15.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:15 smithi153 ceph-mon[32621]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:18:15.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:15 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:15.101 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:15 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:15.417 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 ceph-mon[29197]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:15.417 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 ceph-mon[29197]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:15.417 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 ceph-mon[29197]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:18:15.417 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 ceph-mon[29197]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:18:15.417 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:15.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:15.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:15 smithi153 sudo[41378]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:18:15.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:15 smithi153 sudo[41378]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:15.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:15 smithi153 sudo[41378]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:15.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:18:15 smithi153 sudo[41378]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:15.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:15 smithi139 ceph-mon[32760]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:15.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:15 smithi139 ceph-mon[32760]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:15.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:15 smithi139 ceph-mon[32760]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:18:15.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:15 smithi139 ceph-mon[32760]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:18:15.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:15 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:15.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:15 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:15.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 sudo[48413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:18:15.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 sudo[48413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:15.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 sudo[48413]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:15.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:15 smithi027 sudo[48413]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:16.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 sudo[48433]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:18:16.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 sudo[48433]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:16.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 sudo[48433]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:16.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 sudo[48433]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:16.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 sudo[41381]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:18:16.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 sudo[41381]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:16.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 sudo[41381]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:16.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 sudo[41381]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:16.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 sudo[41384]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:18:16.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 sudo[41384]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:16.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 sudo[41384]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:16.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 sudo[41384]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:16.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:16.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:16.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:16.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:16.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:16.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:16.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:16.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:16.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:16.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:17.028 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:16 smithi139 sudo[41058]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:18:17.028 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:16 smithi139 sudo[41058]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:17.028 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:16 smithi139 sudo[41058]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:17.029 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:16 smithi139 sudo[41058]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:17.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 sudo[41061]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:18:17.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 sudo[41061]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:18:17.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 sudo[41061]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 sudo[41061]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:17.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:17.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4064126095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "205e94f9-7782-426b-b5bd-1b56c93d55c9"}]: dispatch 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4064126095' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "205e94f9-7782-426b-b5bd-1b56c93d55c9"}]': finished 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:18:17.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:17.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:17.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:17.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:17 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/516766817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4064126095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "205e94f9-7782-426b-b5bd-1b56c93d55c9"}]: dispatch 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4064126095' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "205e94f9-7782-426b-b5bd-1b56c93d55c9"}]': finished 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:17.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:17.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:17 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/516766817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:18:18.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:18.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:18.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4064126095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "205e94f9-7782-426b-b5bd-1b56c93d55c9"}]: dispatch 2024-04-25T01:18:18.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:18.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:18.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:18:18.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:18:18.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4064126095' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "205e94f9-7782-426b-b5bd-1b56c93d55c9"}]': finished 2024-04-25T01:18:18.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:18:18.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:18.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:18.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:18:18.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:17 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/516766817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:18:18.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:18 smithi139 ceph-mon[32760]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:18.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:18 smithi027 ceph-mon[29197]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:19.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:18 smithi153 ceph-mon[32621]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:20.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:20 smithi139 ceph-mon[32760]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:20.871 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:20 smithi027 ceph-mon[29197]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:21.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:20 smithi153 ceph-mon[32621]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:22.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:22 smithi027 ceph-mon[29197]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:23.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:22 smithi153 ceph-mon[32621]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:23.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:22 smithi139 ceph-mon[32760]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:24.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:24 smithi027 ceph-mon[29197]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:24.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:18:24.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:24.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:24 smithi027 ceph-mon[29197]: Deploying daemon osd.3 on smithi027 2024-04-25T01:18:25.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:24 smithi153 ceph-mon[32621]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:25.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:18:25.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:25.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:24 smithi153 ceph-mon[32621]: Deploying daemon osd.3 on smithi027 2024-04-25T01:18:25.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:24 smithi139 ceph-mon[32760]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:25.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:18:25.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:25.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:24 smithi139 ceph-mon[32760]: Deploying daemon osd.3 on smithi027 2024-04-25T01:18:26.798 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:26 smithi027 ceph-mon[29197]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:26.831 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:26 smithi153 ceph-mon[32621]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:27.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:26 smithi139 ceph-mon[32760]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:28.847 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:28 smithi139 ceph-mon[32760]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:28.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:28 smithi027 ceph-mon[29197]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:29.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:28 smithi153 ceph-mon[32621]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:31.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:31 smithi153 ceph-mon[32621]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:31.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:31.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:31.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:31.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:31 smithi139 ceph-mon[32760]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:31.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:31 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:31.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:31 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:31.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:31 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:31.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:31 smithi027 ceph-mon[29197]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:31.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:31 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:31.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:31 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:31.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:31 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:33.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:33 smithi153 ceph-mon[32621]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:33.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:33 smithi139 ceph-mon[32760]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:33.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:33 smithi027 ceph-mon[29197]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.742+0000 7f9a67fff700 1 -- 172.21.15.27:0/3085715928 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9a8006a620 con 0x7f9a680a2b90 2024-04-25T01:18:33.725 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 3 on host 'smithi027' 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9a680a2b90 msgr2=0x7f9a680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9a680a2b90 0x7f9a680a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800f440 tx=0x7f9a780079e0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a80101190 msgr2=0x7f9a801051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a80101190 0x7f9a801051f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000fa80 tx=0x7f9a7000fab0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 shutdown_connections 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9a680a2b90 0x7f9a680a5050 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a80102d30 0x7f9a80103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a80101190 0x7f9a801051f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 --2- 172.21.15.27:0/3085715928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a80102390 0x7f9a80103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 >> 172.21.15.27:0/3085715928 conn(0x7f9a800fc730 msgr2=0x7f9a80069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 shutdown_connections 2024-04-25T01:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:18:33.744+0000 7f9a8549e700 1 -- 172.21.15.27:0/3085715928 wait complete. 2024-04-25T01:18:34.430 DEBUG:teuthology.orchestra.run.smithi027:osd.3> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T01:18:34.432 INFO:tasks.cephadm:Deploying osd.4 on smithi139 with /dev/vg_nvme/lv_3... 2024-04-25T01:18:34.432 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:18:34.678 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:18:34.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:34 smithi139 ceph-mon[32760]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:34.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:34.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:34.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:34 smithi139 ceph-mon[32760]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:18:34.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:34.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:34 smithi027 ceph-mon[29197]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:34.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:34.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:34.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:34 smithi027 ceph-mon[29197]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:18:34.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:34.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:34 smithi153 ceph-mon[32621]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:34 smithi153 ceph-mon[32621]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:35.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:35.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:35 smithi027 ceph-mon[29197]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:18:35.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:35 smithi027 ceph-mon[29197]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:18:35.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:35 smithi027 ceph-mon[29197]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:18:35.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:35.949 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:18:35 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3[50502]: 2024-04-25T01:18:35.730+0000 7fa2b3c1d700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T01:18:35.949 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:18:35 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3[50502]: 2024-04-25T01:18:35.737+0000 7fa2afa10700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:18:36.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:35 smithi153 ceph-mon[32621]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:18:36.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:35 smithi153 ceph-mon[32621]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:18:36.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:35 smithi153 ceph-mon[32621]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:18:36.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:36.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:35 smithi139 ceph-mon[32760]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:18:36.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:35 smithi139 ceph-mon[32760]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:18:36.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:35 smithi139 ceph-mon[32760]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:18:36.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:36.833 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:18:36.833 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:18:36.833 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-04-25T01:18:36.833 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-04-25T01:18:36.833 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113005 s, 928 MB/s 2024-04-25T01:18:36.833 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-04-25T01:18:36.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:36 smithi027 ceph-mon[29197]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:36.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:36 smithi027 ceph-mon[29197]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:18:36.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:36 smithi027 ceph-mon[29197]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:18:36.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:36.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:37.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:36 smithi153 ceph-mon[32621]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:37.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:36 smithi153 ceph-mon[32621]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:18:37.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:36 smithi153 ceph-mon[32621]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:18:37.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:37.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:37.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:36 smithi139 ceph-mon[32760]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:37.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:36 smithi139 ceph-mon[32760]: from='osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:18:37.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:36 smithi139 ceph-mon[32760]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:18:37.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:37.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:37.489 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_3 2024-04-25T01:18:37.733 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:18:37.989 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:37 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:18:37.989 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:37 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:18:37.989 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:37 smithi139 ceph-mon[32760]: osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] boot 2024-04-25T01:18:37.989 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:37 smithi139 ceph-mon[32760]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:18:37.989 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:37 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:37.990 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:37 smithi139 ceph-mon[32760]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:18:38.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:37 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:18:38.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:37 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:18:38.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:37 smithi153 ceph-mon[32621]: osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] boot 2024-04-25T01:18:38.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:37 smithi153 ceph-mon[32621]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:18:38.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:37 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:38.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:37 smithi153 ceph-mon[32621]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:18:38.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:37 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:18:38.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:37 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:18:38.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:37 smithi027 ceph-mon[29197]: osd.3 [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] boot 2024-04-25T01:18:38.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:37 smithi027 ceph-mon[29197]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:18:38.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:37 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:18:38.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:37 smithi027 ceph-mon[29197]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.315+0000 7f32c420d700 1 -- 172.21.15.139:0/3869548951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103ec0 msgr2=0x7f32bc10a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.315+0000 7f32c420d700 1 --2- 172.21.15.139:0/3869548951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103ec0 0x7f32bc10a770 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f32b400b0a0 tx=0x7f32b400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 -- 172.21.15.139:0/3869548951 shutdown_connections 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 --2- 172.21.15.139:0/3869548951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103ec0 0x7f32bc10a770 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 --2- 172.21.15.139:0/3869548951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32bc100540 0x7f32bc100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 --2- 172.21.15.139:0/3869548951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc103520 0x7f32bc103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 -- 172.21.15.139:0/3869548951 >> 172.21.15.139:0/3869548951 conn(0x7f32bc0fbec0 msgr2=0x7f32bc0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 -- 172.21.15.139:0/3869548951 shutdown_connections 2024-04-25T01:18:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 -- 172.21.15.139:0/3869548951 wait complete. 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 Processor -- start 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 -- start start 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.316+0000 7f32c420d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc100540 0x7f32bc1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c420d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103520 0x7f32bc1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c420d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32bc103ec0 0x7f32bc19f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c420d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc10de50 con 0x7f32bc103ec0 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c420d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc10dcd0 con 0x7f32bc103520 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c420d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc10dfd0 con 0x7f32bc100540 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103520 0x7f32bc1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103520 0x7f32bc1988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:34908/0 (socket says 172.21.15.139:34908) 2024-04-25T01:18:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 -- 172.21.15.139:0/390473811 learned_addr learned my addr 172.21.15.139:0/390473811 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c1fa9700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc100540 0x7f32bc1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c27aa700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32bc103ec0 0x7f32bc19f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 -- 172.21.15.139:0/390473811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc100540 msgr2=0x7f32bc1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc100540 0x7f32bc1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 -- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32bc103ec0 msgr2=0x7f32bc19f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32bc103ec0 0x7f32bc19f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 -- 172.21.15.139:0/390473811 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32bc1a0010 con 0x7f32bc103520 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c17a8700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103520 0x7f32bc1988f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c970 tx=0x7f32ac00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32b2ffd700 1 -- 172.21.15.139:0/390473811 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0078d0 con 0x7f32bc103520 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32b2ffd700 1 -- 172.21.15.139:0/390473811 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ac018480 con 0x7f32bc103520 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32bc1a0300 con 0x7f32bc103520 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32b2ffd700 1 -- 172.21.15.139:0/390473811 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00f6b0 con 0x7f32bc103520 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.317+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32bc1a07f0 con 0x7f32bc103520 2024-04-25T01:18:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.318+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32bc068f20 con 0x7f32bc103520 2024-04-25T01:18:39.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.319+0000 7f32b2ffd700 1 -- 172.21.15.139:0/390473811 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f32ac018630 con 0x7f32bc103520 2024-04-25T01:18:39.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.320+0000 7f32b2ffd700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32a80a2c20 0x7f32a80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.320+0000 7f32b2ffd700 1 -- 172.21.15.139:0/390473811 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f32ac0ce1e0 con 0x7f32bc103520 2024-04-25T01:18:39.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.320+0000 7f32c1fa9700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32a80a2c20 0x7f32a80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.321+0000 7f32c1fa9700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32a80a2c20 0x7f32a80a50e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f32bc069d10 tx=0x7f32b800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:39.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.324+0000 7f32b2ffd700 1 -- 172.21.15.139:0/390473811 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f32ac017e90 con 0x7f32bc103520 2024-04-25T01:18:39.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi027 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:39 smithi027 ceph-mon[29197]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:18:39.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:39.526+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f32bc062670 con 0x7f32a80a2c20 2024-04-25T01:18:39.510 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:39.510 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi027 2024-04-25T01:18:39.510 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.510 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:39.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.512 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:39 smithi139 ceph-mon[32760]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi027 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:39.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:39.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:39.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:39.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:39 smithi153 ceph-mon[32621]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:18:40.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:18:40.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:18:40.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:40.441 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:40 smithi139 ceph-mon[32760]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:18:40.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:18:40.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:18:40.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:40.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:40 smithi153 ceph-mon[32621]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:18:40.678 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:18:40.679 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:18:40.679 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:40.679 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:40 smithi027 ceph-mon[29197]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:18:41.351 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:41 smithi139 ceph-mon[32760]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:41.351 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:41 smithi139 ceph-mon[32760]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:41.351 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:41 smithi139 ceph-mon[32760]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:18:41.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:41 smithi153 ceph-mon[32621]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:41.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:41 smithi153 ceph-mon[32621]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:41.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:41 smithi153 ceph-mon[32621]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:18:41.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:41 smithi027 ceph-mon[29197]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:41.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:41 smithi027 ceph-mon[29197]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:41.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:41 smithi027 ceph-mon[29197]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:18:42.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:42 smithi153 ceph-mon[32621]: from='client.? 172.21.15.139:0/2738692445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]: dispatch 2024-04-25T01:18:42.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:42 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]: dispatch 2024-04-25T01:18:42.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:42 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]': finished 2024-04-25T01:18:42.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:42 smithi153 ceph-mon[32621]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:18:42.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:18:42.535 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:42 smithi139 ceph-mon[32760]: from='client.? 172.21.15.139:0/2738692445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]: dispatch 2024-04-25T01:18:42.536 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:42 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]: dispatch 2024-04-25T01:18:42.536 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:42 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]': finished 2024-04-25T01:18:42.536 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:42 smithi139 ceph-mon[32760]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:18:42.536 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:18:42.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:42 smithi027 ceph-mon[29197]: from='client.? 172.21.15.139:0/2738692445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]: dispatch 2024-04-25T01:18:42.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:42 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]: dispatch 2024-04-25T01:18:42.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:42 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57cc1b06-8233-435d-af98-2de145cfcc8d"}]': finished 2024-04-25T01:18:42.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:42 smithi027 ceph-mon[29197]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:18:42.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:18:43.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:43 smithi153 ceph-mon[32621]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:43.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:43 smithi153 ceph-mon[32621]: from='client.? 172.21.15.139:0/2428553455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:18:43.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:43 smithi139 ceph-mon[32760]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:43.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:43 smithi139 ceph-mon[32760]: from='client.? 172.21.15.139:0/2428553455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:18:43.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:43 smithi027 ceph-mon[29197]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:43.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:43 smithi027 ceph-mon[29197]: from='client.? 172.21.15.139:0/2428553455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:18:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:45 smithi153 ceph-mon[32621]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T01:18:45.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:45 smithi139 ceph-mon[32760]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T01:18:45.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:45 smithi027 ceph-mon[29197]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T01:18:47.514 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:47 smithi139 ceph-mon[32760]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:18:47.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:47 smithi153 ceph-mon[32621]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:18:47.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:47 smithi027 ceph-mon[29197]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:18:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:49 smithi153 ceph-mon[32621]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:18:49.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:49 smithi139 ceph-mon[32760]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:18:49.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:49 smithi027 ceph-mon[29197]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:18:50.418 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:18:50.418 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:50.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:18:50.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:50.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:18:50.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:51.457 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:51 smithi139 ceph-mon[32760]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T01:18:51.457 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:51 smithi139 ceph-mon[32760]: Deploying daemon osd.4 on smithi139 2024-04-25T01:18:51.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:51 smithi153 ceph-mon[32621]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T01:18:51.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:51 smithi153 ceph-mon[32621]: Deploying daemon osd.4 on smithi139 2024-04-25T01:18:51.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:51 smithi027 ceph-mon[29197]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T01:18:51.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:51 smithi027 ceph-mon[29197]: Deploying daemon osd.4 on smithi139 2024-04-25T01:18:52.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:52 smithi153 ceph-mon[32621]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:18:52.537 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:52 smithi139 ceph-mon[32760]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:18:52.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:52 smithi027 ceph-mon[29197]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:18:54.524 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:54 smithi139 ceph-mon[32760]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T01:18:54.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:54 smithi027 ceph-mon[29197]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T01:18:54.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:54 smithi153 ceph-mon[32621]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T01:18:55.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:55 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:55.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:55 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:55.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:55 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:56.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:55 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:56.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:55 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:56.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:55 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:56.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:55 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:56.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:55 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:56.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:55 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:57.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:56 smithi153 ceph-mon[32621]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:57.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:56 smithi139 ceph-mon[32760]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:57.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:56 smithi027 ceph-mon[29197]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:58.574 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 4 on host 'smithi139' 2024-04-25T01:18:58.574 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.591+0000 7f32b2ffd700 1 -- 172.21.15.139:0/390473811 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f32bc062670 con 0x7f32a80a2c20 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32a80a2c20 msgr2=0x7f32a80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32a80a2c20 0x7f32a80a50e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f32bc069d10 tx=0x7f32b800a3b0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103520 msgr2=0x7f32bc1988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103520 0x7f32bc1988f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c970 tx=0x7f32ac00cc80 comp rx=0 tx=0).stop 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 shutdown_connections 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32a80a2c20 0x7f32a80a50e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32bc103ec0 0x7f32bc19f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32bc100540 0x7f32bc1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 --2- 172.21.15.139:0/390473811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32bc103520 0x7f32bc1988f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 >> 172.21.15.139:0/390473811 conn(0x7f32bc0fbec0 msgr2=0x7f32bc106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 shutdown_connections 2024-04-25T01:18:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:18:58.593+0000 7f32c420d700 1 -- 172.21.15.139:0/390473811 wait complete. 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:18:59.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:18:58 smithi153 ceph-mon[32621]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:18:59.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:18:58 smithi139 ceph-mon[32760]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:18:59.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:18:59.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:18:58 smithi027 ceph-mon[29197]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:18:59.278 DEBUG:teuthology.orchestra.run.smithi139:osd.4> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T01:18:59.280 INFO:tasks.cephadm:Deploying osd.5 on smithi153 with /dev/vg_nvme/lv_3... 2024-04-25T01:18:59.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:18:59.526 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:19:00.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:00 smithi139 ceph-mon[32760]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:19:00.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:00 smithi139 ceph-mon[32760]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:19:00.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:00 smithi139 ceph-mon[32760]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:19:00.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:00.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:00 smithi139 ceph-mon[32760]: from='osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:19:00.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:00 smithi139 ceph-mon[32760]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:19:00.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:00 smithi027 ceph-mon[29197]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:19:00.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:00 smithi027 ceph-mon[29197]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:19:00.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:00 smithi027 ceph-mon[29197]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:19:00.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:00.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:00 smithi027 ceph-mon[29197]: from='osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:19:00.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:00 smithi027 ceph-mon[29197]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:19:00.954 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:00 smithi153 ceph-mon[32621]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:19:00.954 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:00 smithi153 ceph-mon[32621]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:19:00.954 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:00 smithi153 ceph-mon[32621]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:19:00.954 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:00.954 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:00 smithi153 ceph-mon[32621]: from='osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:19:00.954 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:00 smithi153 ceph-mon[32621]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:19:01.295 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:19:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4[43854]: 2024-04-25T01:19:01.048+0000 7f9da4617700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T01:19:01.295 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:19:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4[43854]: 2024-04-25T01:19:01.053+0000 7f9d9c402700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:19:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:19:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:19:01.719 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-25T01:19:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-25T01:19:01.719 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117246 s, 894 MB/s 2024-04-25T01:19:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-25T01:19:01.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:01 smithi153 ceph-mon[32621]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:19:01.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:01 smithi153 ceph-mon[32621]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:19:01.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:01.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:01.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:01 smithi153 ceph-mon[32621]: from='osd.4 ' entity='osd.4' 2024-04-25T01:19:01.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:01 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:01.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:01 smithi027 ceph-mon[29197]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:19:01.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:01 smithi027 ceph-mon[29197]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:19:01.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:01.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:01.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:01 smithi027 ceph-mon[29197]: from='osd.4 ' entity='osd.4' 2024-04-25T01:19:01.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:01 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:02.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:01 smithi139 ceph-mon[32760]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:19:02.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:01 smithi139 ceph-mon[32760]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:19:02.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:02.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:02.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:01 smithi139 ceph-mon[32760]: from='osd.4 ' entity='osd.4' 2024-04-25T01:19:02.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:01 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:02.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-04-25T01:19:02.668 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:19:02.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:19:02.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] boot 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:02.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:19:02.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:19:02.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] boot 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:02.796 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: osd.4 [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] boot 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:02.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:03.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:03 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi139 2024-04-25T01:19:03.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:03 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi139 to 8544M 2024-04-25T01:19:03.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:03 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi139 2024-04-25T01:19:03.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:03 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi139 to 8544M 2024-04-25T01:19:04.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:03 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi139 2024-04-25T01:19:04.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:03 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi139 to 8544M 2024-04-25T01:19:04.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.365+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1849698085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8102290 msgr2=0x7fd6e81026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.365+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1849698085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8102290 0x7fd6e81026b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009990 tx=0x7fd6e4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1849698085 shutdown_connections 2024-04-25T01:19:04.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1849698085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e8103e30 0x7fd6e810a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1849698085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8102290 0x7fd6e81026b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1849698085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6e8103490 0x7fd6e81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1849698085 >> 172.21.15.153:0/1849698085 conn(0x7fd6e80fd830 msgr2=0x7fd6e80ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:04.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1849698085 shutdown_connections 2024-04-25T01:19:04.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1849698085 wait complete. 2024-04-25T01:19:04.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 Processor -- start 2024-04-25T01:19:04.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.366+0000 7fd6efbbb700 1 -- start start 2024-04-25T01:19:04.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6efbbb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e8102290 0x7fd6e8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6efbbb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6e8103490 0x7fd6e8198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6efbbb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8103e30 0x7fd6e819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6efbbb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e810df50 con 0x7fd6e8102290 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6efbbb700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e810ddd0 con 0x7fd6e8103490 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6efbbb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e810e0d0 con 0x7fd6e8103e30 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8103e30 0x7fd6e819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8103e30 0x7fd6e819fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48292/0 (socket says 172.21.15.153:48292) 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 -- 172.21.15.153:0/1411093834 learned_addr learned my addr 172.21.15.153:0/1411093834 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ed957700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e8102290 0x7fd6e8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ed156700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6e8103490 0x7fd6e8198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 -- 172.21.15.153:0/1411093834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6e8103490 msgr2=0x7fd6e8198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6e8103490 0x7fd6e8198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 -- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e8102290 msgr2=0x7fd6e8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e8102290 0x7fd6e8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 -- 172.21.15.153:0/1411093834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e81a0290 con 0x7fd6e8103e30 2024-04-25T01:19:04.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6ee158700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8103e30 0x7fd6e819fbf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000b9a0 tx=0x7fd6e000bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6deffd700 1 -- 172.21.15.153:0/1411093834 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0019070 con 0x7fd6e8103e30 2024-04-25T01:19:04.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6deffd700 1 -- 172.21.15.153:0/1411093834 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6e000ce10 con 0x7fd6e8103e30 2024-04-25T01:19:04.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e81a0580 con 0x7fd6e8103e30 2024-04-25T01:19:04.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6deffd700 1 -- 172.21.15.153:0/1411093834 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00073d0 con 0x7fd6e8103e30 2024-04-25T01:19:04.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.367+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e81a0af0 con 0x7fd6e8103e30 2024-04-25T01:19:04.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.368+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e8068f20 con 0x7fd6e8103e30 2024-04-25T01:19:04.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.369+0000 7fd6deffd700 1 -- 172.21.15.153:0/1411093834 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6e001c070 con 0x7fd6e8103e30 2024-04-25T01:19:04.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.370+0000 7fd6deffd700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd6d40a2c20 0x7fd6d40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.370+0000 7fd6deffd700 1 -- 172.21.15.153:0/1411093834 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fd6e00cd380 con 0x7fd6e8103e30 2024-04-25T01:19:04.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.370+0000 7fd6ed957700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd6d40a2c20 0x7fd6d40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.371+0000 7fd6ed957700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd6d40a2c20 0x7fd6d40a50e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4003890 tx=0x7fd6e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.375+0000 7fd6deffd700 1 -- 172.21.15.153:0/1411093834 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6e009b350 con 0x7fd6e8103e30 2024-04-25T01:19:04.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:04.576+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd6e8062670 con 0x7fd6d40a2c20 2024-04-25T01:19:04.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:04 smithi027 ceph-mon[29197]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:04.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:04 smithi027 ceph-mon[29197]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:19:04.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:04 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:04.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:04 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:04.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:04 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:05.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:04 smithi153 ceph-mon[32621]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:05.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:04 smithi153 ceph-mon[32621]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:19:05.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:04 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:05.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:04 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:05.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:04 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:05.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:04 smithi139 ceph-mon[32760]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:05.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:04 smithi139 ceph-mon[32760]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:19:05.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:04 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:05.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:04 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:05.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:04 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:05.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:05 smithi153 ceph-mon[32621]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:05.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:05 smithi027 ceph-mon[29197]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:06.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:05 smithi139 ceph-mon[32760]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:06.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:06 smithi153 ceph-mon[32621]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:06.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:06 smithi027 ceph-mon[29197]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:07.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:06 smithi139 ceph-mon[32760]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:07.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:07 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/3241780612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]: dispatch 2024-04-25T01:19:07.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:07 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]: dispatch 2024-04-25T01:19:07.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:07 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]': finished 2024-04-25T01:19:07.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:07 smithi027 ceph-mon[29197]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:19:07.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:08.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:07 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/3241780612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]: dispatch 2024-04-25T01:19:08.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:07 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]: dispatch 2024-04-25T01:19:08.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:07 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]': finished 2024-04-25T01:19:08.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:07 smithi153 ceph-mon[32621]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:19:08.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:08.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:07 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/3241780612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]: dispatch 2024-04-25T01:19:08.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:07 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]: dispatch 2024-04-25T01:19:08.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:07 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "087587ee-f7ae-4546-a39a-2a6a774c54ef"}]': finished 2024-04-25T01:19:08.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:07 smithi139 ceph-mon[32760]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:19:08.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:08.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:08 smithi027 ceph-mon[29197]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:08.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:08 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/77215465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:09.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:08 smithi153 ceph-mon[32621]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:09.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:08 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/77215465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:09.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:08 smithi139 ceph-mon[32760]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:09.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:08 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/77215465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:10.947 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:10 smithi153 ceph-mon[32621]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:10.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:10 smithi027 ceph-mon[29197]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:11.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:10 smithi139 ceph-mon[32760]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:12.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:12 smithi027 ceph-mon[29197]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:13.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:12 smithi153 ceph-mon[32621]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:13.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:12 smithi139 ceph-mon[32760]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:14.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:14 smithi027 ceph-mon[29197]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:14.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:14 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:19:14.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:14 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:15.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:14 smithi153 ceph-mon[32621]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:15.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:14 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:19:15.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:14 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:15.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:14 smithi139 ceph-mon[32760]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:15.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:14 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:19:15.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:14 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:15.722 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:15 smithi153 ceph-mon[32621]: Deploying daemon osd.5 on smithi153 2024-04-25T01:19:15.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:15 smithi027 ceph-mon[29197]: Deploying daemon osd.5 on smithi153 2024-04-25T01:19:16.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:15 smithi139 ceph-mon[32760]: Deploying daemon osd.5 on smithi153 2024-04-25T01:19:16.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:16 smithi027 ceph-mon[29197]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:17.020 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:16 smithi153 ceph-mon[32621]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:17.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:16 smithi139 ceph-mon[32760]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:18.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:18 smithi027 ceph-mon[29197]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:19.017 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:18 smithi153 ceph-mon[32621]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:19.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:18 smithi139 ceph-mon[32760]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:21.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:20 smithi027 ceph-mon[29197]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:21.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:21.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:21.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:20 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:21.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:20 smithi153 ceph-mon[32621]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:21.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:21.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:21.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:20 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:21.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:20 smithi139 ceph-mon[32760]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:21.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:21.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:21.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:20 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:22 smithi153 ceph-mon[32621]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:22 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:22 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:22 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:22 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:23.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:22 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:22 smithi027 ceph-mon[29197]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:23.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:22 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:22 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:22 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:23.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:22 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:23.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:22 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:22 smithi139 ceph-mon[32760]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:23.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:22 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.296 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:22 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:22 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:23.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:22 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:23.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:22 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:23.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.595+0000 7fd6deffd700 1 -- 172.21.15.153:0/1411093834 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd6e8062670 con 0x7fd6d40a2c20 2024-04-25T01:19:23.578 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 5 on host 'smithi153' 2024-04-25T01:19:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd6d40a2c20 msgr2=0x7fd6d40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd6d40a2c20 0x7fd6d40a50e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4003890 tx=0x7fd6e4005fd0 comp rx=0 tx=0).stop 2024-04-25T01:19:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8103e30 msgr2=0x7fd6e819fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8103e30 0x7fd6e819fbf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000b9a0 tx=0x7fd6e000bcb0 comp rx=0 tx=0).stop 2024-04-25T01:19:23.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 shutdown_connections 2024-04-25T01:19:23.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd6d40a2c20 0x7fd6d40a50e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:23.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6e8103e30 0x7fd6e819fbf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:23.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e8102290 0x7fd6e8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:23.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 --2- 172.21.15.153:0/1411093834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6e8103490 0x7fd6e8198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:23.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.597+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 >> 172.21.15.153:0/1411093834 conn(0x7fd6e80fd830 msgr2=0x7fd6e8106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:23.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.598+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 shutdown_connections 2024-04-25T01:19:23.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:19:23.598+0000 7fd6efbbb700 1 -- 172.21.15.153:0/1411093834 wait complete. 2024-04-25T01:19:24.239 DEBUG:teuthology.orchestra.run.smithi153:osd.5> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T01:19:24.241 INFO:tasks.cephadm:Deploying osd.6 on smithi027 with /dev/vg_nvme/lv_2... 2024-04-25T01:19:24.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:19:24.501 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:19:24.789 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:24 smithi027 ceph-mon[29197]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:24.789 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:24.789 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:24.789 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:24.789 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:24 smithi027 ceph-mon[29197]: from='osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:19:24.789 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:24 smithi027 ceph-mon[29197]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:19:24.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:24 smithi139 ceph-mon[32760]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:24.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:24.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:24.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:24.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:24 smithi139 ceph-mon[32760]: from='osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:19:24.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:24 smithi139 ceph-mon[32760]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:19:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:24 smithi153 ceph-mon[32621]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:25.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:25.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:24 smithi153 ceph-mon[32621]: from='osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:19:25.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:24 smithi153 ceph-mon[32621]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:19:25.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:25 smithi153 ceph-mon[32621]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:19:25.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:25 smithi153 ceph-mon[32621]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:19:25.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:25 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:25.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:25 smithi153 ceph-mon[32621]: from='osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:19:25.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:25 smithi153 ceph-mon[32621]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:19:25.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:25 smithi027 ceph-mon[29197]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:19:25.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:25 smithi027 ceph-mon[29197]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:19:25.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:25 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:25.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:25 smithi027 ceph-mon[29197]: from='osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:19:25.827 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:25 smithi027 ceph-mon[29197]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:19:26.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:25 smithi139 ceph-mon[32760]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:19:26.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:25 smithi139 ceph-mon[32760]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:19:26.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:25 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:26.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:25 smithi139 ceph-mon[32760]: from='osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:19:26.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:25 smithi139 ceph-mon[32760]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:19:26.317 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:19:26 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:19:26.104+0000 7fdaaf1c3700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T01:19:26.317 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:19:26 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:19:26.114+0000 7fdaa6fae700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:19:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:19:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:19:26.586 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T01:19:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T01:19:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114539 s, 915 MB/s 2024-04-25T01:19:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T01:19:26.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:26 smithi153 ceph-mon[32621]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:26.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:26 smithi153 ceph-mon[32621]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:19:26.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:26 smithi153 ceph-mon[32621]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:19:26.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:26 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:26.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:26 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:26.622 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:26 smithi153 ceph-mon[32621]: from='osd.5 ' entity='osd.5' 2024-04-25T01:19:26.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:26 smithi027 ceph-mon[29197]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:26.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:26 smithi027 ceph-mon[29197]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:19:26.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:26 smithi027 ceph-mon[29197]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:19:26.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:26.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:26 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:26.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:26 smithi027 ceph-mon[29197]: from='osd.5 ' entity='osd.5' 2024-04-25T01:19:27.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:26 smithi139 ceph-mon[32760]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:19:27.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:26 smithi139 ceph-mon[32760]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:19:27.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:26 smithi139 ceph-mon[32760]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:19:27.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:26 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:27.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:26 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:27.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:26 smithi139 ceph-mon[32760]: from='osd.5 ' entity='osd.5' 2024-04-25T01:19:27.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_2 2024-04-25T01:19:27.560 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] boot 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:27.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:27.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:27.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:27.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:27.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:27.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:27 smithi027 ceph-mon[29197]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:19:28.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:19:28.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] boot 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:28.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:27 smithi153 ceph-mon[32621]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:19:28.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: osd.5 [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] boot 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:28.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:27 smithi139 ceph-mon[32760]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:19:28.873 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:28 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi153 2024-04-25T01:19:28.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:28 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-25T01:19:28.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:28 smithi027 ceph-mon[29197]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:28.960 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:28 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi153 2024-04-25T01:19:28.960 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:28 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-25T01:19:28.960 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:28 smithi139 ceph-mon[32760]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:29.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:28 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi153 2024-04-25T01:19:29.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:28 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-25T01:19:29.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:28 smithi153 ceph-mon[32621]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:29.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1826850191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc4100b50 msgr2=0x7f1dc4100f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:29.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1826850191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc4100b50 0x7f1dc4100f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009990 tx=0x7f1db4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:19:29.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1826850191 shutdown_connections 2024-04-25T01:19:29.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1826850191 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc4101e70 0x7f1dc41022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:29.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1826850191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc4100b50 0x7f1dc4100f70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:29.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1826850191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc41014b0 0x7f1dc4101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:29.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1826850191 >> 172.21.15.27:0/1826850191 conn(0x7f1dc40fc790 msgr2=0x7f1dc40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:29.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1826850191 shutdown_connections 2024-04-25T01:19:29.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1826850191 wait complete. 2024-04-25T01:19:29.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 Processor -- start 2024-04-25T01:19:29.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.482+0000 7f1dca3f1700 1 -- start start 2024-04-25T01:19:29.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dca3f1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc4100b50 0x7f1dc419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:29.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dca3f1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc41014b0 0x7f1dc419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:29.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dca3f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc4101e70 0x7f1dc41a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:29.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dca3f1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc4110150 con 0x7f1dc41014b0 2024-04-25T01:19:29.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dca3f1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc410ffd0 con 0x7f1dc4100b50 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dca3f1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc41102d0 con 0x7f1dc4101e70 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc41014b0 0x7f1dc419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc41014b0 0x7f1dc419ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48030/0 (socket says 172.21.15.27:48030) 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc3fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc4100b50 0x7f1dc419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 -- 172.21.15.27:0/1800736246 learned_addr learned my addr 172.21.15.27:0/1800736246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc898e700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc4101e70 0x7f1dc41a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 -- 172.21.15.27:0/1800736246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc4101e70 msgr2=0x7f1dc41a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc4101e70 0x7f1dc41a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 -- 172.21.15.27:0/1800736246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc4100b50 msgr2=0x7f1dc419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc4100b50 0x7f1dc419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 -- 172.21.15.27:0/1800736246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc41a2390 con 0x7f1dc41014b0 2024-04-25T01:19:29.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc3fff700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc4100b50 0x7f1dc419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:19:29.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc37fe700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc41014b0 0x7f1dc419ac70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1db800b560 tx=0x7f1db800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:29.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.483+0000 7f1dc17fa700 1 -- 172.21.15.27:0/1800736246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8015020 con 0x7f1dc41014b0 2024-04-25T01:19:29.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.484+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc41a2680 con 0x7f1dc41014b0 2024-04-25T01:19:29.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.484+0000 7f1dc17fa700 1 -- 172.21.15.27:0/1800736246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1db8011d10 con 0x7f1dc41014b0 2024-04-25T01:19:29.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.484+0000 7f1dc17fa700 1 -- 172.21.15.27:0/1800736246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8010e50 con 0x7f1dc41014b0 2024-04-25T01:19:29.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.484+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc41a2c40 con 0x7f1dc41014b0 2024-04-25T01:19:29.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.485+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc4068f20 con 0x7f1dc41014b0 2024-04-25T01:19:29.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.486+0000 7f1dc17fa700 1 -- 172.21.15.27:0/1800736246 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1db8018020 con 0x7f1dc41014b0 2024-04-25T01:19:29.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.486+0000 7f1dc17fa700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1dac0a2dc0 0x7f1dac0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:29.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.487+0000 7f1dc3fff700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1dac0a2dc0 0x7f1dac0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:29.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.487+0000 7f1dc17fa700 1 -- 172.21.15.27:0/1800736246 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f1db80cde30 con 0x7f1dc41014b0 2024-04-25T01:19:29.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.487+0000 7f1dc3fff700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1dac0a2dc0 0x7f1dac0a5280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1db400b5c0 tx=0x7f1db4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.490+0000 7f1dc17fa700 1 -- 172.21.15.27:0/1800736246 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1db801f020 con 0x7f1dc41014b0 2024-04-25T01:19:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:29.712+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1dc4062670 con 0x7f1dac0a2dc0 2024-04-25T01:19:29.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:29 smithi027 ceph-mon[29197]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:19:30.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:29 smithi153 ceph-mon[32621]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:19:30.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:29 smithi139 ceph-mon[32760]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:19:30.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:30 smithi027 ceph-mon[29197]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:30.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:30 smithi027 ceph-mon[29197]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:19:30.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:30 smithi027 ceph-mon[29197]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:30.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:30.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:30.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:31.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:30 smithi153 ceph-mon[32621]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:31.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:30 smithi153 ceph-mon[32621]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:19:31.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:30 smithi153 ceph-mon[32621]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:31.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:31.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:31.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:31.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:30 smithi139 ceph-mon[32760]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:31.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:30 smithi139 ceph-mon[32760]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:19:31.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:30 smithi139 ceph-mon[32760]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:31.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:31.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:31.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:31.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:31 smithi027 ceph-mon[29197]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:19:32.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:31 smithi153 ceph-mon[32621]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:19:32.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:31 smithi139 ceph-mon[32760]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:19:32.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:32 smithi027 ceph-mon[29197]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:32.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:32 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/534160404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6834f4e7-9977-44db-8075-4e7596c95a0d"}]: dispatch 2024-04-25T01:19:32.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:32 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/534160404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6834f4e7-9977-44db-8075-4e7596c95a0d"}]': finished 2024-04-25T01:19:32.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:32 smithi027 ceph-mon[29197]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:19:32.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:32 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:33.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:32 smithi153 ceph-mon[32621]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:33.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:32 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/534160404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6834f4e7-9977-44db-8075-4e7596c95a0d"}]: dispatch 2024-04-25T01:19:33.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:32 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/534160404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6834f4e7-9977-44db-8075-4e7596c95a0d"}]': finished 2024-04-25T01:19:33.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:32 smithi153 ceph-mon[32621]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:19:33.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:32 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:33.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:32 smithi139 ceph-mon[32760]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:33.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:32 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/534160404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6834f4e7-9977-44db-8075-4e7596c95a0d"}]: dispatch 2024-04-25T01:19:33.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:32 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/534160404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6834f4e7-9977-44db-8075-4e7596c95a0d"}]': finished 2024-04-25T01:19:33.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:32 smithi139 ceph-mon[32760]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:19:33.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:32 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:33.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:33 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3560993371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:34.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:33 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3560993371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:34.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:33 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3560993371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:34 smithi027 ceph-mon[29197]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-04-25T01:19:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:34 smithi153 ceph-mon[32621]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-04-25T01:19:35.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:34 smithi139 ceph-mon[32760]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-04-25T01:19:36.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:36 smithi027 ceph-mon[29197]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:19:37.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:36 smithi153 ceph-mon[32621]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:19:37.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:36 smithi139 ceph-mon[32760]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:19:38.892 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:38 smithi027 ceph-mon[29197]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-25T01:19:39.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:38 smithi153 ceph-mon[32621]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-25T01:19:39.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:38 smithi139 ceph-mon[32760]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-25T01:19:40.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:40 smithi027 ceph-mon[29197]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T01:19:40.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:19:40.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:41.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:40 smithi153 ceph-mon[32621]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T01:19:41.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:19:41.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:41.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:40 smithi139 ceph-mon[32760]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T01:19:41.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:19:41.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:41.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:41 smithi027 ceph-mon[29197]: Deploying daemon osd.6 on smithi027 2024-04-25T01:19:42.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:41 smithi153 ceph-mon[32621]: Deploying daemon osd.6 on smithi027 2024-04-25T01:19:42.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:41 smithi139 ceph-mon[32760]: Deploying daemon osd.6 on smithi027 2024-04-25T01:19:42.810 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:42 smithi027 ceph-mon[29197]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:19:43.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:42 smithi153 ceph-mon[32621]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:19:43.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:42 smithi139 ceph-mon[32760]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:19:44.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:44 smithi027 ceph-mon[29197]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T01:19:45.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:44 smithi153 ceph-mon[32621]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T01:19:45.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:44 smithi139 ceph-mon[32760]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T01:19:46.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:46 smithi027 ceph-mon[29197]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:46.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:46.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:46.921 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:46 smithi153 ceph-mon[32621]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:47.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:47.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:46 smithi139 ceph-mon[32760]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:47.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:47.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:47.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:48.911 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:48 smithi027 ceph-mon[29197]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:49.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:48 smithi153 ceph-mon[32621]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:49.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:48 smithi139 ceph-mon[32760]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:50.471 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 6 on host 'smithi027' 2024-04-25T01:19:50.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.489+0000 7f1dc17fa700 1 -- 172.21.15.27:0/1800736246 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1dc4062670 con 0x7f1dac0a2dc0 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.491+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1dac0a2dc0 msgr2=0x7f1dac0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.491+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1dac0a2dc0 0x7f1dac0a5280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1db400b5c0 tx=0x7f1db4005fb0 comp rx=0 tx=0).stop 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc41014b0 msgr2=0x7f1dc419ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc41014b0 0x7f1dc419ac70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1db800b560 tx=0x7f1db800b870 comp rx=0 tx=0).stop 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 shutdown_connections 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1dac0a2dc0 0x7f1dac0a5280 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc4101e70 0x7f1dc41a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc4100b50 0x7f1dc419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 --2- 172.21.15.27:0/1800736246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc41014b0 0x7f1dc419ac70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 >> 172.21.15.27:0/1800736246 conn(0x7f1dc40fc790 msgr2=0x7f1dc40fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 shutdown_connections 2024-04-25T01:19:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:19:50.492+0000 7f1dca3f1700 1 -- 172.21.15.27:0/1800736246 wait complete. 2024-04-25T01:19:50.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:50 smithi027 ceph-mon[29197]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:50.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:50 smithi027 ceph-mon[29197]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:19:50.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:50.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:50 smithi153 ceph-mon[32621]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:51.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:50 smithi153 ceph-mon[32621]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:19:51.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:50 smithi139 ceph-mon[32760]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:51.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:50 smithi139 ceph-mon[32760]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:19:51.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.321 DEBUG:teuthology.orchestra.run.smithi027:osd.6> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T01:19:51.324 INFO:tasks.cephadm:Deploying osd.7 on smithi139 with /dev/vg_nvme/lv_2... 2024-04-25T01:19:51.324 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:19:51.568 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:19:51.925 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:51 smithi139 ceph-mon[32760]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:19:51.925 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:51 smithi139 ceph-mon[32760]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:19:51.925 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:51 smithi139 ceph-mon[32760]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:19:51.925 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:51.925 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.925 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.926 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:51 smithi139 ceph-mon[32760]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:19:51.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-mon[29197]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:19:51.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-mon[29197]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:19:51.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-mon[29197]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:19:51.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:51.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:51.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-mon[29197]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:19:51.948 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:19:51.656+0000 7fbb0fe7d700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T01:19:51.949 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:19:51 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:19:51.666+0000 7fbb0bc70700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:19:52.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:51 smithi153 ceph-mon[32621]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:19:52.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:51 smithi153 ceph-mon[32621]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:19:52.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:51 smithi153 ceph-mon[32621]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:19:52.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:52.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:52.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:52.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:51 smithi153 ceph-mon[32621]: from='osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:19:52.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:52 smithi027 ceph-mon[29197]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:52.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:52 smithi027 ceph-mon[29197]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:19:52.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:52 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:52.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:52 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:52.975 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:52 smithi139 ceph-mon[32760]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:52.975 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:52 smithi139 ceph-mon[32760]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:19:52.975 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:52 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:52.975 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:52 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:53.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:52 smithi153 ceph-mon[32621]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:53.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:52 smithi153 ceph-mon[32621]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:19:53.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:52 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:53.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:52 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:53.710 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:19:53.710 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:19:53.710 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-04-25T01:19:53.710 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-04-25T01:19:53.710 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112846 s, 929 MB/s 2024-04-25T01:19:53.710 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-04-25T01:19:53.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:53 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:19:53.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:53 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:19:53.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:53.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:53 smithi027 ceph-mon[29197]: osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] boot 2024-04-25T01:19:53.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:53 smithi027 ceph-mon[29197]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:19:53.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:54.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:53 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:19:54.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:53 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:19:54.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:54.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:53 smithi153 ceph-mon[32621]: osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] boot 2024-04-25T01:19:54.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:53 smithi153 ceph-mon[32621]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:19:54.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:54.036 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:53 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:19:54.036 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:53 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:19:54.036 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:54.036 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:53 smithi139 ceph-mon[32760]: osd.6 [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] boot 2024-04-25T01:19:54.036 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:53 smithi139 ceph-mon[32760]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:19:54.036 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:19:54.348 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_2 2024-04-25T01:19:54.584 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:19:54.939 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:54 smithi139 ceph-mon[32760]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:54.940 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:54 smithi139 ceph-mon[32760]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:19:54.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:54 smithi027 ceph-mon[29197]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:54.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:54 smithi027 ceph-mon[29197]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:19:55.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:54 smithi153 ceph-mon[32621]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:19:55.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:54 smithi153 ceph-mon[32621]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:19:56.194 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.209+0000 7f14231a0700 1 -- 172.21.15.139:0/2618816841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c102c60 msgr2=0x7f141c109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.209+0000 7f14231a0700 1 --2- 172.21.15.139:0/2618816841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c102c60 0x7f141c109510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f141400b0d0 tx=0x7f141400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 -- 172.21.15.139:0/2618816841 shutdown_connections 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 --2- 172.21.15.139:0/2618816841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c102c60 0x7f141c109510 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 --2- 172.21.15.139:0/2618816841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1010c0 0x7f141c1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 --2- 172.21.15.139:0/2618816841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c1022c0 0x7f141c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 -- 172.21.15.139:0/2618816841 >> 172.21.15.139:0/2618816841 conn(0x7f141c0fc640 msgr2=0x7f141c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 -- 172.21.15.139:0/2618816841 shutdown_connections 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 -- 172.21.15.139:0/2618816841 wait complete. 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 Processor -- start 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.210+0000 7f14231a0700 1 -- start start 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14231a0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c1010c0 0x7f141c06aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:56.195 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14231a0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1022c0 0x7f141c19d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14231a0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c102c60 0x7f141c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14231a0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f141c10cda0 con 0x7f141c102c60 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14231a0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f141c10cc20 con 0x7f141c1010c0 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14231a0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f141c10cf20 con 0x7f141c1022c0 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c1010c0 0x7f141c06aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c1010c0 0x7f141c06aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:39940/0 (socket says 172.21.15.139:39940) 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 -- 172.21.15.139:0/3441341557 learned_addr learned my addr 172.21.15.139:0/3441341557 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142299f700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c102c60 0x7f141c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142199d700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1022c0 0x7f141c19d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 -- 172.21.15.139:0/3441341557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1022c0 msgr2=0x7f141c19d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1022c0 0x7f141c19d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 -- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c102c60 msgr2=0x7f141c19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c102c60 0x7f141c19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:56.196 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 -- 172.21.15.139:0/3441341557 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c1a00e0 con 0x7f141c1010c0 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142299f700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c102c60 0x7f141c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142199d700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1022c0 0x7f141c19d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f142219e700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c1010c0 0x7f141c06aa90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f141800b500 tx=0x7f141800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14137fe700 1 -- 172.21.15.139:0/3441341557 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418015020 con 0x7f141c1010c0 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14137fe700 1 -- 172.21.15.139:0/3441341557 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1418011d10 con 0x7f141c1010c0 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141c1a03d0 con 0x7f141c1010c0 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.211+0000 7f14137fe700 1 -- 172.21.15.139:0/3441341557 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141800f2e0 con 0x7f141c1010c0 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.212+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141c1a0940 con 0x7f141c1010c0 2024-04-25T01:19:56.197 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.212+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141c068f20 con 0x7f141c1010c0 2024-04-25T01:19:56.198 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.214+0000 7f14137fe700 1 -- 172.21.15.139:0/3441341557 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1418018020 con 0x7f141c1010c0 2024-04-25T01:19:56.198 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.214+0000 7f14137fe700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14080a2c20 0x7f14080a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:56.198 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.214+0000 7f14137fe700 1 -- 172.21.15.139:0/3441341557 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f14180ce130 con 0x7f141c1010c0 2024-04-25T01:19:56.198 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.214+0000 7f142199d700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14080a2c20 0x7f14080a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:56.198 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.215+0000 7f142199d700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14080a2c20 0x7f14080a50e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f141c06aff0 tx=0x7f140c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:56.200 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.218+0000 7f14137fe700 1 -- 172.21.15.139:0/3441341557 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f141801f020 con 0x7f141c1010c0 2024-04-25T01:19:56.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi027 2024-04-25T01:19:56.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:56.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:19:56.422+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f141c062670 con 0x7f14080a2c20 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi027 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:56.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:56.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:56.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:56.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi027 2024-04-25T01:19:56.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:56.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:19:57.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:57 smithi139 ceph-mon[32760]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:19:57.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:57.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:57.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:57.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:57 smithi027 ceph-mon[29197]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:19:57.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:57.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:57.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:57.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:57 smithi153 ceph-mon[32621]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:19:57.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:57.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:57.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:58.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:58 smithi139 ceph-mon[32760]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:58.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:58 smithi027 ceph-mon[29197]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:58.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:58 smithi153 ceph-mon[32621]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:59.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:59 smithi027 ceph-mon[29197]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:19:59.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:59 smithi027 ceph-mon[29197]: from='client.? 172.21.15.139:0/3139379771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]: dispatch 2024-04-25T01:19:59.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:59 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]: dispatch 2024-04-25T01:19:59.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:59 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]': finished 2024-04-25T01:19:59.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:59 smithi027 ceph-mon[29197]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:19:59.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:19:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:19:59.481 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:59 smithi139 ceph-mon[32760]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:19:59.482 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:59 smithi139 ceph-mon[32760]: from='client.? 172.21.15.139:0/3139379771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]: dispatch 2024-04-25T01:19:59.482 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:59 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]: dispatch 2024-04-25T01:19:59.482 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:59 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]': finished 2024-04-25T01:19:59.482 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:59 smithi139 ceph-mon[32760]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:19:59.482 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:19:59 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:19:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:59 smithi153 ceph-mon[32621]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:19:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:59 smithi153 ceph-mon[32621]: from='client.? 172.21.15.139:0/3139379771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]: dispatch 2024-04-25T01:19:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:59 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]: dispatch 2024-04-25T01:19:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:59 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6f1241c-9012-47d6-80e3-ef957fca03f7"}]': finished 2024-04-25T01:19:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:59 smithi153 ceph-mon[32621]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:19:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:19:59 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:00.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:00 smithi027 ceph-mon[29197]: from='client.? 172.21.15.139:0/1443013231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:00.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:00 smithi027 ceph-mon[29197]: overall HEALTH_OK 2024-04-25T01:20:00.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:00 smithi153 ceph-mon[32621]: from='client.? 172.21.15.139:0/1443013231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:00.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:00 smithi153 ceph-mon[32621]: overall HEALTH_OK 2024-04-25T01:20:00.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:00 smithi139 ceph-mon[32760]: from='client.? 172.21.15.139:0/1443013231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:00.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:00 smithi139 ceph-mon[32760]: overall HEALTH_OK 2024-04-25T01:20:01.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:01 smithi027 ceph-mon[29197]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:01.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:01 smithi153 ceph-mon[32621]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:01.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:01 smithi139 ceph-mon[32760]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:03.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:03 smithi027 ceph-mon[29197]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:03.522 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:03 smithi139 ceph-mon[32760]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:03.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:03 smithi153 ceph-mon[32621]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:05.412 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:05 smithi139 ceph-mon[32760]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:05.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:05 smithi027 ceph-mon[29197]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:05.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:05 smithi153 ceph-mon[32621]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:07.415 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:07 smithi139 ceph-mon[32760]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:07.415 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:20:07.415 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:07 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:07.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:07 smithi027 ceph-mon[29197]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:07.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:20:07.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:07 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:07.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:07 smithi153 ceph-mon[32621]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:07.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:20:07.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:07 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:08.381 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:08 smithi139 ceph-mon[32760]: Deploying daemon osd.7 on smithi139 2024-04-25T01:20:08.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:08 smithi027 ceph-mon[29197]: Deploying daemon osd.7 on smithi139 2024-04-25T01:20:08.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:08 smithi153 ceph-mon[32621]: Deploying daemon osd.7 on smithi139 2024-04-25T01:20:09.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:09 smithi027 ceph-mon[29197]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:09.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:09 smithi153 ceph-mon[32621]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:09.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:09 smithi139 ceph-mon[32760]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:11.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:11 smithi027 ceph-mon[29197]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:11.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:11 smithi153 ceph-mon[32621]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:11.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:11 smithi139 ceph-mon[32760]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:13.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:13 smithi153 ceph-mon[32621]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:13.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:13.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:13.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:13.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:13 smithi139 ceph-mon[32760]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:13.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:13.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:13.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:13.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:13 smithi027 ceph-mon[29197]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:13.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:13.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:13.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:15.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:15 smithi153 ceph-mon[32621]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:15.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:15 smithi139 ceph-mon[32760]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:15.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:15 smithi027 ceph-mon[29197]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:15.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.789+0000 7f14137fe700 1 -- 172.21.15.139:0/3441341557 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f141c062670 con 0x7f14080a2c20 2024-04-25T01:20:15.772 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 7 on host 'smithi139' 2024-04-25T01:20:15.775 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14080a2c20 msgr2=0x7f14080a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14080a2c20 0x7f14080a50e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f141c06aff0 tx=0x7f140c008040 comp rx=0 tx=0).stop 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c1010c0 msgr2=0x7f141c06aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c1010c0 0x7f141c06aa90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f141800b500 tx=0x7f141800b8c0 comp rx=0 tx=0).stop 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 shutdown_connections 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f14080a2c20 0x7f14080a50e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c102c60 0x7f141c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c1010c0 0x7f141c06aa90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 --2- 172.21.15.139:0/3441341557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c1022c0 0x7f141c19d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 >> 172.21.15.139:0/3441341557 conn(0x7f141c0fc640 msgr2=0x7f141c1055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.792+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 shutdown_connections 2024-04-25T01:20:15.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:20:15.793+0000 7f14231a0700 1 -- 172.21.15.139:0/3441341557 wait complete. 2024-04-25T01:20:16.449 DEBUG:teuthology.orchestra.run.smithi139:osd.7> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T01:20:16.451 INFO:tasks.cephadm:Deploying osd.8 on smithi153 with /dev/vg_nvme/lv_2... 2024-04-25T01:20:16.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:20:16.722 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:16 smithi139 ceph-mon[32760]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:16 smithi139 ceph-mon[32760]: from='osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:16 smithi139 ceph-mon[32760]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:16.985 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:16 smithi153 ceph-mon[32621]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:16.986 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:16.986 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:16.986 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:16 smithi153 ceph-mon[32621]: from='osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:20:16.986 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:16 smithi153 ceph-mon[32621]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:20:16.986 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:16.986 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:17.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:16 smithi027 ceph-mon[29197]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:17.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:17.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:17.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:16 smithi027 ceph-mon[29197]: from='osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:20:17.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:16 smithi027 ceph-mon[29197]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:20:17.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:17.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:17.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:17 smithi153 ceph-mon[32621]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:20:17.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:17 smithi153 ceph-mon[32621]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:20:17.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:17.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:17 smithi153 ceph-mon[32621]: from='osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:20:17.948 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:17 smithi153 ceph-mon[32621]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:20:18.040 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:20:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:20:17.768+0000 7f2fad11d700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T01:20:18.041 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:20:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:20:17.782+0000 7f2fa8f10700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:20:18.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:17 smithi139 ceph-mon[32760]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:20:18.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:17 smithi139 ceph-mon[32760]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:20:18.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:18.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:17 smithi139 ceph-mon[32760]: from='osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:20:18.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:17 smithi139 ceph-mon[32760]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:20:18.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:17 smithi027 ceph-mon[29197]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:20:18.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:17 smithi027 ceph-mon[29197]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:20:18.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:18.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:17 smithi027 ceph-mon[29197]: from='osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:20:18.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:17 smithi027 ceph-mon[29197]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:20:18.821 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:20:18.821 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:20:18.821 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-25T01:20:18.821 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-25T01:20:18.821 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118394 s, 886 MB/s 2024-04-25T01:20:18.821 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-25T01:20:18.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:18 smithi153 ceph-mon[32621]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:18.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:18 smithi153 ceph-mon[32621]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:20:18.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:18 smithi153 ceph-mon[32621]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:20:18.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:18 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:18.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:18 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:18.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:18 smithi027 ceph-mon[29197]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:18.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:18 smithi027 ceph-mon[29197]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:20:18.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:18 smithi027 ceph-mon[29197]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:20:18.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:18.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:19.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:18 smithi139 ceph-mon[32760]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:20:19.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:18 smithi139 ceph-mon[32760]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:20:19.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:18 smithi139 ceph-mon[32760]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:20:19.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:18 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:19.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:18 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:19.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-04-25T01:20:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:20:20.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] boot 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:20.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:20.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:20.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:20.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.030 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:19 smithi153 ceph-mon[32621]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:20:20.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:20:20.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:20:20.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] boot 2024-04-25T01:20:20.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T01:20:20.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:19 smithi139 ceph-mon[32760]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:20:20.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: osd.7 [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] boot 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:20.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:20.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:19 smithi027 ceph-mon[29197]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:20:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:20 smithi153 ceph-mon[32621]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:20 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi139 2024-04-25T01:20:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:20 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi139 to 5696M 2024-04-25T01:20:21.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:20 smithi153 ceph-mon[32621]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:20:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:20 smithi139 ceph-mon[32760]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:20 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi139 2024-04-25T01:20:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:20 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi139 to 5696M 2024-04-25T01:20:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:20 smithi139 ceph-mon[32760]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:20:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:20 smithi027 ceph-mon[29197]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:21.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:20 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi139 2024-04-25T01:20:21.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:20 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi139 to 5696M 2024-04-25T01:20:21.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:20 smithi027 ceph-mon[29197]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:20:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 -- 172.21.15.153:0/1222719175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102d30 msgr2=0x7f77b81095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 --2- 172.21.15.153:0/1222719175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102d30 0x7f77b81095e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f77b000b0a0 tx=0x7f77b000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:20:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 -- 172.21.15.153:0/1222719175 shutdown_connections 2024-04-25T01:20:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 --2- 172.21.15.153:0/1222719175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102d30 0x7f77b81095e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 --2- 172.21.15.153:0/1222719175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b8101190 0x7f77b81015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 --2- 172.21.15.153:0/1222719175 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77b8102390 0x7f77b81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 -- 172.21.15.153:0/1222719175 >> 172.21.15.153:0/1222719175 conn(0x7f77b80fc730 msgr2=0x7f77b80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 -- 172.21.15.153:0/1222719175 shutdown_connections 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.380+0000 7f77bedd3700 1 -- 172.21.15.153:0/1222719175 wait complete. 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bedd3700 1 Processor -- start 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bedd3700 1 -- start start 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bedd3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b8101190 0x7f77b8197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bedd3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102390 0x7f77b8197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bedd3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77b8102d30 0x7f77b819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bedd3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f77b810cd60 con 0x7f77b8101190 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bedd3700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f77b810cbe0 con 0x7f77b8102d30 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bedd3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f77b810cee0 con 0x7f77b8102390 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77b7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102390 0x7f77b8197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77b7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102390 0x7f77b8197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52150/0 (socket says 172.21.15.153:52150) 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77b7fff700 1 -- 172.21.15.153:0/4060210748 learned_addr learned my addr 172.21.15.153:0/4060210748 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bcb6f700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b8101190 0x7f77b8197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77bd370700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77b8102d30 0x7f77b819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77b7fff700 1 -- 172.21.15.153:0/4060210748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77b8102d30 msgr2=0x7f77b819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77b7fff700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77b8102d30 0x7f77b819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77b7fff700 1 -- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b8101190 msgr2=0x7f77b8197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77b7fff700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b8101190 0x7f77b8197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.381+0000 7f77b7fff700 1 -- 172.21.15.153:0/4060210748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b819ef80 con 0x7f77b8102390 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.382+0000 7f77b7fff700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102390 0x7f77b8197860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f77a400c9e0 tx=0x7f77a400ccf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.382+0000 7f77b5ffb700 1 -- 172.21.15.153:0/4060210748 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4007ac0 con 0x7f77b8102390 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.382+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b819f270 con 0x7f77b8102390 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.382+0000 7f77b5ffb700 1 -- 172.21.15.153:0/4060210748 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a4007c60 con 0x7f77b8102390 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.382+0000 7f77b5ffb700 1 -- 172.21.15.153:0/4060210748 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4018730 con 0x7f77b8102390 2024-04-25T01:20:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.382+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b819f800 con 0x7f77b8102390 2024-04-25T01:20:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.383+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b8068f20 con 0x7f77b8102390 2024-04-25T01:20:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.384+0000 7f77b5ffb700 1 -- 172.21.15.153:0/4060210748 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f77a40188d0 con 0x7f77b8102390 2024-04-25T01:20:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.384+0000 7f77b5ffb700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f77a80a2c20 0x7f77a80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.385+0000 7f77b5ffb700 1 -- 172.21.15.153:0/4060210748 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f77a40ce470 con 0x7f77b8102390 2024-04-25T01:20:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.385+0000 7f77bcb6f700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f77a80a2c20 0x7f77a80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:21.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.385+0000 7f77bcb6f700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f77a80a2c20 0x7f77a80a50e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f77b81021f0 tx=0x7f77ac007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:21.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.389+0000 7f77b5ffb700 1 -- 172.21.15.153:0/4060210748 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f77a4014030 con 0x7f77b8102390 2024-04-25T01:20:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:21.592+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f77b8062670 con 0x7f77a80a2c20 2024-04-25T01:20:22.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:22.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:22.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:22.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:22.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:22.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:22.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:22.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:22.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:23.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:22 smithi153 ceph-mon[32621]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:23.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:22 smithi153 ceph-mon[32621]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:23.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:22 smithi153 ceph-mon[32621]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:20:23.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:22 smithi139 ceph-mon[32760]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:23.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:22 smithi139 ceph-mon[32760]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:23.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:22 smithi139 ceph-mon[32760]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:20:23.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:22 smithi027 ceph-mon[29197]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:23.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:22 smithi027 ceph-mon[29197]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:23.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:22 smithi027 ceph-mon[29197]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:20:24.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:23 smithi139 ceph-mon[32760]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:20:24.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:23 smithi027 ceph-mon[29197]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:20:24.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:23 smithi153 ceph-mon[32621]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:20:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:24 smithi153 ceph-mon[32621]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:24 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/4102811625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]: dispatch 2024-04-25T01:20:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:24 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]: dispatch 2024-04-25T01:20:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:24 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]': finished 2024-04-25T01:20:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:24 smithi153 ceph-mon[32621]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T01:20:25.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:25.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:24 smithi139 ceph-mon[32760]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:25.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:24 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/4102811625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]: dispatch 2024-04-25T01:20:25.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:24 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]: dispatch 2024-04-25T01:20:25.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:24 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]': finished 2024-04-25T01:20:25.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:24 smithi139 ceph-mon[32760]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T01:20:25.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:25.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:24 smithi027 ceph-mon[29197]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:25.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:24 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/4102811625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]: dispatch 2024-04-25T01:20:25.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:24 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]: dispatch 2024-04-25T01:20:25.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:24 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ceef8a0-8a21-4256-ab43-ad6e4c1cda16"}]': finished 2024-04-25T01:20:25.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:24 smithi027 ceph-mon[29197]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T01:20:25.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:26.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:25 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/1330629812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:26.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:25 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/1330629812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:26.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:25 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/1330629812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:27.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:26 smithi139 ceph-mon[32760]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:27.179 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:26 smithi153 ceph-mon[32621]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:27.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:26 smithi027 ceph-mon[29197]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:29.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:28 smithi153 ceph-mon[32621]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:29.122 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:28 smithi139 ceph-mon[32760]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:29.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:28 smithi027 ceph-mon[29197]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:31.005 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:30 smithi153 ceph-mon[32621]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:31.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:30 smithi027 ceph-mon[29197]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:31.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:30 smithi139 ceph-mon[32760]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:32.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:20:32.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:32.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:31 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:20:32.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:31 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:32.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:31 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:20:32.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:31 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:33.004 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:32 smithi153 ceph-mon[32621]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:33.004 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:32 smithi153 ceph-mon[32621]: Deploying daemon osd.8 on smithi153 2024-04-25T01:20:33.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:32 smithi027 ceph-mon[29197]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:33.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:32 smithi027 ceph-mon[29197]: Deploying daemon osd.8 on smithi153 2024-04-25T01:20:33.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:32 smithi139 ceph-mon[32760]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:33.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:32 smithi139 ceph-mon[32760]: Deploying daemon osd.8 on smithi153 2024-04-25T01:20:35.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:34 smithi027 ceph-mon[29197]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:35.223 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:34 smithi153 ceph-mon[32621]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:35.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:34 smithi139 ceph-mon[32760]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:37.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:36 smithi153 ceph-mon[32621]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:37.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:36 smithi027 ceph-mon[29197]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:37.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:36 smithi139 ceph-mon[32760]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:38.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:38 smithi027 ceph-mon[29197]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:38.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:38.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:38.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:38.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:38 smithi153 ceph-mon[32621]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:38.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:38.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:38.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:38 smithi139 ceph-mon[32760]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:40.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:40 smithi027 ceph-mon[29197]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:40.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:40.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:40.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:40.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:40.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:40 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:40.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:40 smithi153 ceph-mon[32621]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:40.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:40.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:40.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:40.744 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:40.744 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:40 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:40.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:40 smithi139 ceph-mon[32760]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:40.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:40.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:40.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:40.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:40.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:40 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:41.094 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 8 on host 'smithi153' 2024-04-25T01:20:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.111+0000 7f77b5ffb700 1 -- 172.21.15.153:0/4060210748 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f77b8062670 con 0x7f77a80a2c20 2024-04-25T01:20:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f77a80a2c20 msgr2=0x7f77a80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:41.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f77a80a2c20 0x7f77a80a50e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f77b81021f0 tx=0x7f77ac007400 comp rx=0 tx=0).stop 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102390 msgr2=0x7f77b8197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102390 0x7f77b8197860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f77a400c9e0 tx=0x7f77a400ccf0 comp rx=0 tx=0).stop 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 shutdown_connections 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f77a80a2c20 0x7f77a80a50e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77b8102d30 0x7f77b819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b8101190 0x7f77b8197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 --2- 172.21.15.153:0/4060210748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77b8102390 0x7f77b8197860 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 >> 172.21.15.153:0/4060210748 conn(0x7f77b80fc730 msgr2=0x7f77b8105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 shutdown_connections 2024-04-25T01:20:41.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:20:41.114+0000 7f77bedd3700 1 -- 172.21.15.153:0/4060210748 wait complete. 2024-04-25T01:20:41.732 DEBUG:teuthology.orchestra.run.smithi153:osd.8> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T01:20:41.734 INFO:tasks.cephadm:Deploying osd.9 on smithi027 with /dev/vg_nvme/lv_1... 2024-04-25T01:20:41.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:20:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:20:42.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:42.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:42.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:42.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:42 smithi027 ceph-mon[29197]: from='osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:20:42.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:42 smithi027 ceph-mon[29197]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:20:42.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:42.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:42.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:42.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:42 smithi153 ceph-mon[32621]: from='osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:20:42.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:42 smithi153 ceph-mon[32621]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:20:42.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:42.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:42.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:42.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:42 smithi139 ceph-mon[32760]: from='osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:20:42.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:42 smithi139 ceph-mon[32760]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:20:43.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:43 smithi153 ceph-mon[32621]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:43.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:43 smithi153 ceph-mon[32621]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:20:43.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:43 smithi153 ceph-mon[32621]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:20:43.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:43 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:43.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:43 smithi153 ceph-mon[32621]: from='osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:20:43.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:43 smithi153 ceph-mon[32621]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:20:43.361 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:20:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:20:43.110+0000 7f1d5f2e1700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T01:20:43.361 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:20:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:20:43.120+0000 7f1d5b0d4700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:20:43.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:43 smithi027 ceph-mon[29197]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:43.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:43 smithi027 ceph-mon[29197]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:20:43.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:43 smithi027 ceph-mon[29197]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:20:43.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:43 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:43.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:43 smithi027 ceph-mon[29197]: from='osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:20:43.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:43 smithi027 ceph-mon[29197]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:20:43.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:43 smithi139 ceph-mon[32760]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:43.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:43 smithi139 ceph-mon[32760]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:20:43.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:43 smithi139 ceph-mon[32760]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:20:43.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:43 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:43.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:43 smithi139 ceph-mon[32760]: from='osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:20:43.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:43 smithi139 ceph-mon[32760]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:20:44.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:44 smithi153 ceph-mon[32621]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:20:44.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:44 smithi153 ceph-mon[32621]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:20:44.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:44.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:44.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:44 smithi153 ceph-mon[32621]: osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] boot 2024-04-25T01:20:44.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:44 smithi153 ceph-mon[32621]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T01:20:44.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:44.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:44 smithi027 ceph-mon[29197]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:20:44.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:44 smithi027 ceph-mon[29197]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:20:44.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:44.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:44.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:44 smithi027 ceph-mon[29197]: osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] boot 2024-04-25T01:20:44.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:44 smithi027 ceph-mon[29197]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T01:20:44.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:44.357 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:20:44.357 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:20:44.357 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T01:20:44.357 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T01:20:44.357 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125985 s, 832 MB/s 2024-04-25T01:20:44.357 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T01:20:44.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:44 smithi139 ceph-mon[32760]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:20:44.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:44 smithi139 ceph-mon[32760]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:20:44.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:44.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:44.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:44 smithi139 ceph-mon[32760]: osd.8 [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] boot 2024-04-25T01:20:44.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:44 smithi139 ceph-mon[32760]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T01:20:44.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:20:45.085 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_1 2024-04-25T01:20:45.336 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:20:45.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:20:45.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:20:45.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:45.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:45 smithi027 ceph-mon[29197]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:45.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:45.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:45 smithi153 ceph-mon[32621]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:45.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:45.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:45.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:20:45.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:45 smithi139 ceph-mon[32760]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T01:20:46.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:46 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi153 2024-04-25T01:20:46.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:46 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-25T01:20:46.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:46 smithi027 ceph-mon[29197]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:20:46.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:46 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi153 2024-04-25T01:20:46.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:46 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-25T01:20:46.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:46 smithi153 ceph-mon[32621]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:20:46.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:46 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi153 2024-04-25T01:20:46.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:46 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-25T01:20:46.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:46 smithi139 ceph-mon[32760]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 -- 172.21.15.27:0/203983202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e601022b0 msgr2=0x7f9e601026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 --2- 172.21.15.27:0/203983202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e601022b0 0x7f9e601026d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e500099c0 tx=0x7f9e50009cd0 comp rx=0 tx=0).stop 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 -- 172.21.15.27:0/203983202 shutdown_connections 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 --2- 172.21.15.27:0/203983202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e60103e50 0x7f9e6010a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 --2- 172.21.15.27:0/203983202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e601022b0 0x7f9e601026d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 --2- 172.21.15.27:0/203983202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9e601034b0 0x7f9e60103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 -- 172.21.15.27:0/203983202 >> 172.21.15.27:0/203983202 conn(0x7f9e600fd850 msgr2=0x7f9e600ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 -- 172.21.15.27:0/203983202 shutdown_connections 2024-04-25T01:20:47.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 -- 172.21.15.27:0/203983202 wait complete. 2024-04-25T01:20:47.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.218+0000 7f9e6527a700 1 Processor -- start 2024-04-25T01:20:47.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e6527a700 1 -- start start 2024-04-25T01:20:47.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e6527a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e601022b0 0x7f9e601984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:47.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e6527a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9e601034b0 0x7f9e60198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:47.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e6527a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e60103e50 0x7f9e6019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:47.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e6527a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010dee0 con 0x7f9e60103e50 2024-04-25T01:20:47.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e6527a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010dd60 con 0x7f9e601034b0 2024-04-25T01:20:47.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e6527a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010e060 con 0x7f9e601022b0 2024-04-25T01:20:47.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:47 smithi027 ceph-mon[29197]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e5f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e60103e50 0x7f9e6019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e5f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e60103e50 0x7f9e6019fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51770/0 (socket says 172.21.15.27:51770) 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e5f7fe700 1 -- 172.21.15.27:0/3327068816 learned_addr learned my addr 172.21.15.27:0/3327068816 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e5e7fc700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9e601034b0 0x7f9e60198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.219+0000 7f9e5effd700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e601022b0 0x7f9e601984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e5f7fe700 1 -- 172.21.15.27:0/3327068816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e601022b0 msgr2=0x7f9e601984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e5f7fe700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e601022b0 0x7f9e601984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e5f7fe700 1 -- 172.21.15.27:0/3327068816 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9e601034b0 msgr2=0x7f9e60198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e5f7fe700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9e601034b0 0x7f9e60198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e5f7fe700 1 -- 172.21.15.27:0/3327068816 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e601a0150 con 0x7f9e60103e50 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e5f7fe700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e60103e50 0x7f9e6019fab0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e58007f10 tx=0x7f9e5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e47fff700 1 -- 172.21.15.27:0/3327068816 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58004bb0 con 0x7f9e60103e50 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e601a0440 con 0x7f9e60103e50 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e47fff700 1 -- 172.21.15.27:0/3327068816 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e58004d50 con 0x7f9e60103e50 2024-04-25T01:20:47.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e47fff700 1 -- 172.21.15.27:0/3327068816 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e580059f0 con 0x7f9e60103e50 2024-04-25T01:20:47.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.220+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e601a0980 con 0x7f9e60103e50 2024-04-25T01:20:47.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.221+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e60068f20 con 0x7f9e60103e50 2024-04-25T01:20:47.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.223+0000 7f9e47fff700 1 -- 172.21.15.27:0/3327068816 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9e5801a020 con 0x7f9e60103e50 2024-04-25T01:20:47.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.223+0000 7f9e47fff700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9e480a2c20 0x7f9e480a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:47.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.223+0000 7f9e5effd700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9e480a2c20 0x7f9e480a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:47.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.223+0000 7f9e47fff700 1 -- 172.21.15.27:0/3327068816 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f9e580cdd70 con 0x7f9e60103e50 2024-04-25T01:20:47.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.224+0000 7f9e5effd700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9e480a2c20 0x7f9e480a50e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000b580 tx=0x7f9e50005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:47.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.228+0000 7f9e47fff700 1 -- 172.21.15.27:0/3327068816 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9e58010070 con 0x7f9e60103e50 2024-04-25T01:20:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:20:47.443+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9e60062670 con 0x7f9e480a2c20 2024-04-25T01:20:47.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:47 smithi153 ceph-mon[32621]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:20:47.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:47 smithi139 ceph-mon[32760]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:20:48.321 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:48 smithi027 ceph-mon[29197]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:20:48.321 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:48 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:48.321 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:48 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:48.322 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:48 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:48.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:48 smithi153 ceph-mon[32621]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:20:48.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:48 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:48.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:48 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:48.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:48 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:48.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:48 smithi139 ceph-mon[32760]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:20:48.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:48.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:48.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:48 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:49.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:49 smithi027 ceph-mon[29197]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:20:49.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:49 smithi027 ceph-mon[29197]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:49.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:49 smithi027 ceph-mon[29197]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:20:49.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:49 smithi153 ceph-mon[32621]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:20:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:49 smithi153 ceph-mon[32621]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:49.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:49 smithi153 ceph-mon[32621]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:20:49.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:49 smithi139 ceph-mon[32760]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:20:49.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:49 smithi139 ceph-mon[32760]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:49.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:49 smithi139 ceph-mon[32760]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:20:51.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:51 smithi027 ceph-mon[29197]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T01:20:51.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:51 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1679061743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "132f821e-9337-40f8-b79b-02322b1a7962"}]: dispatch 2024-04-25T01:20:51.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:51 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1679061743' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "132f821e-9337-40f8-b79b-02322b1a7962"}]': finished 2024-04-25T01:20:51.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:51 smithi027 ceph-mon[29197]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T01:20:51.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:20:51.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:51 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/573928224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:51.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:51 smithi153 ceph-mon[32621]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T01:20:51.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:51 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1679061743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "132f821e-9337-40f8-b79b-02322b1a7962"}]: dispatch 2024-04-25T01:20:51.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:51 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1679061743' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "132f821e-9337-40f8-b79b-02322b1a7962"}]': finished 2024-04-25T01:20:51.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:51 smithi153 ceph-mon[32621]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T01:20:51.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:20:51.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:51 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/573928224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:51.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:51 smithi139 ceph-mon[32760]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T01:20:51.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:51 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1679061743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "132f821e-9337-40f8-b79b-02322b1a7962"}]: dispatch 2024-04-25T01:20:51.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:51 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1679061743' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "132f821e-9337-40f8-b79b-02322b1a7962"}]': finished 2024-04-25T01:20:51.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:51 smithi139 ceph-mon[32760]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T01:20:51.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:20:51.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:51 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/573928224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:53.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:53 smithi027 ceph-mon[29197]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T01:20:53.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:53 smithi153 ceph-mon[32621]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T01:20:53.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:53 smithi139 ceph-mon[32760]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T01:20:55.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:55 smithi027 ceph-mon[29197]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T01:20:55.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:55 smithi153 ceph-mon[32621]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T01:20:55.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:55 smithi139 ceph-mon[32760]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T01:20:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:57 smithi027 ceph-mon[29197]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:20:57.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:57 smithi153 ceph-mon[32621]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:20:57.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:57 smithi139 ceph-mon[32760]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:20:59.159 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:59 smithi027 ceph-mon[29197]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-25T01:20:59.159 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:20:59.159 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:20:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:59 smithi153 ceph-mon[32621]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-25T01:20:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:59 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:20:59.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:20:59 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:59.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:59 smithi139 ceph-mon[32760]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-25T01:20:59.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:59 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:20:59.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:20:59 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:00.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:00 smithi027 ceph-mon[29197]: Deploying daemon osd.9 on smithi027 2024-04-25T01:21:00.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:00 smithi153 ceph-mon[32621]: Deploying daemon osd.9 on smithi027 2024-04-25T01:21:00.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:00 smithi139 ceph-mon[32760]: Deploying daemon osd.9 on smithi027 2024-04-25T01:21:01.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:01 smithi027 ceph-mon[29197]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:01.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:01 smithi153 ceph-mon[32621]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:01.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:01 smithi139 ceph-mon[32760]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:03.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:03 smithi027 ceph-mon[29197]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:03.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:03 smithi153 ceph-mon[32621]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:03.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:03 smithi139 ceph-mon[32760]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:05.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:05 smithi027 ceph-mon[29197]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:05.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:05.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:05.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:05.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:05 smithi153 ceph-mon[32621]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:05.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:05.530 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:05.531 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:05.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:05 smithi139 ceph-mon[32760]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:05.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:05.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:05.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:07.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:07 smithi027 ceph-mon[29197]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:07.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:07 smithi153 ceph-mon[32621]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:07.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:07 smithi139 ceph-mon[32760]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:08.647 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 9 on host 'smithi027' 2024-04-25T01:21:08.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.665+0000 7f9e47fff700 1 -- 172.21.15.27:0/3327068816 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9e60062670 con 0x7f9e480a2c20 2024-04-25T01:21:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9e480a2c20 msgr2=0x7f9e480a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9e480a2c20 0x7f9e480a50e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000b580 tx=0x7f9e50005fb0 comp rx=0 tx=0).stop 2024-04-25T01:21:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e60103e50 msgr2=0x7f9e6019fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e60103e50 0x7f9e6019fab0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e58007f10 tx=0x7f9e5800d3b0 comp rx=0 tx=0).stop 2024-04-25T01:21:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 shutdown_connections 2024-04-25T01:21:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9e480a2c20 0x7f9e480a50e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e60103e50 0x7f9e6019fab0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e601022b0 0x7f9e601984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 --2- 172.21.15.27:0/3327068816 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9e601034b0 0x7f9e60198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 >> 172.21.15.27:0/3327068816 conn(0x7f9e600fd850 msgr2=0x7f9e601067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 shutdown_connections 2024-04-25T01:21:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:21:08.668+0000 7f9e6527a700 1 -- 172.21.15.27:0/3327068816 wait complete. 2024-04-25T01:21:08.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:08.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:08.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:08.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:08.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:08.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:08.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:21:08.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:08.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:08.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:08 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:09.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:09.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:21:09.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:08 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:09.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:08 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:09.488 DEBUG:teuthology.orchestra.run.smithi027:osd.9> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T01:21:09.490 INFO:tasks.cephadm:Deploying osd.10 on smithi139 with /dev/vg_nvme/lv_1... 2024-04-25T01:21:09.490 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:21:09.738 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:21:10.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:10 smithi027 ceph-mon[29197]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:21:10.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:10 smithi027 ceph-mon[29197]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T01:21:10.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:10 smithi027 ceph-mon[29197]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:21:10.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:10 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:10.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:21:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:21:10.178+0000 7f12bbaff700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T01:21:10.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:21:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:21:10.193+0000 7f12b78f2700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:10 smithi153 ceph-mon[32621]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:21:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:10 smithi153 ceph-mon[32621]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T01:21:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:10 smithi153 ceph-mon[32621]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:21:10.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:10 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:10.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:10 smithi139 ceph-mon[32760]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:21:10.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:10 smithi139 ceph-mon[32760]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T01:21:10.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:10 smithi139 ceph-mon[32760]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T01:21:10.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:10 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:11.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:11 smithi027 ceph-mon[29197]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:11.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:11 smithi027 ceph-mon[29197]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:21:11.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:11 smithi027 ceph-mon[29197]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:21:11.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:11.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:11 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:11 smithi153 ceph-mon[32621]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:11.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:11 smithi153 ceph-mon[32621]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:21:11.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:11 smithi153 ceph-mon[32621]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:21:11.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:11.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:11 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:11.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:11 smithi139 ceph-mon[32760]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:11.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:11 smithi139 ceph-mon[32760]: from='osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T01:21:11.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:11 smithi139 ceph-mon[32760]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:21:11.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:11.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:11 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:11.710 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:21:11.710 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:21:11.710 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-04-25T01:21:11.710 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-04-25T01:21:11.710 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138815 s, 755 MB/s 2024-04-25T01:21:11.710 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-04-25T01:21:12.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:12 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:21:12.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:12 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:21:12.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:12 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:12.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:12 smithi027 ceph-mon[29197]: osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] boot 2024-04-25T01:21:12.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:12 smithi027 ceph-mon[29197]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T01:21:12.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:12 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:12.489 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:12 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:21:12.489 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:12 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:21:12.489 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:12 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:12.489 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:12 smithi139 ceph-mon[32760]: osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] boot 2024-04-25T01:21:12.489 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:12 smithi139 ceph-mon[32760]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T01:21:12.489 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:12 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:12.523 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_1 2024-04-25T01:21:12.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:12 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:21:12.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:12 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:21:12.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:12 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:12.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:12 smithi153 ceph-mon[32621]: osd.9 [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] boot 2024-04-25T01:21:12.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:12 smithi153 ceph-mon[32621]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T01:21:12.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:12 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:21:12.766 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:21:13.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:13.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:13 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:13.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:13 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:21:13.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T01:21:13.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:13.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:13 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:14.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 -- 172.21.15.139:0/1402223986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 msgr2=0x7f1d241009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:14.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 --2- 172.21.15.139:0/1402223986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 0x7f1d241009c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d14009990 tx=0x7f1d14009ca0 comp rx=0 tx=0).stop 2024-04-25T01:21:14.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 -- 172.21.15.139:0/1402223986 shutdown_connections 2024-04-25T01:21:14.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 --2- 172.21.15.139:0/1402223986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d24103f20 0x7f1d2410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 --2- 172.21.15.139:0/1402223986 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 0x7f1d241009c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 --2- 172.21.15.139:0/1402223986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d24103580 0x7f1d241039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 -- 172.21.15.139:0/1402223986 >> 172.21.15.139:0/1402223986 conn(0x7f1d240fbf00 msgr2=0x7f1d240fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 -- 172.21.15.139:0/1402223986 shutdown_connections 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 -- 172.21.15.139:0/1402223986 wait complete. 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 Processor -- start 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.456+0000 7f1d296e9700 1 -- start start 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d296e9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 0x7f1d241984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d296e9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d24103580 0x7f1d24198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d296e9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d24103f20 0x7f1d2419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d296e9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2410deb0 con 0x7f1d24103f20 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d296e9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2410dd30 con 0x7f1d241005a0 2024-04-25T01:21:14.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d296e9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2410e030 con 0x7f1d24103580 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 0x7f1d241984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 0x7f1d241984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:39388/0 (socket says 172.21.15.139:39388) 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 -- 172.21.15.139:0/4097966967 learned_addr learned my addr 172.21.15.139:0/4097966967 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d237fe700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d24103f20 0x7f1d2419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d227fc700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d24103580 0x7f1d24198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 -- 172.21.15.139:0/4097966967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d24103580 msgr2=0x7f1d24198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d24103580 0x7f1d24198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 -- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d24103f20 msgr2=0x7f1d2419fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d24103f20 0x7f1d2419fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 -- 172.21.15.139:0/4097966967 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d241a0150 con 0x7f1d241005a0 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d237fe700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d24103f20 0x7f1d2419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d22ffd700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 0x7f1d241984f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1d140039c0 tx=0x7f1d140047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:14.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d0bfff700 1 -- 172.21.15.139:0/4097966967 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14003d30 con 0x7f1d241005a0 2024-04-25T01:21:14.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d0bfff700 1 -- 172.21.15.139:0/4097966967 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d14004830 con 0x7f1d241005a0 2024-04-25T01:21:14.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d241a03e0 con 0x7f1d241005a0 2024-04-25T01:21:14.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d0bfff700 1 -- 172.21.15.139:0/4097966967 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d140054b0 con 0x7f1d241005a0 2024-04-25T01:21:14.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.457+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d241a07d0 con 0x7f1d241005a0 2024-04-25T01:21:14.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.459+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d24068f20 con 0x7f1d241005a0 2024-04-25T01:21:14.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.459+0000 7f1d0bfff700 1 -- 172.21.15.139:0/4097966967 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1d14003ed0 con 0x7f1d241005a0 2024-04-25T01:21:14.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.460+0000 7f1d0bfff700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1d0c0a2c20 0x7f1d0c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:14.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.460+0000 7f1d0bfff700 1 -- 172.21.15.139:0/4097966967 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f1d140cf8c0 con 0x7f1d241005a0 2024-04-25T01:21:14.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.460+0000 7f1d227fc700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1d0c0a2c20 0x7f1d0c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:14.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.461+0000 7f1d227fc700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1d0c0a2c20 0x7f1d0c0a50e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1d18005950 tx=0x7f1d180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:14.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.465+0000 7f1d0bfff700 1 -- 172.21.15.139:0/4097966967 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1d14013070 con 0x7f1d241005a0 2024-04-25T01:21:14.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:14 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi027 2024-04-25T01:21:14.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:14 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T01:21:14.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:14 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi027 2024-04-25T01:21:14.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:14 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T01:21:14.649 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:14.667+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1d24062670 con 0x7f1d0c0a2c20 2024-04-25T01:21:14.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:14 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi027 2024-04-25T01:21:14.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:14 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T01:21:15.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:15 smithi153 ceph-mon[32621]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:15.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:15 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:15.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:15 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:15.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:15 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:15.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:15 smithi139 ceph-mon[32760]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:15.529 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:15 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:15.529 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:15 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:15.529 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:15 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:15 smithi027 ceph-mon[29197]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:15.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:15 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:16.475 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:16 smithi139 ceph-mon[32760]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:16.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:16 smithi153 ceph-mon[32621]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:16.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:16 smithi027 ceph-mon[29197]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:17.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:17 smithi153 ceph-mon[32621]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:17.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:17 smithi139 ceph-mon[32760]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:17.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:17 smithi027 ceph-mon[29197]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:18.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:18 smithi153 ceph-mon[32621]: from='client.? 172.21.15.139:0/3260741401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]: dispatch 2024-04-25T01:21:18.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:18 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]: dispatch 2024-04-25T01:21:18.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:18 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]': finished 2024-04-25T01:21:18.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:18 smithi153 ceph-mon[32621]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T01:21:18.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:18 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:18.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:18 smithi153 ceph-mon[32621]: from='client.? 172.21.15.139:0/474893653' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:18.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:18 smithi027 ceph-mon[29197]: from='client.? 172.21.15.139:0/3260741401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]: dispatch 2024-04-25T01:21:18.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:18 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]: dispatch 2024-04-25T01:21:18.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:18 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]': finished 2024-04-25T01:21:18.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:18 smithi027 ceph-mon[29197]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T01:21:18.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:18 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:18.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:18 smithi027 ceph-mon[29197]: from='client.? 172.21.15.139:0/474893653' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:18.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:18 smithi139 ceph-mon[32760]: from='client.? 172.21.15.139:0/3260741401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]: dispatch 2024-04-25T01:21:18.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:18 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]: dispatch 2024-04-25T01:21:18.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:18 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6ee88f9-6df0-4fbe-af82-401e2c864c2d"}]': finished 2024-04-25T01:21:18.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:18 smithi139 ceph-mon[32760]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T01:21:18.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:18 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:18.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:18 smithi139 ceph-mon[32760]: from='client.? 172.21.15.139:0/474893653' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:19.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:19 smithi153 ceph-mon[32621]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:19.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:19 smithi139 ceph-mon[32760]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:19.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:19 smithi027 ceph-mon[29197]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:21.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:21 smithi139 ceph-mon[32760]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:21.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:21 smithi153 ceph-mon[32621]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:21.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:21 smithi027 ceph-mon[29197]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:23.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:23 smithi153 ceph-mon[32621]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:23.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:23 smithi139 ceph-mon[32760]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:23.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:23 smithi027 ceph-mon[29197]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:25 smithi153 ceph-mon[32621]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:25 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:21:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:25 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:25 smithi139 ceph-mon[32760]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:25 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:21:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:25 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:25.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:25 smithi027 ceph-mon[29197]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:25.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:25 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:21:25.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:25 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:26.516 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:26 smithi139 ceph-mon[32760]: Deploying daemon osd.10 on smithi139 2024-04-25T01:21:26.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:26 smithi153 ceph-mon[32621]: Deploying daemon osd.10 on smithi139 2024-04-25T01:21:26.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:26 smithi027 ceph-mon[29197]: Deploying daemon osd.10 on smithi139 2024-04-25T01:21:27.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:27 smithi153 ceph-mon[32621]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:27.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:27 smithi139 ceph-mon[32760]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:27.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:27 smithi027 ceph-mon[29197]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:28.537 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:28 smithi139 ceph-mon[32760]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:28.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:28 smithi027 ceph-mon[29197]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:28.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:28 smithi153 ceph-mon[32621]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:30.463 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:30 smithi139 ceph-mon[32760]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:30.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:30 smithi027 ceph-mon[29197]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:30.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:30 smithi153 ceph-mon[32621]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:31.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:31.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:31.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:31 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:31.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:31 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:31.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:31 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:31.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:31 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:31.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:31 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:31.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:31 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:31.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:31 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:32.739 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:32 smithi139 ceph-mon[32760]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:32.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:32 smithi153 ceph-mon[32621]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:32.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:32 smithi027 ceph-mon[29197]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:34.417 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 10 on host 'smithi139' 2024-04-25T01:21:34.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.435+0000 7f1d0bfff700 1 -- 172.21.15.139:0/4097966967 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1d24062670 con 0x7f1d0c0a2c20 2024-04-25T01:21:34.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1d0c0a2c20 msgr2=0x7f1d0c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:34.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1d0c0a2c20 0x7f1d0c0a50e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1d18005950 tx=0x7f1d180058e0 comp rx=0 tx=0).stop 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 msgr2=0x7f1d241984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 0x7f1d241984f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1d140039c0 tx=0x7f1d140047b0 comp rx=0 tx=0).stop 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 shutdown_connections 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1d0c0a2c20 0x7f1d0c0a50e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d24103f20 0x7f1d2419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d241005a0 0x7f1d241984f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 --2- 172.21.15.139:0/4097966967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d24103580 0x7f1d24198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 >> 172.21.15.139:0/4097966967 conn(0x7f1d240fbf00 msgr2=0x7f1d241068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 shutdown_connections 2024-04-25T01:21:34.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-04-25T01:21:34.438+0000 7f1d296e9700 1 -- 172.21.15.139:0/4097966967 wait complete. 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:34 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:34.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:34.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:21:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:21:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:34.948 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:34 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:35.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:35.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:34 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:35.071 DEBUG:teuthology.orchestra.run.smithi139:osd.10> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T01:21:35.073 INFO:tasks.cephadm:Deploying osd.11 on smithi153 with /dev/vg_nvme/lv_1... 2024-04-25T01:21:35.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:21:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:21:36.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:36 smithi139 ceph-mon[32760]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:36.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:36 smithi139 ceph-mon[32760]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:21:36.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:36 smithi139 ceph-mon[32760]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T01:21:36.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:36 smithi139 ceph-mon[32760]: from='osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:21:36.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:36.528 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:36 smithi139 ceph-mon[32760]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:21:36.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:36 smithi027 ceph-mon[29197]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:36.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:36 smithi027 ceph-mon[29197]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:21:36.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:36 smithi027 ceph-mon[29197]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T01:21:36.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:36 smithi027 ceph-mon[29197]: from='osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:21:36.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:36.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:36 smithi027 ceph-mon[29197]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:21:36.721 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:36 smithi153 ceph-mon[32621]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:36.721 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:36 smithi153 ceph-mon[32621]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:21:36.721 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:36 smithi153 ceph-mon[32621]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T01:21:36.721 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:36 smithi153 ceph-mon[32621]: from='osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:21:36.721 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:36.721 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:36 smithi153 ceph-mon[32621]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-04-25T01:21:37.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:21:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:21:36.856+0000 7f5ad65b6700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T01:21:37.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:21:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:21:36.867+0000 7f5ace3a1700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:37.434 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:21:37.434 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:21:37.434 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-25T01:21:37.434 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-25T01:21:37.434 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0117384 s, 893 MB/s 2024-04-25T01:21:37.434 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-25T01:21:37.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:37 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:21:37.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:37 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:21:37.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:37 smithi027 ceph-mon[29197]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:21:37.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:37 smithi027 ceph-mon[29197]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:21:37.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:37 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:37.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:37 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:37.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:37 smithi027 ceph-mon[29197]: from='osd.10 ' entity='osd.10' 2024-04-25T01:21:37.737 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:37 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:21:37.737 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:37 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:21:37.737 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:37 smithi153 ceph-mon[32621]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:21:37.737 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:37 smithi153 ceph-mon[32621]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:21:37.738 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:37 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:37.738 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:37 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:37.738 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:37 smithi153 ceph-mon[32621]: from='osd.10 ' entity='osd.10' 2024-04-25T01:21:37.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:37 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:21:37.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:37 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:21:37.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:37 smithi139 ceph-mon[32760]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-04-25T01:21:37.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:37 smithi139 ceph-mon[32760]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:21:37.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:37 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:37.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:37 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:37.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:37 smithi139 ceph-mon[32760]: from='osd.10 ' entity='osd.10' 2024-04-25T01:21:38.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-04-25T01:21:38.478 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:21:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:38 smithi153 ceph-mon[32621]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:38 smithi153 ceph-mon[32621]: osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] boot 2024-04-25T01:21:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:38 smithi153 ceph-mon[32621]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T01:21:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:38.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:38 smithi027 ceph-mon[29197]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:38.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:38.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:38 smithi027 ceph-mon[29197]: osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] boot 2024-04-25T01:21:38.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:38 smithi027 ceph-mon[29197]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T01:21:38.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:38 smithi139 ceph-mon[32760]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:21:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:38 smithi139 ceph-mon[32760]: osd.10 [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] boot 2024-04-25T01:21:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:38 smithi139 ceph-mon[32760]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T01:21:38.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi139 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi139 to 4272M 2024-04-25T01:21:39.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:39.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:39.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.742 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi139 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi139 to 4272M 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:39.743 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T01:21:39.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi139 2024-04-25T01:21:39.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:39.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi139 to 4272M 2024-04-25T01:21:39.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:39.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:39.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:39.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:40.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.321+0000 7f63e0c34700 1 -- 172.21.15.153:0/3312259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102d20 msgr2=0x7f63dc1095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.321+0000 7f63e0c34700 1 --2- 172.21.15.153:0/3312259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102d20 0x7f63dc1095d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f63d400b0a0 tx=0x7f63d400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 -- 172.21.15.153:0/3312259 shutdown_connections 2024-04-25T01:21:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 --2- 172.21.15.153:0/3312259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102d20 0x7f63dc1095d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 --2- 172.21.15.153:0/3312259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63dc101180 0x7f63dc1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 --2- 172.21.15.153:0/3312259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63dc102380 0x7f63dc1027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 -- 172.21.15.153:0/3312259 >> 172.21.15.153:0/3312259 conn(0x7f63dc0fc720 msgr2=0x7f63dc0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 -- 172.21.15.153:0/3312259 shutdown_connections 2024-04-25T01:21:40.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 -- 172.21.15.153:0/3312259 wait complete. 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 Processor -- start 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.322+0000 7f63e0c34700 1 -- start start 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63e0c34700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63dc101180 0x7f63dc19a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63e0c34700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102380 0x7f63dc19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63e0c34700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63dc102d20 0x7f63dc1a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63e0c34700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc10cd70 con 0x7f63dc102d20 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63e0c34700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc10cbf0 con 0x7f63dc101180 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63e0c34700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc10cef0 con 0x7f63dc102380 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102380 0x7f63dc19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102380 0x7f63dc19aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60976/0 (socket says 172.21.15.153:60976) 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 -- 172.21.15.153:0/1168781636 learned_addr learned my addr 172.21.15.153:0/1168781636 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:21:40.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63dad9d700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63dc102d20 0x7f63dc1a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63da59c700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63dc101180 0x7f63dc19a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 -- 172.21.15.153:0/1168781636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63dc101180 msgr2=0x7f63dc19a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63dc101180 0x7f63dc19a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 -- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63dc102d20 msgr2=0x7f63dc1a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63dc102d20 0x7f63dc1a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 -- 172.21.15.153:0/1168781636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc1a21e0 con 0x7f63dc102380 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63d9d9b700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102380 0x7f63dc19aac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f63d000c9e0 tx=0x7f63d000ccf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63cb7fe700 1 -- 172.21.15.153:0/1168781636 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0007ac0 con 0x7f63dc102380 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63cb7fe700 1 -- 172.21.15.153:0/1168781636 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63d0007c60 con 0x7f63dc102380 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc1a24d0 con 0x7f63dc102380 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63cb7fe700 1 -- 172.21.15.153:0/1168781636 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0018660 con 0x7f63dc102380 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.323+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc1a2a10 con 0x7f63dc102380 2024-04-25T01:21:40.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.324+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63dc068f20 con 0x7f63dc102380 2024-04-25T01:21:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.326+0000 7f63cb7fe700 1 -- 172.21.15.153:0/1168781636 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f63d000f450 con 0x7f63dc102380 2024-04-25T01:21:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.327+0000 7f63cb7fe700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f63c40a2c20 0x7f63c40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.327+0000 7f63cb7fe700 1 -- 172.21.15.153:0/1168781636 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f63d00cebd0 con 0x7f63dc102380 2024-04-25T01:21:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.327+0000 7f63da59c700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f63c40a2c20 0x7f63c40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.327+0000 7f63da59c700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f63c40a2c20 0x7f63c40a50e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f63dc1021e0 tx=0x7f63cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:40.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.330+0000 7f63cb7fe700 1 -- 172.21.15.153:0/1168781636 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f63d0014030 con 0x7f63dc102380 2024-04-25T01:21:40.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:21:40.534+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f63dc1a2cd0 con 0x7f63c40a2c20 2024-04-25T01:21:40.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:40 smithi027 ceph-mon[29197]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:40.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:40 smithi027 ceph-mon[29197]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:21:40.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:40 smithi027 ceph-mon[29197]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:21:40.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:40 smithi153 ceph-mon[32621]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:40.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:40 smithi153 ceph-mon[32621]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:21:40.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:40 smithi153 ceph-mon[32621]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:21:40.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:40 smithi139 ceph-mon[32760]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:40.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:40 smithi139 ceph-mon[32760]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:21:40.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:40 smithi139 ceph-mon[32760]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:21:41.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:41 smithi027 ceph-mon[29197]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:41.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:41.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:41.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:41.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:41 smithi027 ceph-mon[29197]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:21:41.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:41 smithi153 ceph-mon[32621]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:41.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:41.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:41.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:41.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:41 smithi153 ceph-mon[32621]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:21:41.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:41 smithi139 ceph-mon[32760]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:41.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:41.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:41.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:41.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:41 smithi139 ceph-mon[32760]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:21:43.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:42 smithi027 ceph-mon[29197]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:43.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:42 smithi153 ceph-mon[32621]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:43.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:42 smithi139 ceph-mon[32760]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:44.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:43 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/147792826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]: dispatch 2024-04-25T01:21:44.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:43 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]: dispatch 2024-04-25T01:21:44.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:43 smithi153 ceph-mon[32621]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]': finished 2024-04-25T01:21:44.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:43 smithi153 ceph-mon[32621]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T01:21:44.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:43 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:21:44.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:43 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/147792826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]: dispatch 2024-04-25T01:21:44.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:43 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]: dispatch 2024-04-25T01:21:44.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:43 smithi027 ceph-mon[29197]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]': finished 2024-04-25T01:21:44.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:43 smithi027 ceph-mon[29197]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T01:21:44.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:43 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:21:44.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:43 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/147792826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]: dispatch 2024-04-25T01:21:44.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:43 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]: dispatch 2024-04-25T01:21:44.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:43 smithi139 ceph-mon[32760]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79072415-ed78-47ee-b950-e02e8c9c30c7"}]': finished 2024-04-25T01:21:44.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:43 smithi139 ceph-mon[32760]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T01:21:44.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:43 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:21:45.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:44 smithi027 ceph-mon[29197]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:45.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:44 smithi027 ceph-mon[29197]: from='client.? 172.21.15.153:0/2086137896' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:45.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:44 smithi153 ceph-mon[32621]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:45.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:44 smithi153 ceph-mon[32621]: from='client.? 172.21.15.153:0/2086137896' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:45.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:44 smithi139 ceph-mon[32760]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:45.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:44 smithi139 ceph-mon[32760]: from='client.? 172.21.15.153:0/2086137896' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:47.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:46 smithi027 ceph-mon[29197]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T01:21:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:46 smithi153 ceph-mon[32621]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T01:21:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:46 smithi139 ceph-mon[32760]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T01:21:49.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:48 smithi153 ceph-mon[32621]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T01:21:49.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:48 smithi027 ceph-mon[29197]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T01:21:49.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:48 smithi139 ceph-mon[32760]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T01:21:51.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:50 smithi027 ceph-mon[29197]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T01:21:51.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:50 smithi153 ceph-mon[32621]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T01:21:51.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:50 smithi139 ceph-mon[32760]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T01:21:52.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:21:52.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:51 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:52.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:51 smithi027 ceph-mon[29197]: Deploying daemon osd.11 on smithi153 2024-04-25T01:21:52.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:21:52.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:51 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:52.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:51 smithi153 ceph-mon[32621]: Deploying daemon osd.11 on smithi153 2024-04-25T01:21:52.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:21:52.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:51 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:52.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:51 smithi139 ceph-mon[32760]: Deploying daemon osd.11 on smithi153 2024-04-25T01:21:53.127 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:52 smithi153 ceph-mon[32621]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:53.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:52 smithi027 ceph-mon[29197]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:53.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:52 smithi139 ceph-mon[32760]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:55.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:54 smithi027 ceph-mon[29197]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T01:21:55.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:54 smithi153 ceph-mon[32621]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T01:21:55.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:54 smithi139 ceph-mon[32760]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T01:21:57.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:56 smithi153 ceph-mon[32621]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:21:57.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:57.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:57.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:56 smithi027 ceph-mon[29197]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:21:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:57.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:56 smithi139 ceph-mon[32760]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:21:57.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:57.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:21:57.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:59.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:21:58 smithi153 ceph-mon[32621]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:21:58 smithi027 ceph-mon[29197]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:21:59.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:21:58 smithi139 ceph-mon[32760]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:00.637 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 11 on host 'smithi153' 2024-04-25T01:22:00.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.654+0000 7f63cb7fe700 1 -- 172.21.15.153:0/1168781636 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f63dc1a2cd0 con 0x7f63c40a2c20 2024-04-25T01:22:00.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.657+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f63c40a2c20 msgr2=0x7f63c40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:00.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.657+0000 7f63e0c34700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f63c40a2c20 0x7f63c40a50e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f63dc1021e0 tx=0x7f63cc009260 comp rx=0 tx=0).stop 2024-04-25T01:22:00.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.657+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102380 msgr2=0x7f63dc19aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.657+0000 7f63e0c34700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102380 0x7f63dc19aac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f63d000c9e0 tx=0x7f63d000ccf0 comp rx=0 tx=0).stop 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.658+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 shutdown_connections 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.658+0000 7f63e0c34700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f63c40a2c20 0x7f63c40a50e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.658+0000 7f63e0c34700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63dc102d20 0x7f63dc1a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.658+0000 7f63e0c34700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63dc101180 0x7f63dc19a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.658+0000 7f63e0c34700 1 --2- 172.21.15.153:0/1168781636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63dc102380 0x7f63dc19aac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.658+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 >> 172.21.15.153:0/1168781636 conn(0x7f63dc0fc720 msgr2=0x7f63dc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.658+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 shutdown_connections 2024-04-25T01:22:00.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:00.658+0000 7f63e0c34700 1 -- 172.21.15.153:0/1168781636 wait complete. 2024-04-25T01:22:01.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:01.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.122 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:01.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:01.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:01.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:22:01.123 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:00 smithi153 ceph-mon[32621]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:22:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:01.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:22:01.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:00 smithi027 ceph-mon[29197]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:22:01.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:00 smithi139 ceph-mon[32760]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:22:01.458 DEBUG:teuthology.orchestra.run.smithi153:osd.11> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T01:22:01.461 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T01:22:01.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:22:01.723 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:22:02.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:02 smithi153 ceph-mon[32621]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:02.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:02 smithi153 ceph-mon[32621]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:22:02.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:02 smithi153 ceph-mon[32621]: from='osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:22:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:02 smithi153 ceph-mon[32621]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T01:22:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:02 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:02.690 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:02 smithi153 ceph-mon[32621]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:22:02.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:02 smithi027 ceph-mon[29197]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:02.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:02 smithi027 ceph-mon[29197]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:22:02.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:02 smithi027 ceph-mon[29197]: from='osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:22:02.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:02 smithi027 ceph-mon[29197]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T01:22:02.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:02 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:02.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:02 smithi027 ceph-mon[29197]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:22:03.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:02 smithi139 ceph-mon[32760]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:03.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:02 smithi139 ceph-mon[32760]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:22:03.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:02 smithi139 ceph-mon[32760]: from='osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:22:03.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:02 smithi139 ceph-mon[32760]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T01:22:03.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:02 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:03.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:02 smithi139 ceph-mon[32760]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-25T01:22:03.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:22:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:22:03.085+0000 7fd2fff25700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T01:22:03.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:22:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:22:03.101+0000 7fd2f7d10700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:22:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 -- 172.21.15.27:0/1737045625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102d20 msgr2=0x7f65c81095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1737045625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102d20 0x7f65c81095d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f65c000b0d0 tx=0x7f65c000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 -- 172.21.15.27:0/1737045625 shutdown_connections 2024-04-25T01:22:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1737045625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102d20 0x7f65c81095d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1737045625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65c8101180 0x7f65c81015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1737045625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8102380 0x7f65c81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 -- 172.21.15.27:0/1737045625 >> 172.21.15.27:0/1737045625 conn(0x7f65c80fc760 msgr2=0x7f65c80feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 -- 172.21.15.27:0/1737045625 shutdown_connections 2024-04-25T01:22:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.560+0000 7f65cfa16700 1 -- 172.21.15.27:0/1737045625 wait complete. 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cfa16700 1 Processor -- start 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cfa16700 1 -- start start 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cfa16700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65c8101180 0x7f65c8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cfa16700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102380 0x7f65c8198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cfa16700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8102d20 0x7f65c819fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cfa16700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65c810cd70 con 0x7f65c8102380 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cfa16700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f65c810cbf0 con 0x7f65c8101180 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cfa16700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65c810cef0 con 0x7f65c8102d20 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65ccfb1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102380 0x7f65c8198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65ccfb1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102380 0x7f65c8198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44314/0 (socket says 172.21.15.27:44314) 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65ccfb1700 1 -- 172.21.15.27:0/1266037492 learned_addr learned my addr 172.21.15.27:0/1266037492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cdfb3700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8102d20 0x7f65c819fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65cd7b2700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65c8101180 0x7f65c8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65ccfb1700 1 -- 172.21.15.27:0/1266037492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8102d20 msgr2=0x7f65c819fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65ccfb1700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8102d20 0x7f65c819fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65ccfb1700 1 -- 172.21.15.27:0/1266037492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65c8101180 msgr2=0x7f65c8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65ccfb1700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65c8101180 0x7f65c8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.561+0000 7f65ccfb1700 1 -- 172.21.15.27:0/1266037492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c81a00b0 con 0x7f65c8102380 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.562+0000 7f65ccfb1700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102380 0x7f65c8198990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f65b800e9f0 tx=0x7f65b800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.562+0000 7f65be7fc700 1 -- 172.21.15.27:0/1266037492 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b800cbd0 con 0x7f65c8102380 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.562+0000 7f65be7fc700 1 -- 172.21.15.27:0/1266037492 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65b800cd70 con 0x7f65c8102380 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.562+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c81a03a0 con 0x7f65c8102380 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.562+0000 7f65be7fc700 1 -- 172.21.15.27:0/1266037492 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8010430 con 0x7f65c8102380 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.562+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c81a0890 con 0x7f65c8102380 2024-04-25T01:22:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.563+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c8068f20 con 0x7f65c8102380 2024-04-25T01:22:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.564+0000 7f65be7fc700 1 -- 172.21.15.27:0/1266037492 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f65b8004da0 con 0x7f65c8102380 2024-04-25T01:22:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.565+0000 7f65be7fc700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65b40a2c20 0x7f65b40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.565+0000 7f65cd7b2700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65b40a2c20 0x7f65b40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.565+0000 7f65be7fc700 1 -- 172.21.15.27:0/1266037492 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f65b80cef10 con 0x7f65c8102380 2024-04-25T01:22:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.565+0000 7f65cd7b2700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65b40a2c20 0x7f65b40a50e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f65c4005950 tx=0x7f65c40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:03.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.571+0000 7f65be7fc700 1 -- 172.21.15.27:0/1266037492 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f65b8053740 con 0x7f65c8102380 2024-04-25T01:22:03.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.775+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f65c81a0b50 con 0x7f65c8102380 2024-04-25T01:22:03.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.776+0000 7f65be7fc700 1 -- 172.21.15.27:0/1266037492 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f65b809c430 con 0x7f65c8102380 2024-04-25T01:22:03.758 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65b40a2c20 msgr2=0x7f65b40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65b40a2c20 0x7f65b40a50e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f65c4005950 tx=0x7f65c40058e0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102380 msgr2=0x7f65c8198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102380 0x7f65c8198990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f65b800e9f0 tx=0x7f65b800ed00 comp rx=0 tx=0).stop 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 shutdown_connections 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65b40a2c20 0x7f65b40a50e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65c8102d20 0x7f65c819fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65c8101180 0x7f65c8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 --2- 172.21.15.27:0/1266037492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c8102380 0x7f65c8198990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:03.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 >> 172.21.15.27:0/1266037492 conn(0x7f65c80fc760 msgr2=0x7f65c80fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:03.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 shutdown_connections 2024-04-25T01:22:03.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:03.778+0000 7f65cfa16700 1 -- 172.21.15.27:0/1266037492 wait complete. 2024-04-25T01:22:03.876 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:03 smithi153 ceph-mon[32621]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:22:03.876 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:03 smithi153 ceph-mon[32621]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:22:03.876 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:03 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:03.876 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:03 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:03.876 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:03 smithi153 ceph-mon[32621]: from='osd.11 ' entity='osd.11' 2024-04-25T01:22:03.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:03 smithi027 ceph-mon[29197]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:22:03.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:03 smithi027 ceph-mon[29197]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:22:03.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:03 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:03.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:03 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:03.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:03 smithi027 ceph-mon[29197]: from='osd.11 ' entity='osd.11' 2024-04-25T01:22:04.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:03 smithi139 ceph-mon[32760]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-25T01:22:04.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:03 smithi139 ceph-mon[32760]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:22:04.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:03 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:04.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:03 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:04.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:03 smithi139 ceph-mon[32760]: from='osd.11 ' entity='osd.11' 2024-04-25T01:22:04.404 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1714008097,"num_in_osds":12,"osd_in_since":1714008103,"num_remapped_pgs":0} 2024-04-25T01:22:04.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:04 smithi027 ceph-mon[29197]: purged_snaps scrub starts 2024-04-25T01:22:04.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:04 smithi027 ceph-mon[29197]: purged_snaps scrub ok 2024-04-25T01:22:04.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:04 smithi027 ceph-mon[29197]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:04.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:04 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:04.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1266037492' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:22:04.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:04 smithi027 ceph-mon[29197]: osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] boot 2024-04-25T01:22:04.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:04 smithi027 ceph-mon[29197]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T01:22:04.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:04 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:04.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:04 smithi153 ceph-mon[32621]: purged_snaps scrub starts 2024-04-25T01:22:04.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:04 smithi153 ceph-mon[32621]: purged_snaps scrub ok 2024-04-25T01:22:04.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:04 smithi153 ceph-mon[32621]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:04.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:04 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:04.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:04 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1266037492' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:22:04.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:04 smithi153 ceph-mon[32621]: osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] boot 2024-04-25T01:22:04.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:04 smithi153 ceph-mon[32621]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T01:22:04.723 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:04 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:05.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:04 smithi139 ceph-mon[32760]: purged_snaps scrub starts 2024-04-25T01:22:05.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:04 smithi139 ceph-mon[32760]: purged_snaps scrub ok 2024-04-25T01:22:05.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:04 smithi139 ceph-mon[32760]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:22:05.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:04 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:05.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:04 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1266037492' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:22:05.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:04 smithi139 ceph-mon[32760]: osd.11 [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] boot 2024-04-25T01:22:05.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:04 smithi139 ceph-mon[32760]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T01:22:05.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:04 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:22:05.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:22:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:22:06.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: Detected new or changed devices on smithi153 2024-04-25T01:22:06.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:06.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:05 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: Detected new or changed devices on smithi153 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:06.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:05 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: Detected new or changed devices on smithi153 2024-04-25T01:22:06.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:06.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-25T01:22:06.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:06.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:06.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:06.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:05 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:07.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:06 smithi027 ceph-mon[29197]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:07.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:06 smithi027 ceph-mon[29197]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T01:22:07.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:06 smithi153 ceph-mon[32621]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:07.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:06 smithi153 ceph-mon[32621]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T01:22:07.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:06 smithi139 ceph-mon[32760]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:07.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:06 smithi139 ceph-mon[32760]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T01:22:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1338844445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102390 msgr2=0x7f6a9c1027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1338844445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102390 0x7f6a9c1027f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0099c0 tx=0x7f6a8c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1338844445 shutdown_connections 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1338844445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a9c102d30 0x7f6a9c1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1338844445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a9c101190 0x7f6a9c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1338844445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102390 0x7f6a9c1027f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1338844445 >> 172.21.15.27:0/1338844445 conn(0x7f6a9c0fc730 msgr2=0x7f6a9c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1338844445 shutdown_connections 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.505+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1338844445 wait complete. 2024-04-25T01:22:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.506+0000 7f6aa0e08700 1 Processor -- start 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.506+0000 7f6aa0e08700 1 -- start start 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.506+0000 7f6aa0e08700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a9c101190 0x7f6a9c06b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.506+0000 7f6aa0e08700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a9c102390 0x7f6a9c19d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.506+0000 7f6aa0e08700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102d30 0x7f6a9c19f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.506+0000 7f6aa0e08700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c10ce00 con 0x7f6a9c102d30 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.506+0000 7f6aa0e08700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c10cc80 con 0x7f6a9c102390 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.506+0000 7f6aa0e08700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c10cf80 con 0x7f6a9c101190 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102d30 0x7f6a9c19f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a9c101190 0x7f6a9c06b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102d30 0x7f6a9c19f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44332/0 (socket says 172.21.15.27:44332) 2024-04-25T01:22:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 -- 172.21.15.27:0/1502430060 learned_addr learned my addr 172.21.15.27:0/1502430060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a99d9b700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a9c102390 0x7f6a9c19d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 -- 172.21.15.27:0/1502430060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a9c101190 msgr2=0x7f6a9c06b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a9c101190 0x7f6a9c06b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 -- 172.21.15.27:0/1502430060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a9c102390 msgr2=0x7f6a9c19d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a9c102390 0x7f6a9c19d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 -- 172.21.15.27:0/1502430060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c1a0180 con 0x7f6a9c102d30 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a9ad9d700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102d30 0x7f6a9c19f8b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a94007d70 tx=0x7f6a9400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a8b7fe700 1 -- 172.21.15.27:0/1502430060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94016020 con 0x7f6a9c102d30 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9c1a0470 con 0x7f6a9c102d30 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a8b7fe700 1 -- 172.21.15.27:0/1502430060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a9400ad40 con 0x7f6a9c102d30 2024-04-25T01:22:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6a8b7fe700 1 -- 172.21.15.27:0/1502430060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94011600 con 0x7f6a9c102d30 2024-04-25T01:22:07.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.507+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a9c1a09b0 con 0x7f6a9c102d30 2024-04-25T01:22:07.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.508+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a9c068f20 con 0x7f6a9c102d30 2024-04-25T01:22:07.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.510+0000 7f6a8b7fe700 1 -- 172.21.15.27:0/1502430060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6a94019020 con 0x7f6a9c102d30 2024-04-25T01:22:07.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.510+0000 7f6a8b7fe700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6a840a2c20 0x7f6a840a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:07.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.510+0000 7f6a9a59c700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6a840a2c20 0x7f6a840a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:07.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.510+0000 7f6a8b7fe700 1 -- 172.21.15.27:0/1502430060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6a940cef60 con 0x7f6a9c102d30 2024-04-25T01:22:07.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.511+0000 7f6a9a59c700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6a840a2c20 0x7f6a840a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c1021f0 tx=0x7f6a90009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:07.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.514+0000 7f6a8b7fe700 1 -- 172.21.15.27:0/1502430060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6a94020020 con 0x7f6a9c102d30 2024-04-25T01:22:07.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.723+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f6a9c062e40 con 0x7f6a9c102d30 2024-04-25T01:22:07.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.723+0000 7f6a8b7fe700 1 -- 172.21.15.27:0/1502430060 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f6a9409c360 con 0x7f6a9c102d30 2024-04-25T01:22:07.705 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:07.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6a840a2c20 msgr2=0x7f6a840a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6a840a2c20 0x7f6a840a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c1021f0 tx=0x7f6a90009190 comp rx=0 tx=0).stop 2024-04-25T01:22:07.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102d30 msgr2=0x7f6a9c19f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102d30 0x7f6a9c19f8b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a94007d70 tx=0x7f6a9400a340 comp rx=0 tx=0).stop 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 shutdown_connections 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6a840a2c20 0x7f6a840a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a9c102d30 0x7f6a9c19f8b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a9c101190 0x7f6a9c06b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 --2- 172.21.15.27:0/1502430060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a9c102390 0x7f6a9c19d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 >> 172.21.15.27:0/1502430060 conn(0x7f6a9c0fc730 msgr2=0x7f6a9c105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 shutdown_connections 2024-04-25T01:22:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:07.725+0000 7f6aa0e08700 1 -- 172.21.15.27:0/1502430060 wait complete. 2024-04-25T01:22:08.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:07 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1502430060' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:22:08.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:07 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1502430060' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:22:08.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:07 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1502430060' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:22:08.452 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714008124,"num_in_osds":12,"osd_in_since":1714008103,"num_remapped_pgs":0} 2024-04-25T01:22:08.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:22:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:22:09.061 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:08 smithi027 ceph-mon[29197]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:09.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:08 smithi153 ceph-mon[32621]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:09.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:08 smithi139 ceph-mon[32760]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 -- 172.21.15.27:0/758591076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc87406a980 msgr2=0x7fc87406ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 --2- 172.21.15.27:0/758591076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc87406a980 0x7fc87406ade0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc868009990 tx=0x7fc868009ca0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 -- 172.21.15.27:0/758591076 shutdown_connections 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 --2- 172.21.15.27:0/758591076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc87406b320 0x7fc87410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 --2- 172.21.15.27:0/758591076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8741082b0 0x7fc874108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 --2- 172.21.15.27:0/758591076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc87406a980 0x7fc87406ade0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 -- 172.21.15.27:0/758591076 >> 172.21.15.27:0/758591076 conn(0x7fc874078100 msgr2=0x7fc874078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 -- 172.21.15.27:0/758591076 shutdown_connections 2024-04-25T01:22:10.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.467+0000 7fc8795ad700 1 -- 172.21.15.27:0/758591076 wait complete. 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8795ad700 1 Processor -- start 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8795ad700 1 -- start start 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8795ad700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc87406a980 0x7fc874101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8795ad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc87406b320 0x7fc874101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8795ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8741082b0 0x7fc874101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8795ad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc874112510 con 0x7fc8741082b0 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8795ad700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc874112390 con 0x7fc87406a980 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8795ad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc874112690 con 0x7fc87406b320 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8737fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8741082b0 0x7fc874101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc872ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc87406a980 0x7fc874101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8737fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8741082b0 0x7fc874101cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60020/0 (socket says 172.21.15.27:60020) 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.468+0000 7fc8737fe700 1 -- 172.21.15.27:0/1898474960 learned_addr learned my addr 172.21.15.27:0/1898474960 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8727fc700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc87406b320 0x7fc874101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8737fe700 1 -- 172.21.15.27:0/1898474960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc87406b320 msgr2=0x7fc874101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8737fe700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc87406b320 0x7fc874101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8737fe700 1 -- 172.21.15.27:0/1898474960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc87406a980 msgr2=0x7fc874101230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8737fe700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc87406a980 0x7fc874101230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8737fe700 1 -- 172.21.15.27:0/1898474960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8741a6f50 con 0x7fc8741082b0 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc872ffd700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc87406a980 0x7fc874101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8737fe700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8741082b0 0x7fc874101cb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc86c007d70 tx=0x7fc86c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc85bfff700 1 -- 172.21.15.27:0/1898474960 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c016020 con 0x7fc8741082b0 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc85bfff700 1 -- 172.21.15.27:0/1898474960 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc86c00ac90 con 0x7fc8741082b0 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8741a7240 con 0x7fc8741082b0 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc85bfff700 1 -- 172.21.15.27:0/1898474960 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c0115c0 con 0x7fc8741082b0 2024-04-25T01:22:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.469+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8741a7700 con 0x7fc8741082b0 2024-04-25T01:22:10.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.470+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc874103960 con 0x7fc8741082b0 2024-04-25T01:22:10.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.472+0000 7fc85bfff700 1 -- 172.21.15.27:0/1898474960 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc86c019020 con 0x7fc8741082b0 2024-04-25T01:22:10.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.472+0000 7fc85bfff700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc85c0a2c20 0x7fc85c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:10.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.473+0000 7fc872ffd700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc85c0a2c20 0x7fc85c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:10.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.473+0000 7fc85bfff700 1 -- 172.21.15.27:0/1898474960 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc86c0cef10 con 0x7fc8741082b0 2024-04-25T01:22:10.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.473+0000 7fc872ffd700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc85c0a2c20 0x7fc85c0a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc864005950 tx=0x7fc86400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:10.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.476+0000 7fc85bfff700 1 -- 172.21.15.27:0/1898474960 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc86c020020 con 0x7fc8741082b0 2024-04-25T01:22:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.682+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc874102760 con 0x7fc8741082b0 2024-04-25T01:22:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.683+0000 7fc85bfff700 1 -- 172.21.15.27:0/1898474960 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18452 (secure 0 0 0) 0x7fc86c09c310 con 0x7fc8741082b0 2024-04-25T01:22:10.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:10.665 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":81,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:22:05.817831+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc85c0a2c20 msgr2=0x7fc85c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc85c0a2c20 0x7fc85c0a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc864005950 tx=0x7fc86400b410 comp rx=0 tx=0).stop 2024-04-25T01:22:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8741082b0 msgr2=0x7fc874101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8741082b0 0x7fc874101cb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc86c007d70 tx=0x7fc86c00a340 comp rx=0 tx=0).stop 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 shutdown_connections 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc85c0a2c20 0x7fc85c0a50e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8741082b0 0x7fc874101cb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc87406a980 0x7fc874101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 --2- 172.21.15.27:0/1898474960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc87406b320 0x7fc874101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.685+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 >> 172.21.15.27:0/1898474960 conn(0x7fc874078100 msgr2=0x7fc8740ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.686+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 shutdown_connections 2024-04-25T01:22:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:10.686+0000 7fc8795ad700 1 -- 172.21.15.27:0/1898474960 wait complete. 2024-04-25T01:22:11.092 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:10 smithi027 ceph-mon[29197]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:11.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:10 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1898474960' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:11.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:10 smithi153 ceph-mon[32621]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:11.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:10 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1898474960' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:11.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:10 smithi139 ceph-mon[32760]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:11.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:10 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1898474960' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:11.411 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:22:11.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T01:22:11.658 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:22:13.173 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:12 smithi027 ceph-mon[29197]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:12 smithi153 ceph-mon[32621]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:13.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:12 smithi139 ceph-mon[32760]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 -- 172.21.15.27:0/3921460290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806ac00 msgr2=0x7f15f806afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 --2- 172.21.15.27:0/3921460290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806ac00 0x7f15f806afe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f15f00099c0 tx=0x7f15f0009cd0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 -- 172.21.15.27:0/3921460290 shutdown_connections 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 --2- 172.21.15.27:0/3921460290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15f81075e0 0x7f15f810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 --2- 172.21.15.27:0/3921460290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806ac00 0x7f15f806afe0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 --2- 172.21.15.27:0/3921460290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15f806b5b0 0x7f15f8106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 -- 172.21.15.27:0/3921460290 >> 172.21.15.27:0/3921460290 conn(0x7f15f80771a0 msgr2=0x7f15f80775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 -- 172.21.15.27:0/3921460290 shutdown_connections 2024-04-25T01:22:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.472+0000 7f15fd880700 1 -- 172.21.15.27:0/3921460290 wait complete. 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15fd880700 1 Processor -- start 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15fd880700 1 -- start start 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15fd880700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15f806ac00 0x7f15f819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15fd880700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806b5b0 0x7f15f819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15fd880700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15f81075e0 0x7f15f8198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15fd880700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f15f81125f0 con 0x7f15f806b5b0 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15fd880700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f15f8112470 con 0x7f15f81075e0 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15fd880700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15f8112770 con 0x7f15f806ac00 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15f67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806b5b0 0x7f15f819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15f67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806b5b0 0x7f15f819ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60042/0 (socket says 172.21.15.27:60042) 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15f67fc700 1 -- 172.21.15.27:0/2935280527 learned_addr learned my addr 172.21.15.27:0/2935280527 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15f6ffd700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15f806ac00 0x7f15f819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.473+0000 7f15f77fe700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15f81075e0 0x7f15f8198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15f67fc700 1 -- 172.21.15.27:0/2935280527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15f806ac00 msgr2=0x7f15f819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15f67fc700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15f806ac00 0x7f15f819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15f67fc700 1 -- 172.21.15.27:0/2935280527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15f81075e0 msgr2=0x7f15f8198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15f67fc700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15f81075e0 0x7f15f8198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15f67fc700 1 -- 172.21.15.27:0/2935280527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f8198ff0 con 0x7f15f806b5b0 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15f67fc700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806b5b0 0x7f15f819ec10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15ec00b4d0 tx=0x7f15ec00b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15fc87e700 1 -- 172.21.15.27:0/2935280527 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ec015020 con 0x7f15f806b5b0 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f81992e0 con 0x7f15f806b5b0 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15fc87e700 1 -- 172.21.15.27:0/2935280527 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15ec011d10 con 0x7f15f806b5b0 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15fc87e700 1 -- 172.21.15.27:0/2935280527 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ec010d50 con 0x7f15f806b5b0 2024-04-25T01:22:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.474+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f81a55a0 con 0x7f15f806b5b0 2024-04-25T01:22:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.475+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f810a7b0 con 0x7f15f806b5b0 2024-04-25T01:22:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.476+0000 7f15fc87e700 1 -- 172.21.15.27:0/2935280527 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f15ec018020 con 0x7f15f806b5b0 2024-04-25T01:22:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.477+0000 7f15fc87e700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f15e00a2c20 0x7f15e00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.477+0000 7f15fc87e700 1 -- 172.21.15.27:0/2935280527 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15ec0ce780 con 0x7f15f806b5b0 2024-04-25T01:22:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.477+0000 7f15f6ffd700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f15e00a2c20 0x7f15e00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.478+0000 7f15f6ffd700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f15e00a2c20 0x7f15e00a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f15f81999d0 tx=0x7f15f0005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.481+0000 7f15fc87e700 1 -- 172.21.15.27:0/2935280527 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f15ec01f020 con 0x7f15f806b5b0 2024-04-25T01:22:13.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.684+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15f8199df0 con 0x7f15f806b5b0 2024-04-25T01:22:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.685+0000 7f15fc87e700 1 -- 172.21.15.27:0/2935280527 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f15f8199df0 con 0x7f15f806b5b0 2024-04-25T01:22:13.667 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f15e00a2c20 msgr2=0x7f15e00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f15e00a2c20 0x7f15e00a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f15f81999d0 tx=0x7f15f0005a90 comp rx=0 tx=0).stop 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806b5b0 msgr2=0x7f15f819ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806b5b0 0x7f15f819ec10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15ec00b4d0 tx=0x7f15ec00b7e0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 shutdown_connections 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f15e00a2c20 0x7f15e00a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15f81075e0 0x7f15f8198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15f806ac00 0x7f15f819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 --2- 172.21.15.27:0/2935280527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f806b5b0 0x7f15f819ec10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 >> 172.21.15.27:0/2935280527 conn(0x7f15f80771a0 msgr2=0x7f15f80ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 shutdown_connections 2024-04-25T01:22:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:13.687+0000 7f15fd880700 1 -- 172.21.15.27:0/2935280527 wait complete. 2024-04-25T01:22:13.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:13 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2935280527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:14.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:13 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2935280527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:14.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:13 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2935280527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:14.348 INFO:tasks.cephadm:Adding mds.a on smithi027 2024-04-25T01:22:14.348 INFO:tasks.cephadm:Adding mds.d on smithi027 2024-04-25T01:22:14.348 INFO:tasks.cephadm:Adding mds.g on smithi027 2024-04-25T01:22:14.348 INFO:tasks.cephadm:Adding mds.j on smithi027 2024-04-25T01:22:14.348 INFO:tasks.cephadm:Adding mds.b on smithi139 2024-04-25T01:22:14.348 INFO:tasks.cephadm:Adding mds.e on smithi139 2024-04-25T01:22:14.348 INFO:tasks.cephadm:Adding mds.h on smithi139 2024-04-25T01:22:14.349 INFO:tasks.cephadm:Adding mds.k on smithi139 2024-04-25T01:22:14.349 INFO:tasks.cephadm:Adding mds.c on smithi153 2024-04-25T01:22:14.349 INFO:tasks.cephadm:Adding mds.f on smithi153 2024-04-25T01:22:14.349 INFO:tasks.cephadm:Adding mds.i on smithi153 2024-04-25T01:22:14.349 INFO:tasks.cephadm:Adding mds.l on smithi153 2024-04-25T01:22:14.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l' 2024-04-25T01:22:14.600 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:22:14.972 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:14 smithi153 ceph-mon[32621]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:15.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:14 smithi027 ceph-mon[29197]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:15.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:14 smithi139 ceph-mon[32760]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.340+0000 7fcb47dde700 1 -- 172.21.15.153:0/1320373771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40102d30 msgr2=0x7fcb40105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.340+0000 7fcb47dde700 1 --2- 172.21.15.153:0/1320373771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40102d30 0x7fcb40105120 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009990 tx=0x7fcb3c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.340+0000 7fcb47dde700 1 -- 172.21.15.153:0/1320373771 shutdown_connections 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.340+0000 7fcb47dde700 1 --2- 172.21.15.153:0/1320373771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb40107f90 0x7fcb4010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.340+0000 7fcb47dde700 1 --2- 172.21.15.153:0/1320373771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40102d30 0x7fcb40105120 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.340+0000 7fcb47dde700 1 --2- 172.21.15.153:0/1320373771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb40105660 0x7fcb40107a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.340+0000 7fcb47dde700 1 -- 172.21.15.153:0/1320373771 >> 172.21.15.153:0/1320373771 conn(0x7fcb400fc760 msgr2=0x7fcb400feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.341+0000 7fcb47dde700 1 -- 172.21.15.153:0/1320373771 shutdown_connections 2024-04-25T01:22:16.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.341+0000 7fcb47dde700 1 -- 172.21.15.153:0/1320373771 wait complete. 2024-04-25T01:22:16.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.341+0000 7fcb47dde700 1 Processor -- start 2024-04-25T01:22:16.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.341+0000 7fcb47dde700 1 -- start start 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.341+0000 7fcb47dde700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb40102d30 0x7fcb40198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.341+0000 7fcb47dde700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb40105660 0x7fcb40198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb47dde700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40107f90 0x7fcb4019fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb47dde700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4010def0 con 0x7fcb40105660 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb47dde700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4010dd70 con 0x7fcb40102d30 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb47dde700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4010e070 con 0x7fcb40107f90 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40107f90 0x7fcb4019fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40107f90 0x7fcb4019fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51944/0 (socket says 172.21.15.153:51944) 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 -- 172.21.15.153:0/2310373533 learned_addr learned my addr 172.21.15.153:0/2310373533 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb45379700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb40105660 0x7fcb40198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb45b7a700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb40102d30 0x7fcb40198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 -- 172.21.15.153:0/2310373533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb40102d30 msgr2=0x7fcb40198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb40102d30 0x7fcb40198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 -- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb40105660 msgr2=0x7fcb40198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb40105660 0x7fcb40198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 -- 172.21.15.153:0/2310373533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb401a01c0 con 0x7fcb40107f90 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb4637b700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40107f90 0x7fcb4019fb20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb38007ee0 tx=0x7fcb3800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb36ffd700 1 -- 172.21.15.153:0/2310373533 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb38004db0 con 0x7fcb40107f90 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb36ffd700 1 -- 172.21.15.153:0/2310373533 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb38005020 con 0x7fcb40107f90 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb40100b80 con 0x7fcb40107f90 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb36ffd700 1 -- 172.21.15.153:0/2310373533 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb38015660 con 0x7fcb40107f90 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.342+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb40101140 con 0x7fcb40107f90 2024-04-25T01:22:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.343+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb40068f20 con 0x7fcb40107f90 2024-04-25T01:22:16.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.345+0000 7fcb36ffd700 1 -- 172.21.15.153:0/2310373533 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcb38005560 con 0x7fcb40107f90 2024-04-25T01:22:16.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.346+0000 7fcb36ffd700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcb2c0a2ca0 0x7fcb2c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:16.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.346+0000 7fcb36ffd700 1 -- 172.21.15.153:0/2310373533 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcb380cedc0 con 0x7fcb40107f90 2024-04-25T01:22:16.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.346+0000 7fcb45b7a700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcb2c0a2ca0 0x7fcb2c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:16.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.347+0000 7fcb45b7a700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcb2c0a2ca0 0x7fcb2c0a5160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00b5c0 tx=0x7fcb3c003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:16.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.349+0000 7fcb36ffd700 1 -- 172.21.15.153:0/2310373533 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcb38010070 con 0x7fcb40107f90 2024-04-25T01:22:16.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.553+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}) -- 0x7fcb40062830 con 0x7fcb2c0a2ca0 2024-04-25T01:22:16.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.561+0000 7fcb36ffd700 1 -- 172.21.15.153:0/2310373533 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fcb40062830 con 0x7fcb2c0a2ca0 2024-04-25T01:22:16.542 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mds.all update... 2024-04-25T01:22:16.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcb2c0a2ca0 msgr2=0x7fcb2c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcb2c0a2ca0 0x7fcb2c0a5160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00b5c0 tx=0x7fcb3c003960 comp rx=0 tx=0).stop 2024-04-25T01:22:16.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40107f90 msgr2=0x7fcb4019fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:16.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40107f90 0x7fcb4019fb20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb38007ee0 tx=0x7fcb3800d3b0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 shutdown_connections 2024-04-25T01:22:16.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcb2c0a2ca0 0x7fcb2c0a5160 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb40107f90 0x7fcb4019fb20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb40102d30 0x7fcb40198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 --2- 172.21.15.153:0/2310373533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb40105660 0x7fcb40198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:16.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.563+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 >> 172.21.15.153:0/2310373533 conn(0x7fcb400fc760 msgr2=0x7fcb401035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:16.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.564+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 shutdown_connections 2024-04-25T01:22:16.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-25T01:22:16.564+0000 7fcb47dde700 1 -- 172.21.15.153:0/2310373533 wait complete. 2024-04-25T01:22:17.146 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:16 smithi153 ceph-mon[32621]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:17.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:17.147 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:16 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:17.174 DEBUG:teuthology.orchestra.run.smithi027:mds.a> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T01:22:17.177 DEBUG:teuthology.orchestra.run.smithi027:mds.d> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T01:22:17.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:16 smithi027 ceph-mon[29197]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:17.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:17.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:16 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:17.197 DEBUG:teuthology.orchestra.run.smithi027:mds.g> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T01:22:17.240 DEBUG:teuthology.orchestra.run.smithi027:mds.j> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T01:22:17.284 DEBUG:teuthology.orchestra.run.smithi139:mds.b> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T01:22:17.286 DEBUG:teuthology.orchestra.run.smithi139:mds.e> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T01:22:17.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:16 smithi139 ceph-mon[32760]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:17.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:17.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:16 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:17.294 DEBUG:teuthology.orchestra.run.smithi139:mds.h> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T01:22:17.337 DEBUG:teuthology.orchestra.run.smithi139:mds.k> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T01:22:17.380 DEBUG:teuthology.orchestra.run.smithi153:mds.c> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T01:22:17.382 DEBUG:teuthology.orchestra.run.smithi153:mds.f> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T01:22:17.424 DEBUG:teuthology.orchestra.run.smithi153:mds.i> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T01:22:17.467 DEBUG:teuthology.orchestra.run.smithi153:mds.l> sudo journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T01:22:17.510 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T01:22:17.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:17.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 -- 172.21.15.27:0/1679429136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd581017b0 msgr2=0x7fbd58101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 --2- 172.21.15.27:0/1679429136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd581017b0 0x7fbd58101bd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009900 tx=0x7fbd4c02a410 comp rx=0 tx=0).stop 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 -- 172.21.15.27:0/1679429136 shutdown_connections 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 --2- 172.21.15.27:0/1679429136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd58103360 0x7fbd58109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 --2- 172.21.15.27:0/1679429136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd581029a0 0x7fbd58102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 --2- 172.21.15.27:0/1679429136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd581017b0 0x7fbd58101bd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 -- 172.21.15.27:0/1679429136 >> 172.21.15.27:0/1679429136 conn(0x7fbd580fcd40 msgr2=0x7fbd580ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 -- 172.21.15.27:0/1679429136 shutdown_connections 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.643+0000 7fbd5e984640 1 -- 172.21.15.27:0/1679429136 wait complete. 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5e984640 1 Processor -- start 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5e984640 1 -- start start 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5e984640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd581017b0 0x7fbd580754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5e984640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd581029a0 0x7fbd58073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5e984640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd58103360 0x7fbd58074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5e984640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5810d470 con 0x7fbd58103360 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5e984640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5810d2f0 con 0x7fbd581017b0 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5e984640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5810d5f0 con 0x7fbd581029a0 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5cf21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd58103360 0x7fbd58074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5cf21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd58103360 0x7fbd58074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60062/0 (socket says 172.21.15.27:60062) 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd5cf21640 1 -- 172.21.15.27:0/2360831280 learned_addr learned my addr 172.21.15.27:0/2360831280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd57fff640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd581017b0 0x7fbd580754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.644+0000 7fbd577fe640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd581029a0 0x7fbd58073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd5cf21640 1 -- 172.21.15.27:0/2360831280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd581029a0 msgr2=0x7fbd58073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd5cf21640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd581029a0 0x7fbd58073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd5cf21640 1 -- 172.21.15.27:0/2360831280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd581017b0 msgr2=0x7fbd580754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd5cf21640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd581017b0 0x7fbd580754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd5cf21640 1 -- 172.21.15.27:0/2360831280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd58074900 con 0x7fbd58103360 2024-04-25T01:22:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd5cf21640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd58103360 0x7fbd58074070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009fd0 tx=0x7fbd5000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd557fa640 1 -- 172.21.15.27:0/2360831280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50017070 con 0x7fbd58103360 2024-04-25T01:22:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd557fa640 1 -- 172.21.15.27:0/2360831280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd50003e50 con 0x7fbd58103360 2024-04-25T01:22:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd581a1c40 con 0x7fbd58103360 2024-04-25T01:22:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd557fa640 1 -- 172.21.15.27:0/2360831280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd500106c0 con 0x7fbd58103360 2024-04-25T01:22:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.645+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd581a2140 con 0x7fbd58103360 2024-04-25T01:22:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.646+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd58108b50 con 0x7fbd58103360 2024-04-25T01:22:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.647+0000 7fbd557fa640 1 -- 172.21.15.27:0/2360831280 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbd50003970 con 0x7fbd58103360 2024-04-25T01:22:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.648+0000 7fbd557fa640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd400a2b90 0x7fbd400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.648+0000 7fbd57fff640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd400a2b90 0x7fbd400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.648+0000 7fbd557fa640 1 -- 172.21.15.27:0/2360831280 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbd500ce800 con 0x7fbd58103360 2024-04-25T01:22:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.648+0000 7fbd57fff640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd400a2b90 0x7fbd400a5050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c02aa00 tx=0x7fbd4c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.652+0000 7fbd557fa640 1 -- 172.21.15.27:0/2360831280 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd5009b5b0 con 0x7fbd58103360 2024-04-25T01:22:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.839+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd5802d3c0 con 0x7fbd58103360 2024-04-25T01:22:17.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.840+0000 7fbd557fa640 1 -- 172.21.15.27:0/2360831280 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18452 (secure 0 0 0) 0x7fbd5009f8b0 con 0x7fbd58103360 2024-04-25T01:22:17.822 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:17.822 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":81,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:22:05.817831+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd400a2b90 msgr2=0x7fbd400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd400a2b90 0x7fbd400a5050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c02aa00 tx=0x7fbd4c0315b0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd58103360 msgr2=0x7fbd58074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd58103360 0x7fbd58074070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009fd0 tx=0x7fbd5000b260 comp rx=0 tx=0).stop 2024-04-25T01:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 shutdown_connections 2024-04-25T01:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd400a2b90 0x7fbd400a5050 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd58103360 0x7fbd58074070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd581029a0 0x7fbd58073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 --2- 172.21.15.27:0/2360831280 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd581017b0 0x7fbd580754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 >> 172.21.15.27:0/2360831280 conn(0x7fbd580fcd40 msgr2=0x7fbd58105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 shutdown_connections 2024-04-25T01:22:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.842+0000 7fbd5e984640 1 -- 172.21.15.27:0/2360831280 wait complete. 2024-04-25T01:22:17.836 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:22:17.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:17.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 -- 172.21.15.27:0/153125826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc103350 msgr2=0x7f54cc109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 --2- 172.21.15.27:0/153125826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc103350 0x7f54cc109be0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009ca0 tx=0x7f54c402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 -- 172.21.15.27:0/153125826 shutdown_connections 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 --2- 172.21.15.27:0/153125826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc103350 0x7f54cc109be0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 --2- 172.21.15.27:0/153125826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54cc102990 0x7f54cc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 --2- 172.21.15.27:0/153125826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54cc1017a0 0x7f54cc101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 -- 172.21.15.27:0/153125826 >> 172.21.15.27:0/153125826 conn(0x7f54cc0fcd10 msgr2=0x7f54cc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 -- 172.21.15.27:0/153125826 shutdown_connections 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.960+0000 7f54d2396640 1 -- 172.21.15.27:0/153125826 wait complete. 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d2396640 1 Processor -- start 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d2396640 1 -- start start 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d2396640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54cc1017a0 0x7f54cc0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d2396640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc102990 0x7f54cc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d2396640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54cc103350 0x7f54cc074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d2396640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc10d420 con 0x7f54cc102990 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d2396640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc10d2a0 con 0x7f54cc1017a0 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d2396640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc10d5a0 con 0x7f54cc103350 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54cb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc102990 0x7f54cc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54cb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc102990 0x7f54cc073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45482/0 (socket says 172.21.15.27:45482) 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54cb7fe640 1 -- 172.21.15.27:0/1245325678 learned_addr learned my addr 172.21.15.27:0/1245325678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54d0933640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54cc103350 0x7f54cc074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.961+0000 7f54cbfff640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54cc1017a0 0x7f54cc0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54cb7fe640 1 -- 172.21.15.27:0/1245325678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54cc103350 msgr2=0x7f54cc074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54cb7fe640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54cc103350 0x7f54cc074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54cb7fe640 1 -- 172.21.15.27:0/1245325678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54cc1017a0 msgr2=0x7f54cc0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54cb7fe640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54cc1017a0 0x7f54cc0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54cb7fe640 1 -- 172.21.15.27:0/1245325678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54cc074900 con 0x7f54cc102990 2024-04-25T01:22:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54cbfff640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54cc1017a0 0x7f54cc0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54cb7fe640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc102990 0x7f54cc073b30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00d770 tx=0x7f54bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54c97fa640 1 -- 172.21.15.27:0/1245325678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc017070 con 0x7f54cc102990 2024-04-25T01:22:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54cc1a1b60 con 0x7f54cc102990 2024-04-25T01:22:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54c97fa640 1 -- 172.21.15.27:0/1245325678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc005500 con 0x7f54cc102990 2024-04-25T01:22:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54c97fa640 1 -- 172.21.15.27:0/1245325678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc004e00 con 0x7f54cc102990 2024-04-25T01:22:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.962+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54cc1a20b0 con 0x7f54cc102990 2024-04-25T01:22:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.963+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54cc108ac0 con 0x7f54cc102990 2024-04-25T01:22:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.964+0000 7f54c97fa640 1 -- 172.21.15.27:0/1245325678 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f54bc005000 con 0x7f54cc102990 2024-04-25T01:22:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.964+0000 7f54c97fa640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f54b40a2b90 0x7f54b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.965+0000 7f54cbfff640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f54b40a2b90 0x7f54b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.965+0000 7f54c97fa640 1 -- 172.21.15.27:0/1245325678 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f54bc0ce610 con 0x7f54cc102990 2024-04-25T01:22:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.965+0000 7f54cbfff640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f54b40a2b90 0x7f54b40a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f54cc1027f0 tx=0x7f54c0005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:17.969+0000 7f54c97fa640 1 -- 172.21.15.27:0/1245325678 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f54bc09ba10 con 0x7f54cc102990 2024-04-25T01:22:18.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:18.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-25T01:22:18.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:18.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:17 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2360831280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:18.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.154+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f54cc02d340 con 0x7f54cc102990 2024-04-25T01:22:18.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.155+0000 7f54c97fa640 1 -- 172.21.15.27:0/1245325678 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f54bc09b310 con 0x7f54cc102990 2024-04-25T01:22:18.139 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:22:18.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f54b40a2b90 msgr2=0x7f54b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f54b40a2b90 0x7f54b40a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f54cc1027f0 tx=0x7f54c0005d30 comp rx=0 tx=0).stop 2024-04-25T01:22:18.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc102990 msgr2=0x7f54cc073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc102990 0x7f54cc073b30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00d770 tx=0x7f54bc00dc30 comp rx=0 tx=0).stop 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 shutdown_connections 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f54b40a2b90 0x7f54b40a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54cc103350 0x7f54cc074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54cc102990 0x7f54cc073b30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 --2- 172.21.15.27:0/1245325678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54cc1017a0 0x7f54cc0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 >> 172.21.15.27:0/1245325678 conn(0x7f54cc0fcd10 msgr2=0x7f54cc105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 shutdown_connections 2024-04-25T01:22:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.157+0000 7f54d2396640 1 -- 172.21.15.27:0/1245325678 wait complete. 2024-04-25T01:22:18.148 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:18.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-25T01:22:18.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:18.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:18.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:18.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:18.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:18.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:18.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:17 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2360831280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:18.150 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:18.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 -- 172.21.15.27:0/1877978533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 msgr2=0x7f8538100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 --2- 172.21.15.27:0/1877978533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 0x7f8538100ac0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8528009ca0 tx=0x7f852802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 -- 172.21.15.27:0/1877978533 shutdown_connections 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 --2- 172.21.15.27:0/1877978533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8538102250 0x7f8538108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 --2- 172.21.15.27:0/1877978533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8538101890 0x7f8538101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 --2- 172.21.15.27:0/1877978533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 0x7f8538100ac0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 -- 172.21.15.27:0/1877978533 >> 172.21.15.27:0/1877978533 conn(0x7f85380fbc30 msgr2=0x7f85380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 -- 172.21.15.27:0/1877978533 shutdown_connections 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 -- 172.21.15.27:0/1877978533 wait complete. 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.270+0000 7f853d088640 1 Processor -- start 2024-04-25T01:22:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853d088640 1 -- start start 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853d088640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 0x7f8538196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853d088640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8538101890 0x7f8538196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853d088640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8538102250 0x7f853819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853d088640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f853810c360 con 0x7f85381006a0 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853d088640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f853810c1e0 con 0x7f8538102250 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853d088640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f853810c4e0 con 0x7f8538101890 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 0x7f8538196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 0x7f8538196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45500/0 (socket says 172.21.15.27:45500) 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 -- 172.21.15.27:0/4228800982 learned_addr learned my addr 172.21.15.27:0/4228800982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853759e640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8538102250 0x7f853819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853659c640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8538101890 0x7f8538196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 -- 172.21.15.27:0/4228800982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8538101890 msgr2=0x7f8538196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8538101890 0x7f8538196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 -- 172.21.15.27:0/4228800982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8538102250 msgr2=0x7f853819dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8538102250 0x7f853819dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 -- 172.21.15.27:0/4228800982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853819e650 con 0x7f85381006a0 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f853759e640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8538102250 0x7f853819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f8536d9d640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 0x7f8538196a30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8528000f80 tx=0x7f85280319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f851ffff640 1 -- 172.21.15.27:0/4228800982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528003c70 con 0x7f85381006a0 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.271+0000 7f851ffff640 1 -- 172.21.15.27:0/4228800982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8528003e10 con 0x7f85381006a0 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.272+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853819e8e0 con 0x7f85381006a0 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.272+0000 7f851ffff640 1 -- 172.21.15.27:0/4228800982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852802fda0 con 0x7f85381006a0 2024-04-25T01:22:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.272+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853819ee30 con 0x7f85381006a0 2024-04-25T01:22:18.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.272+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8538107a40 con 0x7f85381006a0 2024-04-25T01:22:18.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.274+0000 7f851ffff640 1 -- 172.21.15.27:0/4228800982 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8528004040 con 0x7f85381006a0 2024-04-25T01:22:18.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.274+0000 7f851ffff640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85200a2b90 0x7f85200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:18.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.274+0000 7f853659c640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85200a2b90 0x7f85200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:18.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.274+0000 7f851ffff640 1 -- 172.21.15.27:0/4228800982 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f85280f01d0 con 0x7f85381006a0 2024-04-25T01:22:18.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.275+0000 7f853659c640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85200a2b90 0x7f85200a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f852c005fd0 tx=0x7f852c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:18.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.279+0000 7f851ffff640 1 -- 172.21.15.27:0/4228800982 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f85280bd660 con 0x7f85381006a0 2024-04-25T01:22:18.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:18.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-25T01:22:18.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:18.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:18.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:18.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:18.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:18.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:18.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:17 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2360831280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:18.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.501+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f853802d3e0 con 0x7f85381006a0 2024-04-25T01:22:18.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.502+0000 7f851ffff640 1 -- 172.21.15.27:0/4228800982 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f85280bcf60 con 0x7f85381006a0 2024-04-25T01:22:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:18.484 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T01:22:18.484 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 1 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85200a2b90 msgr2=0x7f85200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85200a2b90 0x7f85200a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f852c005fd0 tx=0x7f852c005850 comp rx=0 tx=0).stop 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 msgr2=0x7f8538196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 0x7f8538196a30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8528000f80 tx=0x7f85280319f0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 shutdown_connections 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85200a2b90 0x7f85200a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8538102250 0x7f853819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8538101890 0x7f8538196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 --2- 172.21.15.27:0/4228800982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85381006a0 0x7f8538196a30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 >> 172.21.15.27:0/4228800982 conn(0x7f85380fbc30 msgr2=0x7f8538104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:18.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 shutdown_connections 2024-04-25T01:22:18.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:18.505+0000 7f853d088640 1 -- 172.21.15.27:0/4228800982 wait complete. 2024-04-25T01:22:19.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:18 smithi153 ceph-mon[32621]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:19.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:18 smithi153 ceph-mon[32621]: Deploying daemon mds.l on smithi153 2024-04-25T01:22:19.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:18 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1245325678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:19.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:18 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4228800982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:19.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:18 smithi027 ceph-mon[29197]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:19.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:18 smithi027 ceph-mon[29197]: Deploying daemon mds.l on smithi153 2024-04-25T01:22:19.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:18 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1245325678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:19.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:18 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4228800982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:19.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:18 smithi139 ceph-mon[32760]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:19.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:18 smithi139 ceph-mon[32760]: Deploying daemon mds.l on smithi153 2024-04-25T01:22:19.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:18 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1245325678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:19.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:18 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4228800982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:19.741 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:19 smithi153 systemd[1]: Starting Ceph mds.l for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:20.003 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:19 smithi153 podman[63063]: 2024-04-25 01:22:19.663096693 +0000 UTC m=+0.018909927 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:20.003 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:19 smithi153 podman[63063]: 2024-04-25 01:22:19.776377993 +0000 UTC m=+0.132191255 container create 5ec7d31dad34259e92f4362d6d3489039f83cb762b7d226e2ca9dcc7e0ebd2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T01:22:20.003 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:19 smithi153 podman[63063]: 2024-04-25 01:22:19.995992546 +0000 UTC m=+0.351805766 container init 5ec7d31dad34259e92f4362d6d3489039f83cb762b7d226e2ca9dcc7e0ebd2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T01:22:20.003 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:19 smithi153 podman[63063]: 2024-04-25 01:22:19.998567502 +0000 UTC m=+0.354380731 container start 5ec7d31dad34259e92f4362d6d3489039f83cb762b7d226e2ca9dcc7e0ebd2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9) 2024-04-25T01:22:20.003 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:20 smithi153 ceph-mds[63082]: -- 172.21.15.153:0/3499643218 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557fd3595a40 con 0x557fd3568800 2024-04-25T01:22:20.277 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l[63078]: starting mds.l at 2024-04-25T01:22:20.277 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:20 smithi153 bash[63063]: 5ec7d31dad34259e92f4362d6d3489039f83cb762b7d226e2ca9dcc7e0ebd2d9 2024-04-25T01:22:20.277 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:22:20 smithi153 systemd[1]: Started Ceph mds.l for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:20.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 -- 172.21.15.27:0/996237276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680102200 msgr2=0x7f86801045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 --2- 172.21.15.27:0/996237276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680102200 0x7f86801045f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f867c009900 tx=0x7f867c02a410 comp rx=0 tx=0).stop 2024-04-25T01:22:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 -- 172.21.15.27:0/996237276 shutdown_connections 2024-04-25T01:22:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 --2- 172.21.15.27:0/996237276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8680107460 0x7f86801098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 --2- 172.21.15.27:0/996237276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8680104b30 0x7f8680106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 --2- 172.21.15.27:0/996237276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680102200 0x7f86801045f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 -- 172.21.15.27:0/996237276 >> 172.21.15.27:0/996237276 conn(0x7f86800fbbf0 msgr2=0x7f86800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 -- 172.21.15.27:0/996237276 shutdown_connections 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.637+0000 7f86882da640 1 -- 172.21.15.27:0/996237276 wait complete. 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f86882da640 1 Processor -- start 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f86882da640 1 -- start start 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f86882da640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8680102200 0x7f8680101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f86882da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8680104b30 0x7f86801001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f86882da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680107460 0x7f86801006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f86882da640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f868010d430 con 0x7f8680107460 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f86882da640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f868010d2b0 con 0x7f8680102200 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f86882da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f868010d5b0 con 0x7f8680104b30 2024-04-25T01:22:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680107460 0x7f86801006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680107460 0x7f86801006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45522/0 (socket says 172.21.15.27:45522) 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 -- 172.21.15.27:0/29790722 learned_addr learned my addr 172.21.15.27:0/29790722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686076640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8680102200 0x7f8680101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8685875640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8680104b30 0x7f86801001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 -- 172.21.15.27:0/29790722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8680104b30 msgr2=0x7f86801001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8680104b30 0x7f86801001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 -- 172.21.15.27:0/29790722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8680102200 msgr2=0x7f8680101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8680102200 0x7f8680101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 -- 172.21.15.27:0/29790722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8680100f80 con 0x7f8680107460 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8685875640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8680104b30 0x7f86801001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.638+0000 7f8686877640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680107460 0x7f86801006f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f867800eb10 tx=0x7f867800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.639+0000 7f86777fe640 1 -- 172.21.15.27:0/29790722 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678004b20 con 0x7f8680107460 2024-04-25T01:22:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.639+0000 7f86777fe640 1 -- 172.21.15.27:0/29790722 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8678004cc0 con 0x7f8680107460 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.639+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86801a1c30 con 0x7f8680107460 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.639+0000 7f86777fe640 1 -- 172.21.15.27:0/29790722 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678010760 con 0x7f8680107460 2024-04-25T01:22:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.639+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86801a2130 con 0x7f8680107460 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.640+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86800fd6e0 con 0x7f8680107460 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.641+0000 7f86777fe640 1 -- 172.21.15.27:0/29790722 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8678005020 con 0x7f8680107460 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.641+0000 7f86777fe640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f866c0a2b90 0x7f866c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.642+0000 7f8686076640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f866c0a2b90 0x7f866c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.642+0000 7f86777fe640 1 -- 172.21.15.27:0/29790722 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f86780cf240 con 0x7f8680107460 2024-04-25T01:22:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.642+0000 7f8686076640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f866c0a2b90 0x7f866c0a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f867c0314e0 tx=0x7f867c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:20.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.646+0000 7f86777fe640 1 -- 172.21.15.27:0/29790722 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f867809c640 con 0x7f8680107460 2024-04-25T01:22:20.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.869+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f868002d3c0 con 0x7f8680107460 2024-04-25T01:22:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.870+0000 7f86777fe640 1 -- 172.21.15.27:0/29790722 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f867809bf40 con 0x7f8680107460 2024-04-25T01:22:20.852 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:20.852 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:22:20.852 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 2 2024-04-25T01:22:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f866c0a2b90 msgr2=0x7f866c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f866c0a2b90 0x7f866c0a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f867c0314e0 tx=0x7f867c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680107460 msgr2=0x7f86801006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680107460 0x7f86801006f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f867800eb10 tx=0x7f867800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 shutdown_connections 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f866c0a2b90 0x7f866c0a5050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8680107460 0x7f86801006f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8680104b30 0x7f86801001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 --2- 172.21.15.27:0/29790722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8680102200 0x7f8680101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 >> 172.21.15.27:0/29790722 conn(0x7f86800fbbf0 msgr2=0x7f8680102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 shutdown_connections 2024-04-25T01:22:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:20.872+0000 7f86882da640 1 -- 172.21.15.27:0/29790722 wait complete. 2024-04-25T01:22:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6832/3706725137,v1:172.21.15.153:6833/3706725137] up:boot 2024-04-25T01:22:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: fsmap 1 up:standby 2024-04-25T01:22:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:22:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:21.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:21.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:21.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: Deploying daemon mds.k on smithi139 2024-04-25T01:22:21.045 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:21 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/29790722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6832/3706725137,v1:172.21.15.153:6833/3706725137] up:boot 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: fsmap 1 up:standby 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:21.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: Deploying daemon mds.k on smithi139 2024-04-25T01:22:21.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:21 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/29790722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6832/3706725137,v1:172.21.15.153:6833/3706725137] up:boot 2024-04-25T01:22:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: fsmap 1 up:standby 2024-04-25T01:22:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:22:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:21.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:21.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:21.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:21.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: Deploying daemon mds.k on smithi139 2024-04-25T01:22:21.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:21 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/29790722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:22.543 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:22 smithi139 systemd[1]: Starting Ceph mds.k for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:22.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:22.939 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:22 smithi139 podman[62151]: 2024-04-25 01:22:22.5789585 +0000 UTC m=+0.018357486 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:22.939 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:22 smithi139 podman[62151]: 2024-04-25 01:22:22.697488128 +0000 UTC m=+0.136887099 container create 7b6f4eeef614a87f2c5a1a82db7318877cdd89a3f418d6ff7ccd07809358e8b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T01:22:22.939 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:22 smithi139 podman[62151]: 2024-04-25 01:22:22.958257292 +0000 UTC m=+0.397656264 container init 7b6f4eeef614a87f2c5a1a82db7318877cdd89a3f418d6ff7ccd07809358e8b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:22:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 -- 172.21.15.27:0/2824966188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988101310 msgr2=0x7f2988101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 --2- 172.21.15.27:0/2824966188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988101310 0x7f2988101790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2980009ca0 tx=0x7f298002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 -- 172.21.15.27:0/2824966188 shutdown_connections 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 --2- 172.21.15.27:0/2824966188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988101310 0x7f2988101790 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 --2- 172.21.15.27:0/2824966188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2988100970 0x7f2988100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 --2- 172.21.15.27:0/2824966188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2988100010 0x7f2988100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 -- 172.21.15.27:0/2824966188 >> 172.21.15.27:0/2824966188 conn(0x7f29880fbc30 msgr2=0x7f29880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 -- 172.21.15.27:0/2824966188 shutdown_connections 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.000+0000 7f298fbc8640 1 -- 172.21.15.27:0/2824966188 wait complete. 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298fbc8640 1 Processor -- start 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298fbc8640 1 -- start start 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298fbc8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988100010 0x7f2988199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298fbc8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2988100970 0x7f298819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298fbc8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2988101310 0x7f29881a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298fbc8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f298810f610 con 0x7f2988100010 2024-04-25T01:22:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298fbc8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f298810f490 con 0x7f2988100970 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298fbc8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f298810f790 con 0x7f2988101310 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298d964640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988100010 0x7f2988199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298d964640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988100010 0x7f2988199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45542/0 (socket says 172.21.15.27:45542) 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298d964640 1 -- 172.21.15.27:0/349189796 learned_addr learned my addr 172.21.15.27:0/349189796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298e165640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2988101310 0x7f29881a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.001+0000 7f298d163640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2988100970 0x7f298819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f298d964640 1 -- 172.21.15.27:0/349189796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2988101310 msgr2=0x7f29881a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f298d964640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2988101310 0x7f29881a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f298d964640 1 -- 172.21.15.27:0/349189796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2988100970 msgr2=0x7f298819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f298d964640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2988100970 0x7f298819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f298d964640 1 -- 172.21.15.27:0/349189796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29881a1810 con 0x7f2988100010 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f298d964640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988100010 0x7f2988199bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f298400ea60 tx=0x7f298400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f297effd640 1 -- 172.21.15.27:0/349189796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298400cde0 con 0x7f2988100010 2024-04-25T01:22:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f297effd640 1 -- 172.21.15.27:0/349189796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29840035f0 con 0x7f2988100010 2024-04-25T01:22:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29881a1b00 con 0x7f2988100010 2024-04-25T01:22:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f297effd640 1 -- 172.21.15.27:0/349189796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984010670 con 0x7f2988100010 2024-04-25T01:22:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.002+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29881a20a0 con 0x7f2988100010 2024-04-25T01:22:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.003+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f298810ad60 con 0x7f2988100010 2024-04-25T01:22:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.004+0000 7f297effd640 1 -- 172.21.15.27:0/349189796 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f29840037a0 con 0x7f2988100010 2024-04-25T01:22:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.005+0000 7f297effd640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f29740a2c10 0x7f29740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.005+0000 7f298d163640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f29740a2c10 0x7f29740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.005+0000 7f297effd640 1 -- 172.21.15.27:0/349189796 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29840cf010 con 0x7f2988100010 2024-04-25T01:22:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.005+0000 7f298d163640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f29740a2c10 0x7f29740a50d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2978005fd0 tx=0x7f2978005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.008+0000 7f297effd640 1 -- 172.21.15.27:0/349189796 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f298409c4a0 con 0x7f2988100010 2024-04-25T01:22:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:23 smithi027 ceph-mon[29197]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:23.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.251+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f298802d170 con 0x7f2988100010 2024-04-25T01:22:23.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.251+0000 7f297effd640 1 -- 172.21.15.27:0/349189796 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f298409bda0 con 0x7f2988100010 2024-04-25T01:22:23.232 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:23.232 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:22:23.232 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-04-25T01:22:23.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.253+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f29740a2c10 msgr2=0x7f29740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:23.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.253+0000 7f298fbc8640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f29740a2c10 0x7f29740a50d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2978005fd0 tx=0x7f2978005850 comp rx=0 tx=0).stop 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988100010 msgr2=0x7f2988199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988100010 0x7f2988199bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f298400ea60 tx=0x7f298400ef20 comp rx=0 tx=0).stop 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 shutdown_connections 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f29740a2c10 0x7f29740a50d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2988101310 0x7f29881a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2988100970 0x7f298819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 --2- 172.21.15.27:0/349189796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2988100010 0x7f2988199bf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 >> 172.21.15.27:0/349189796 conn(0x7f29880fbc30 msgr2=0x7f29880fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 shutdown_connections 2024-04-25T01:22:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:23.254+0000 7f298fbc8640 1 -- 172.21.15.27:0/349189796 wait complete. 2024-04-25T01:22:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:23 smithi153 ceph-mon[32621]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:23.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:23 smithi139 ceph-mon[32760]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:23.293 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:22 smithi139 podman[62151]: 2024-04-25 01:22:22.961309352 +0000 UTC m=+0.400708333 container start 7b6f4eeef614a87f2c5a1a82db7318877cdd89a3f418d6ff7ccd07809358e8b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T01:22:23.293 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:22 smithi139 ceph-mds[62170]: -- 172.21.15.139:0/1654604427 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55adec8c7a40 con 0x55adec89ac00 2024-04-25T01:22:23.294 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k[62166]: starting mds.k at 2024-04-25T01:22:23.294 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:23 smithi139 bash[62151]: 7b6f4eeef614a87f2c5a1a82db7318877cdd89a3f418d6ff7ccd07809358e8b0 2024-04-25T01:22:23.294 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:22:23 smithi139 systemd[1]: Started Ceph mds.k for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.139:6832/4036036788,v1:172.21.15.139:6833/4036036788] up:boot 2024-04-25T01:22:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: fsmap 2 up:standby 2024-04-25T01:22:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:22:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:24.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:24.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: Deploying daemon mds.j on smithi027 2024-04-25T01:22:24.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:24 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/349189796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:24.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.139:6832/4036036788,v1:172.21.15.139:6833/4036036788] up:boot 2024-04-25T01:22:24.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: fsmap 2 up:standby 2024-04-25T01:22:24.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:22:24.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:24.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:24.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:24.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: Deploying daemon mds.j on smithi027 2024-04-25T01:22:24.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:24 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/349189796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.139:6832/4036036788,v1:172.21.15.139:6833/4036036788] up:boot 2024-04-25T01:22:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: fsmap 2 up:standby 2024-04-25T01:22:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:22:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:24.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:24.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: Deploying daemon mds.j on smithi027 2024-04-25T01:22:24.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:24 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/349189796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:25.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:25.308 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:25 smithi027 ceph-mon[29197]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:25.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.416+0000 7fb553d90640 1 -- 172.21.15.27:0/1651460375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 msgr2=0x7fb54c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.416+0000 7fb553d90640 1 --2- 172.21.15.27:0/1651460375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 0x7fb54c108ac0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb544009ca0 tx=0x7fb54402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.416+0000 7fb553d90640 1 -- 172.21.15.27:0/1651460375 shutdown_connections 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.416+0000 7fb553d90640 1 --2- 172.21.15.27:0/1651460375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 0x7fb54c108ac0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.416+0000 7fb553d90640 1 --2- 172.21.15.27:0/1651460375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb54c101870 0x7fb54c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.416+0000 7fb553d90640 1 --2- 172.21.15.27:0/1651460375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb54c100680 0x7fb54c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.416+0000 7fb553d90640 1 -- 172.21.15.27:0/1651460375 >> 172.21.15.27:0/1651460375 conn(0x7fb54c0fbbf0 msgr2=0x7fb54c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 -- 172.21.15.27:0/1651460375 shutdown_connections 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 -- 172.21.15.27:0/1651460375 wait complete. 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 Processor -- start 2024-04-25T01:22:25.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 -- start start 2024-04-25T01:22:25.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb54c100680 0x7fb54c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb54c101870 0x7fb54c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 0x7fb54c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c10c300 con 0x7fb54c102230 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c10c180 con 0x7fb54c101870 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb553d90640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c10c480 con 0x7fb54c100680 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.417+0000 7fb55232d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 0x7fb54c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55232d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 0x7fb54c19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45550/0 (socket says 172.21.15.27:45550) 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55232d640 1 -- 172.21.15.27:0/3076370942 learned_addr learned my addr 172.21.15.27:0/3076370942 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55132b640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb54c101870 0x7fb54c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb551b2c640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb54c100680 0x7fb54c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55232d640 1 -- 172.21.15.27:0/3076370942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb54c100680 msgr2=0x7fb54c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55232d640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb54c100680 0x7fb54c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55232d640 1 -- 172.21.15.27:0/3076370942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb54c101870 msgr2=0x7fb54c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55232d640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb54c101870 0x7fb54c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55232d640 1 -- 172.21.15.27:0/3076370942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c19f660 con 0x7fb54c102230 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb551b2c640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb54c100680 0x7fb54c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55132b640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb54c101870 0x7fb54c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb55232d640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 0x7fb54c19eff0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb544005e30 tx=0x7fb544031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb542ffd640 1 -- 172.21.15.27:0/3076370942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544003c30 con 0x7fb54c102230 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54c19f8f0 con 0x7fb54c102230 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb542ffd640 1 -- 172.21.15.27:0/3076370942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb544003dd0 con 0x7fb54c102230 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54c19fe90 con 0x7fb54c102230 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.418+0000 7fb542ffd640 1 -- 172.21.15.27:0/3076370942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54402fde0 con 0x7fb54c102230 2024-04-25T01:22:25.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.419+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54c1a0180 con 0x7fb54c102230 2024-04-25T01:22:25.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.421+0000 7fb542ffd640 1 -- 172.21.15.27:0/3076370942 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb544004050 con 0x7fb54c102230 2024-04-25T01:22:25.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.421+0000 7fb542ffd640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb5380a2b90 0x7fb5380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:25.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.422+0000 7fb551b2c640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb5380a2b90 0x7fb5380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:25.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.422+0000 7fb542ffd640 1 -- 172.21.15.27:0/3076370942 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb5440f0090 con 0x7fb54c102230 2024-04-25T01:22:25.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.422+0000 7fb551b2c640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb5380a2b90 0x7fb5380a5050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb548005d00 tx=0x7fb548005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:25.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.425+0000 7fb542ffd640 1 -- 172.21.15.27:0/3076370942 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb5440bd590 con 0x7fb54c102230 2024-04-25T01:22:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:25 smithi153 ceph-mon[32621]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:25 smithi139 ceph-mon[32760]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:25.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.662+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb54c1a0350 con 0x7fb54c102230 2024-04-25T01:22:25.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.663+0000 7fb542ffd640 1 -- 172.21.15.27:0/3076370942 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fb5440bce90 con 0x7fb54c102230 2024-04-25T01:22:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:25.643 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:22:25.644 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-04-25T01:22:25.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb5380a2b90 msgr2=0x7fb5380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb5380a2b90 0x7fb5380a5050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb548005d00 tx=0x7fb548005c90 comp rx=0 tx=0).stop 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 msgr2=0x7fb54c19eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 0x7fb54c19eff0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb544005e30 tx=0x7fb544031ee0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 shutdown_connections 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb5380a2b90 0x7fb5380a5050 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c102230 0x7fb54c19eff0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb54c101870 0x7fb54c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 --2- 172.21.15.27:0/3076370942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb54c100680 0x7fb54c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 >> 172.21.15.27:0/3076370942 conn(0x7fb54c0fbbf0 msgr2=0x7fb54c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 shutdown_connections 2024-04-25T01:22:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:25.665+0000 7fb553d90640 1 -- 172.21.15.27:0/3076370942 wait complete. 2024-04-25T01:22:25.658 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:25 smithi027 systemd[1]: Starting Ceph mds.j for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:25.917 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:25 smithi027 podman[69597]: 2024-04-25 01:22:25.6830075 +0000 UTC m=+0.020683320 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:25.917 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:25 smithi027 podman[69597]: 2024-04-25 01:22:25.814160659 +0000 UTC m=+0.151836450 container create 7a740179891c491d8603db77ffe031c065a1ec74bc8574d81f5113f3d1a74118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:26 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3076370942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:26 smithi027 podman[69597]: 2024-04-25 01:22:26.050432663 +0000 UTC m=+0.388108456 container init 7a740179891c491d8603db77ffe031c065a1ec74bc8574d81f5113f3d1a74118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:26 smithi027 podman[69597]: 2024-04-25 01:22:26.05320014 +0000 UTC m=+0.390875933 container start 7a740179891c491d8603db77ffe031c065a1ec74bc8574d81f5113f3d1a74118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:26 smithi027 ceph-mds[69618]: -- 172.21.15.27:0/3918141536 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5643c24d9a40 con 0x5643c24acc00 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j[69614]: starting mds.j at 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:26 smithi027 bash[69597]: 7a740179891c491d8603db77ffe031c065a1ec74bc8574d81f5113f3d1a74118 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:22:26 smithi027 systemd[1]: Started Ceph mds.j for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:26 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3076370942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:26.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:26 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3076370942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.27:6834/2965551549,v1:172.21.15.27:6835/2965551549] up:boot 2024-04-25T01:22:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: fsmap 3 up:standby 2024-04-25T01:22:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:22:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:27 smithi027 ceph-mon[29197]: Deploying daemon mds.d on smithi027 2024-04-25T01:22:27.429 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:27.429 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.27:6834/2965551549,v1:172.21.15.27:6835/2965551549] up:boot 2024-04-25T01:22:27.429 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: fsmap 3 up:standby 2024-04-25T01:22:27.429 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:22:27.429 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.429 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.429 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.429 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:27.430 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:27.430 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:27.430 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:27 smithi153 ceph-mon[32621]: Deploying daemon mds.d on smithi027 2024-04-25T01:22:27.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:27.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.27:6834/2965551549,v1:172.21.15.27:6835/2965551549] up:boot 2024-04-25T01:22:27.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: fsmap 3 up:standby 2024-04-25T01:22:27.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:22:27.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:27.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:27.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:27.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:27.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:27 smithi139 ceph-mon[32760]: Deploying daemon mds.d on smithi027 2024-04-25T01:22:27.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.791+0000 7f6d3ceda640 1 -- 172.21.15.27:0/474978998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d38101890 msgr2=0x7f6d38101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.791+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/474978998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d38101890 0x7f6d38101d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009900 tx=0x7f6d2002a410 comp rx=0 tx=0).stop 2024-04-25T01:22:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.791+0000 7f6d3ceda640 1 -- 172.21.15.27:0/474978998 shutdown_connections 2024-04-25T01:22:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.791+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/474978998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d38102250 0x7f6d38108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.791+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/474978998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d38101890 0x7f6d38101d10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.791+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/474978998 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d381006a0 0x7f6d38100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.791+0000 7f6d3ceda640 1 -- 172.21.15.27:0/474978998 >> 172.21.15.27:0/474978998 conn(0x7f6d380fbc30 msgr2=0x7f6d380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.791+0000 7f6d3ceda640 1 -- 172.21.15.27:0/474978998 shutdown_connections 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 -- 172.21.15.27:0/474978998 wait complete. 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 Processor -- start 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 -- start start 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d381006a0 0x7f6d38199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d38101890 0x7f6d3819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d38102250 0x7f6d381a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3810d440 con 0x7f6d381006a0 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3810d2c0 con 0x7f6d38101890 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.792+0000 7f6d3ceda640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3810d5c0 con 0x7f6d38102250 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d381006a0 0x7f6d38199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d381006a0 0x7f6d38199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41692/0 (socket says 172.21.15.27:41692) 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 -- 172.21.15.27:0/614249291 learned_addr learned my addr 172.21.15.27:0/614249291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d36d9d640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d38102250 0x7f6d381a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d2ffff640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d38101890 0x7f6d3819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 -- 172.21.15.27:0/614249291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d38102250 msgr2=0x7f6d381a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d38102250 0x7f6d381a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 -- 172.21.15.27:0/614249291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d38101890 msgr2=0x7f6d3819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d38101890 0x7f6d3819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 -- 172.21.15.27:0/614249291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d381a1860 con 0x7f6d381006a0 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3659c640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d381006a0 0x7f6d38199c50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c970 tx=0x7f6d2800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d2f7fe640 1 -- 172.21.15.27:0/614249291 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28007c10 con 0x7f6d381006a0 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d2f7fe640 1 -- 172.21.15.27:0/614249291 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d28007db0 con 0x7f6d381006a0 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d381a1b50 con 0x7f6d381006a0 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d2f7fe640 1 -- 172.21.15.27:0/614249291 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800f940 con 0x7f6d381006a0 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.793+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d381a20a0 con 0x7f6d381006a0 2024-04-25T01:22:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.794+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d381a2270 con 0x7f6d381006a0 2024-04-25T01:22:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.795+0000 7f6d2f7fe640 1 -- 172.21.15.27:0/614249291 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6d2800fae0 con 0x7f6d381006a0 2024-04-25T01:22:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.796+0000 7f6d2f7fe640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d240a2b90 0x7f6d240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.796+0000 7f6d2ffff640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d240a2b90 0x7f6d240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.796+0000 7f6d2f7fe640 1 -- 172.21.15.27:0/614249291 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d280cf020 con 0x7f6d381006a0 2024-04-25T01:22:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.796+0000 7f6d2ffff640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d240a2b90 0x7f6d240a5050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6d20000c00 tx=0x7f6d200315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:27.800+0000 7f6d2f7fe640 1 -- 172.21.15.27:0/614249291 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d2809c420 con 0x7f6d381006a0 2024-04-25T01:22:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.026+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d381a2440 con 0x7f6d381006a0 2024-04-25T01:22:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.026+0000 7f6d2f7fe640 1 -- 172.21.15.27:0/614249291 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f6d2809bd20 con 0x7f6d381006a0 2024-04-25T01:22:28.007 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:28.007 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:22:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 4 2024-04-25T01:22:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d240a2b90 msgr2=0x7f6d240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d240a2b90 0x7f6d240a5050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6d20000c00 tx=0x7f6d200315d0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d381006a0 msgr2=0x7f6d38199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d381006a0 0x7f6d38199c50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c970 tx=0x7f6d2800ce30 comp rx=0 tx=0).stop 2024-04-25T01:22:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 shutdown_connections 2024-04-25T01:22:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d240a2b90 0x7f6d240a5050 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d38102250 0x7f6d381a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d38101890 0x7f6d3819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 --2- 172.21.15.27:0/614249291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d381006a0 0x7f6d38199c50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 >> 172.21.15.27:0/614249291 conn(0x7f6d380fbc30 msgr2=0x7f6d381094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 shutdown_connections 2024-04-25T01:22:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:28.028+0000 7f6d3ceda640 1 -- 172.21.15.27:0/614249291 wait complete. 2024-04-25T01:22:28.311 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:28 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/614249291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:28.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:28 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/614249291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:28.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:28 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/614249291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:28.697 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:28 smithi027 systemd[1]: Starting Ceph mds.d for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:28.978 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:28 smithi027 podman[69950]: 2024-04-25 01:22:28.647129608 +0000 UTC m=+0.018972755 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:28.979 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:28 smithi027 podman[69950]: 2024-04-25 01:22:28.770810609 +0000 UTC m=+0.142653786 container create b7ee065b9d57978bf98177a772f75907e95e792f9c79d2edda615d47cca41568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T01:22:28.979 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:28 smithi027 podman[69950]: 2024-04-25 01:22:28.998318111 +0000 UTC m=+0.370161247 container init b7ee065b9d57978bf98177a772f75907e95e792f9c79d2edda615d47cca41568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:22:29.444 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:29 smithi139 ceph-mon[32760]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:29.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:29 smithi027 ceph-mon[29197]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:29.446 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:29 smithi027 podman[69950]: 2024-04-25 01:22:29.001332901 +0000 UTC m=+0.373176037 container start b7ee065b9d57978bf98177a772f75907e95e792f9c79d2edda615d47cca41568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:22:29.446 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:29 smithi027 ceph-mds[69969]: -- 172.21.15.27:0/1640929796 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563259abba40 con 0x563259a8ec00 2024-04-25T01:22:29.446 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d[69965]: starting mds.d at 2024-04-25T01:22:29.447 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:29 smithi027 bash[69950]: b7ee065b9d57978bf98177a772f75907e95e792f9c79d2edda615d47cca41568 2024-04-25T01:22:29.447 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:22:29 smithi027 systemd[1]: Started Ceph mds.d for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:29 smithi153 ceph-mon[32621]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:30.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/4264641458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 msgr2=0x7fd7e4109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/4264641458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 0x7fd7e4109bf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009ca0 tx=0x7fd7d402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/4264641458 shutdown_connections 2024-04-25T01:22:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/4264641458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 0x7fd7e4109bf0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/4264641458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7e41029a0 0x7fd7e4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/4264641458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e41017b0 0x7fd7e4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/4264641458 >> 172.21.15.27:0/4264641458 conn(0x7fd7e40fcd40 msgr2=0x7fd7e40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/4264641458 shutdown_connections 2024-04-25T01:22:30.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.159+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/4264641458 wait complete. 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e8cf5640 1 Processor -- start 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e8cf5640 1 -- start start 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e8cf5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e41017b0 0x7fd7e40754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e8cf5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7e41029a0 0x7fd7e4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e8cf5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 0x7fd7e4074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e8cf5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e410d480 con 0x7fd7e4103360 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e8cf5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e410d300 con 0x7fd7e41017b0 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e8cf5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e410d600 con 0x7fd7e41029a0 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e41017b0 0x7fd7e40754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 0x7fd7e4074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e41017b0 0x7fd7e40754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:39408/0 (socket says 172.21.15.27:39408) 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e259c640 1 -- 172.21.15.27:0/3452702076 learned_addr learned my addr 172.21.15.27:0/3452702076 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.160+0000 7fd7e1d9b640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7e41029a0 0x7fd7e4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e259c640 1 -- 172.21.15.27:0/3452702076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7e41029a0 msgr2=0x7fd7e4073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e259c640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7e41029a0 0x7fd7e4073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e259c640 1 -- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 msgr2=0x7fd7e4074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e259c640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 0x7fd7e4074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e259c640 1 -- 172.21.15.27:0/3452702076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e4074900 con 0x7fd7e41017b0 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e1d9b640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7e41029a0 0x7fd7e4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e2d9d640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 0x7fd7e4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e259c640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e41017b0 0x7fd7e40754f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00e9c0 tx=0x7fd7dc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7d37fe640 1 -- 172.21.15.27:0/3452702076 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00cd80 con 0x7fd7e41017b0 2024-04-25T01:22:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e41a1b50 con 0x7fd7e41017b0 2024-04-25T01:22:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7d37fe640 1 -- 172.21.15.27:0/3452702076 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7dc0035f0 con 0x7fd7e41017b0 2024-04-25T01:22:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7d37fe640 1 -- 172.21.15.27:0/3452702076 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc010690 con 0x7fd7e41017b0 2024-04-25T01:22:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.161+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e41a20a0 con 0x7fd7e41017b0 2024-04-25T01:22:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.162+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e4108a70 con 0x7fd7e41017b0 2024-04-25T01:22:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.163+0000 7fd7d37fe640 1 -- 172.21.15.27:0/3452702076 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd7dc0037a0 con 0x7fd7e41017b0 2024-04-25T01:22:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.164+0000 7fd7d37fe640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd7cc0a2b90 0x7fd7cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.164+0000 7fd7e1d9b640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd7cc0a2b90 0x7fd7cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.164+0000 7fd7d37fe640 1 -- 172.21.15.27:0/3452702076 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd7dc0cf030 con 0x7fd7e41017b0 2024-04-25T01:22:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.164+0000 7fd7e1d9b640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd7cc0a2b90 0x7fd7cc0a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4075260 tx=0x7fd7d8005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:30.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.167+0000 7fd7d37fe640 1 -- 172.21.15.27:0/3452702076 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd7dc09c430 con 0x7fd7e41017b0 2024-04-25T01:22:30.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.394+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7e402d210 con 0x7fd7e41017b0 2024-04-25T01:22:30.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.394+0000 7fd7d37fe640 1 -- 172.21.15.27:0/3452702076 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7fd7dc09bd30 con 0x7fd7e41017b0 2024-04-25T01:22:30.375 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:30.375 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:22:30.376 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 5 2024-04-25T01:22:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd7cc0a2b90 msgr2=0x7fd7cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd7cc0a2b90 0x7fd7cc0a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4075260 tx=0x7fd7d8005e50 comp rx=0 tx=0).stop 2024-04-25T01:22:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e41017b0 msgr2=0x7fd7e40754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e41017b0 0x7fd7e40754f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc00e9c0 tx=0x7fd7dc00ee80 comp rx=0 tx=0).stop 2024-04-25T01:22:30.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 shutdown_connections 2024-04-25T01:22:30.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd7cc0a2b90 0x7fd7cc0a5050 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7e4103360 0x7fd7e4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7e41029a0 0x7fd7e4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 --2- 172.21.15.27:0/3452702076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e41017b0 0x7fd7e40754f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:30.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.396+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 >> 172.21.15.27:0/3452702076 conn(0x7fd7e40fcd40 msgr2=0x7fd7e4105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:30.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.397+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 shutdown_connections 2024-04-25T01:22:30.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:30.397+0000 7fd7e8cf5640 1 -- 172.21.15.27:0/3452702076 wait complete. 2024-04-25T01:22:30.387 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.27:6836/3543255516,v1:172.21.15.27:6837/3543255516] up:boot 2024-04-25T01:22:30.387 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: fsmap 4 up:standby 2024-04-25T01:22:30.387 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:22:30.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:30.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:30.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:30.388 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:30 smithi027 ceph-mon[29197]: Deploying daemon mds.e on smithi139 2024-04-25T01:22:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.27:6836/3543255516,v1:172.21.15.27:6837/3543255516] up:boot 2024-04-25T01:22:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: fsmap 4 up:standby 2024-04-25T01:22:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:22:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:30.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:30.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:30.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:30 smithi153 ceph-mon[32621]: Deploying daemon mds.e on smithi139 2024-04-25T01:22:30.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.27:6836/3543255516,v1:172.21.15.27:6837/3543255516] up:boot 2024-04-25T01:22:30.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: fsmap 4 up:standby 2024-04-25T01:22:30.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:22:30.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:30.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:30.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:30.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:30.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:30 smithi139 ceph-mon[32760]: Deploying daemon mds.e on smithi139 2024-04-25T01:22:31.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:31 smithi153 ceph-mon[32621]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:31.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:31 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3452702076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:31.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:31 smithi139 ceph-mon[32760]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:31.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:31 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3452702076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:31.544 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:31 smithi139 systemd[1]: Starting Ceph mds.e for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:31.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:31 smithi027 ceph-mon[29197]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:31.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:31 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3452702076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:31.961 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:31 smithi139 podman[62471]: 2024-04-25 01:22:31.632130402 +0000 UTC m=+0.018080953 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:31.961 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:31 smithi139 podman[62471]: 2024-04-25 01:22:31.743664004 +0000 UTC m=+0.129614540 container create 9d7cc3edf7cf7f5bce374332e4f84fcf2b19a46cfa59a4bad3b7826a933e5993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T01:22:32.293 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:31 smithi139 podman[62471]: 2024-04-25 01:22:31.97986893 +0000 UTC m=+0.365819466 container init 9d7cc3edf7cf7f5bce374332e4f84fcf2b19a46cfa59a4bad3b7826a933e5993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:22:32.293 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:31 smithi139 podman[62471]: 2024-04-25 01:22:31.983254887 +0000 UTC m=+0.369205424 container start 9d7cc3edf7cf7f5bce374332e4f84fcf2b19a46cfa59a4bad3b7826a933e5993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:22:32.294 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:32 smithi139 ceph-mds[62490]: -- 172.21.15.139:0/1040338779 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627efc9da40 con 0x5627efc70c00 2024-04-25T01:22:32.294 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:32 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e[62486]: starting mds.e at 2024-04-25T01:22:32.294 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:32 smithi139 bash[62471]: 9d7cc3edf7cf7f5bce374332e4f84fcf2b19a46cfa59a4bad3b7826a933e5993 2024-04-25T01:22:32.294 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:22:32 smithi139 systemd[1]: Started Ceph mds.e for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:32.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:32.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 -- 172.21.15.27:0/4235746413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c103340 msgr2=0x7fb29c109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/4235746413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c103340 0x7fb29c109bd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb294009ca0 tx=0x7fb29402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 -- 172.21.15.27:0/4235746413 shutdown_connections 2024-04-25T01:22:32.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/4235746413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c103340 0x7fb29c109bd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/4235746413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb29c102980 0x7fb29c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/4235746413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb29c101790 0x7fb29c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 -- 172.21.15.27:0/4235746413 >> 172.21.15.27:0/4235746413 conn(0x7fb29c0fcd40 msgr2=0x7fb29c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 -- 172.21.15.27:0/4235746413 shutdown_connections 2024-04-25T01:22:32.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.524+0000 7fb2a21dc640 1 -- 172.21.15.27:0/4235746413 wait complete. 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb2a21dc640 1 Processor -- start 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb2a21dc640 1 -- start start 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb2a21dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c101790 0x7fb29c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb2a21dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb29c102980 0x7fb29c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb2a21dc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb29c103340 0x7fb29c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb2a21dc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c10d410 con 0x7fb29c101790 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb2a21dc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c10d290 con 0x7fb29c103340 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb2a21dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c10d590 con 0x7fb29c102980 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb29b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c101790 0x7fb29c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb29b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c101790 0x7fb29c197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41740/0 (socket says 172.21.15.27:41740) 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb29b7fe640 1 -- 172.21.15.27:0/89412908 learned_addr learned my addr 172.21.15.27:0/89412908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:32.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb29bfff640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb29c103340 0x7fb29c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.525+0000 7fb29affd640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb29c102980 0x7fb29c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb29b7fe640 1 -- 172.21.15.27:0/89412908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb29c102980 msgr2=0x7fb29c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb29b7fe640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb29c102980 0x7fb29c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb29b7fe640 1 -- 172.21.15.27:0/89412908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb29c103340 msgr2=0x7fb29c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb29b7fe640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb29c103340 0x7fb29c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb29b7fe640 1 -- 172.21.15.27:0/89412908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c19f680 con 0x7fb29c101790 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb29b7fe640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c101790 0x7fb29c197a60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00e9c0 tx=0x7fb28c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb298ff9640 1 -- 172.21.15.27:0/89412908 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c00cd80 con 0x7fb29c101790 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb298ff9640 1 -- 172.21.15.27:0/89412908 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28c0035f0 con 0x7fb29c101790 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29c19f970 con 0x7fb29c101790 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb298ff9640 1 -- 172.21.15.27:0/89412908 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c010690 con 0x7fb29c101790 2024-04-25T01:22:32.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.526+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29c19fe70 con 0x7fb29c101790 2024-04-25T01:22:32.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.527+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb29c108ab0 con 0x7fb29c101790 2024-04-25T01:22:32.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.528+0000 7fb298ff9640 1 -- 172.21.15.27:0/89412908 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb28c0037a0 con 0x7fb29c101790 2024-04-25T01:22:32.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.529+0000 7fb298ff9640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb2840a2b90 0x7fb2840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.529+0000 7fb29affd640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb2840a2b90 0x7fb2840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.529+0000 7fb298ff9640 1 -- 172.21.15.27:0/89412908 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb28c0ceef0 con 0x7fb29c101790 2024-04-25T01:22:32.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.529+0000 7fb29affd640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb2840a2b90 0x7fb2840a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb29c198f70 tx=0x7fb290005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.533+0000 7fb298ff9640 1 -- 172.21.15.27:0/89412908 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb28c09c2f0 con 0x7fb29c101790 2024-04-25T01:22:32.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.759+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb29c02d3f0 con 0x7fb29c101790 2024-04-25T01:22:32.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.759+0000 7fb298ff9640 1 -- 172.21.15.27:0/89412908 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7fb28c09bbf0 con 0x7fb29c101790 2024-04-25T01:22:32.740 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:32.740 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T01:22:32.741 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 6 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb2840a2b90 msgr2=0x7fb2840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb2840a2b90 0x7fb2840a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb29c198f70 tx=0x7fb290005f90 comp rx=0 tx=0).stop 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c101790 msgr2=0x7fb29c197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c101790 0x7fb29c197a60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00e9c0 tx=0x7fb28c00ee80 comp rx=0 tx=0).stop 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 shutdown_connections 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb2840a2b90 0x7fb2840a5050 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb29c103340 0x7fb29c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb29c102980 0x7fb29c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 --2- 172.21.15.27:0/89412908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb29c101790 0x7fb29c197a60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 >> 172.21.15.27:0/89412908 conn(0x7fb29c0fcd40 msgr2=0x7fb29c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 shutdown_connections 2024-04-25T01:22:32.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:32.761+0000 7fb2a21dc640 1 -- 172.21.15.27:0/89412908 wait complete. 2024-04-25T01:22:33.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: Deploying daemon mds.f on smithi153 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.139:6834/418855472,v1:172.21.15.139:6835/418855472] up:boot 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: fsmap 5 up:standby 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:22:33.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:33 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/89412908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:33.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:33.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:33.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:33.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:33.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: Deploying daemon mds.f on smithi153 2024-04-25T01:22:33.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.139:6834/418855472,v1:172.21.15.139:6835/418855472] up:boot 2024-04-25T01:22:33.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: fsmap 5 up:standby 2024-04-25T01:22:33.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:22:33.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:33 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/89412908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:33.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: Deploying daemon mds.f on smithi153 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.139:6834/418855472,v1:172.21.15.139:6835/418855472] up:boot 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: fsmap 5 up:standby 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:22:33.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:33 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/89412908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:34.524 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 systemd[1]: Starting Ceph mds.f for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:34.754 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:34.777 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 podman[63383]: 2024-04-25 01:22:34.446215464 +0000 UTC m=+0.018399606 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:34.777 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 podman[63383]: 2024-04-25 01:22:34.547523813 +0000 UTC m=+0.119707945 container create 80297802d4b3fa7f0e69b986469141df334fc4c27b41fa2cc418c7a4c7d2df48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T01:22:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.889+0000 7f0ad96a7640 1 -- 172.21.15.27:0/716533943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4102230 msgr2=0x7f0ad4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.889+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/716533943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4102230 0x7f0ad4108ac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009ca0 tx=0x7f0ac402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.889+0000 7f0ad96a7640 1 -- 172.21.15.27:0/716533943 shutdown_connections 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.889+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/716533943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4102230 0x7f0ad4108ac0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.889+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/716533943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ad4101870 0x7f0ad4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.889+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/716533943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ad4100680 0x7f0ad4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.890+0000 7f0ad96a7640 1 -- 172.21.15.27:0/716533943 >> 172.21.15.27:0/716533943 conn(0x7f0ad40fbbf0 msgr2=0x7f0ad40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.890+0000 7f0ad96a7640 1 -- 172.21.15.27:0/716533943 shutdown_connections 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.890+0000 7f0ad96a7640 1 -- 172.21.15.27:0/716533943 wait complete. 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.890+0000 7f0ad96a7640 1 Processor -- start 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.890+0000 7f0ad96a7640 1 -- start start 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.890+0000 7f0ad96a7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ad4100680 0x7f0ad4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad96a7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4101870 0x7f0ad4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad96a7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ad4102230 0x7f0ad419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad96a7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad410c300 con 0x7f0ad4101870 2024-04-25T01:22:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad96a7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad410c180 con 0x7f0ad4102230 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad96a7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad410c480 con 0x7f0ad4100680 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4101870 0x7f0ad4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4101870 0x7f0ad4197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41746/0 (socket says 172.21.15.27:41746) 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 -- 172.21.15.27:0/4287421799 learned_addr learned my addr 172.21.15.27:0/4287421799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad2ffd640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ad4100680 0x7f0ad4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad37fe640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ad4102230 0x7f0ad419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 -- 172.21.15.27:0/4287421799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ad4100680 msgr2=0x7f0ad4197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ad4100680 0x7f0ad4197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 -- 172.21.15.27:0/4287421799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ad4102230 msgr2=0x7f0ad419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ad4102230 0x7f0ad419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 -- 172.21.15.27:0/4287421799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad419f680 con 0x7f0ad4101870 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad27fc640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4101870 0x7f0ad4197fa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800e9f0 tx=0x7f0ac800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0abbfff640 1 -- 172.21.15.27:0/4287421799 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac800cd80 con 0x7f0ad4101870 2024-04-25T01:22:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0abbfff640 1 -- 172.21.15.27:0/4287421799 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac8010910 con 0x7f0ad4101870 2024-04-25T01:22:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad419f970 con 0x7f0ad4101870 2024-04-25T01:22:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0abbfff640 1 -- 172.21.15.27:0/4287421799 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8018990 con 0x7f0ad4101870 2024-04-25T01:22:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.891+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad419fec0 con 0x7f0ad4101870 2024-04-25T01:22:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.892+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad41a00f0 con 0x7f0ad4101870 2024-04-25T01:22:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.894+0000 7f0abbfff640 1 -- 172.21.15.27:0/4287421799 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0ac8010ac0 con 0x7f0ad4101870 2024-04-25T01:22:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.894+0000 7f0abbfff640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0abc0a2b90 0x7f0abc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.895+0000 7f0ad2ffd640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0abc0a2b90 0x7f0abc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.895+0000 7f0abbfff640 1 -- 172.21.15.27:0/4287421799 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0ac80cf3a0 con 0x7f0ad4101870 2024-04-25T01:22:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.895+0000 7f0ad2ffd640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0abc0a2b90 0x7f0abc0a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ad41016d0 tx=0x7f0acc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:34.898+0000 7f0abbfff640 1 -- 172.21.15.27:0/4287421799 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ac809c7a0 con 0x7f0ad4101870 2024-04-25T01:22:35.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.125+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ad41a02c0 con 0x7f0ad4101870 2024-04-25T01:22:35.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.125+0000 7f0abbfff640 1 -- 172.21.15.27:0/4287421799 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7f0ac809c0a0 con 0x7f0ad4101870 2024-04-25T01:22:35.106 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:35.106 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T01:22:35.106 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 6 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0abc0a2b90 msgr2=0x7f0abc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0abc0a2b90 0x7f0abc0a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ad41016d0 tx=0x7f0acc005f90 comp rx=0 tx=0).stop 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4101870 msgr2=0x7f0ad4197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4101870 0x7f0ad4197fa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800e9f0 tx=0x7f0ac800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 shutdown_connections 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0abc0a2b90 0x7f0abc0a5050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ad4102230 0x7f0ad419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad4101870 0x7f0ad4197fa0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 --2- 172.21.15.27:0/4287421799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ad4100680 0x7f0ad4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 >> 172.21.15.27:0/4287421799 conn(0x7f0ad40fbbf0 msgr2=0x7f0ad40fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:35.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 shutdown_connections 2024-04-25T01:22:35.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:35.127+0000 7f0ad96a7640 1 -- 172.21.15.27:0/4287421799 wait complete. 2024-04-25T01:22:35.115 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 podman[63383]: 2024-04-25 01:22:34.800077892 +0000 UTC m=+0.372262024 container init 80297802d4b3fa7f0e69b986469141df334fc4c27b41fa2cc418c7a4c7d2df48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:22:35.115 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 podman[63383]: 2024-04-25 01:22:34.802900457 +0000 UTC m=+0.375084598 container start 80297802d4b3fa7f0e69b986469141df334fc4c27b41fa2cc418c7a4c7d2df48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T01:22:35.115 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 ceph-mds[63402]: -- 172.21.15.153:0/1600833002 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561006d45a40 con 0x561006d18c00 2024-04-25T01:22:35.115 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f[63398]: starting mds.f at 2024-04-25T01:22:35.115 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 bash[63383]: 80297802d4b3fa7f0e69b986469141df334fc4c27b41fa2cc418c7a4c7d2df48 2024-04-25T01:22:35.116 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:22:34 smithi153 systemd[1]: Started Ceph mds.f for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:35.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:35 smithi027 ceph-mon[29197]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:35.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:35.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:35.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:35 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:35.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:35 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4287421799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:35.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:35 smithi153 ceph-mon[32621]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:35.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:35.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:35.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:35 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:35.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:35 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4287421799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:35.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:35 smithi139 ceph-mon[32760]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:35.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:35.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:35.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:35.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:35 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:35.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:35 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4287421799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:36.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:36 smithi027 ceph-mon[29197]: Deploying daemon mds.c on smithi153 2024-04-25T01:22:36.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:36 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6834/1981209233,v1:172.21.15.153:6835/1981209233] up:boot 2024-04-25T01:22:36.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:36 smithi027 ceph-mon[29197]: fsmap 6 up:standby 2024-04-25T01:22:36.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:22:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:36 smithi153 ceph-mon[32621]: Deploying daemon mds.c on smithi153 2024-04-25T01:22:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:36 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6834/1981209233,v1:172.21.15.153:6835/1981209233] up:boot 2024-04-25T01:22:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:36 smithi153 ceph-mon[32621]: fsmap 6 up:standby 2024-04-25T01:22:36.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:22:36.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:36 smithi139 ceph-mon[32760]: Deploying daemon mds.c on smithi153 2024-04-25T01:22:36.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:36 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6834/1981209233,v1:172.21.15.153:6835/1981209233] up:boot 2024-04-25T01:22:36.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:36 smithi139 ceph-mon[32760]: fsmap 6 up:standby 2024-04-25T01:22:36.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:22:37.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 -- 172.21.15.27:0/3374532864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 msgr2=0x7f87f0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 --2- 172.21.15.27:0/3374532864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 0x7f87f0101d10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009900 tx=0x7f87d802a410 comp rx=0 tx=0).stop 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 -- 172.21.15.27:0/3374532864 shutdown_connections 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 --2- 172.21.15.27:0/3374532864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87f0102250 0x7f87f0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 --2- 172.21.15.27:0/3374532864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 0x7f87f0101d10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 --2- 172.21.15.27:0/3374532864 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87f01006a0 0x7f87f0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 -- 172.21.15.27:0/3374532864 >> 172.21.15.27:0/3374532864 conn(0x7f87f00fbc30 msgr2=0x7f87f00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 -- 172.21.15.27:0/3374532864 shutdown_connections 2024-04-25T01:22:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.254+0000 7f87f6316640 1 -- 172.21.15.27:0/3374532864 wait complete. 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87f6316640 1 Processor -- start 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87f6316640 1 -- start start 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87f6316640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87f01006a0 0x7f87f006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87f6316640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 0x7f87f019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87f6316640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87f0102250 0x7f87f019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87f6316640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f87f010c320 con 0x7f87f0101890 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87f6316640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f87f010c1a0 con 0x7f87f01006a0 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87f6316640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87f010c4a0 con 0x7f87f0102250 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.255+0000 7f87ef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 0x7f87f019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 0x7f87f019c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41758/0 (socket says 172.21.15.27:41758) 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ef7fe640 1 -- 172.21.15.27:0/3064064694 learned_addr learned my addr 172.21.15.27:0/3064064694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87effff640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87f01006a0 0x7f87f006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87f48b3640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87f0102250 0x7f87f019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ef7fe640 1 -- 172.21.15.27:0/3064064694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87f0102250 msgr2=0x7f87f019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ef7fe640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87f0102250 0x7f87f019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ef7fe640 1 -- 172.21.15.27:0/3064064694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87f01006a0 msgr2=0x7f87f006a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ef7fe640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87f01006a0 0x7f87f006a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ef7fe640 1 -- 172.21.15.27:0/3064064694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f019f6c0 con 0x7f87f0101890 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ef7fe640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 0x7f87f019c7c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009230 tx=0x7f87d8031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ed7fa640 1 -- 172.21.15.27:0/3064064694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d803f030 con 0x7f87f0101890 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ed7fa640 1 -- 172.21.15.27:0/3064064694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87d8003ad0 con 0x7f87f0101890 2024-04-25T01:22:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f019f950 con 0x7f87f0101890 2024-04-25T01:22:37.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87ed7fa640 1 -- 172.21.15.27:0/3064064694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8035070 con 0x7f87f0101890 2024-04-25T01:22:37.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.256+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f019fe40 con 0x7f87f0101890 2024-04-25T01:22:37.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.257+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f0107a40 con 0x7f87f0101890 2024-04-25T01:22:37.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.258+0000 7f87ed7fa640 1 -- 172.21.15.27:0/3064064694 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f87d8042030 con 0x7f87f0101890 2024-04-25T01:22:37.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.259+0000 7f87ed7fa640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f87dc0a2b90 0x7f87dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.259+0000 7f87effff640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f87dc0a2b90 0x7f87dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.259+0000 7f87ed7fa640 1 -- 172.21.15.27:0/3064064694 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f87d80ef080 con 0x7f87f0101890 2024-04-25T01:22:37.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.260+0000 7f87effff640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f87dc0a2b90 0x7f87dc0a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f87e0005fd0 tx=0x7f87e000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.264+0000 7f87ed7fa640 1 -- 172.21.15.27:0/3064064694 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f87d80bbe30 con 0x7f87f0101890 2024-04-25T01:22:37.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:37 smithi153 ceph-mon[32621]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:37.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:37 smithi027 ceph-mon[29197]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:37.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.489+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87f002d380 con 0x7f87f0101890 2024-04-25T01:22:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.490+0000 7f87ed7fa640 1 -- 172.21.15.27:0/3064064694 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f87d80c0130 con 0x7f87f0101890 2024-04-25T01:22:37.470 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:37.470 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T01:22:37.471 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-04-25T01:22:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f87dc0a2b90 msgr2=0x7f87dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f87dc0a2b90 0x7f87dc0a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f87e0005fd0 tx=0x7f87e000b410 comp rx=0 tx=0).stop 2024-04-25T01:22:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 msgr2=0x7f87f019c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 0x7f87f019c7c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009230 tx=0x7f87d8031b60 comp rx=0 tx=0).stop 2024-04-25T01:22:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 shutdown_connections 2024-04-25T01:22:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f87dc0a2b90 0x7f87dc0a5050 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87f0102250 0x7f87f019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87f0101890 0x7f87f019c7c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 --2- 172.21.15.27:0/3064064694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87f01006a0 0x7f87f006a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 >> 172.21.15.27:0/3064064694 conn(0x7f87f00fbc30 msgr2=0x7f87f0104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 shutdown_connections 2024-04-25T01:22:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:37.493+0000 7f87f6316640 1 -- 172.21.15.27:0/3064064694 wait complete. 2024-04-25T01:22:37.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:37 smithi139 ceph-mon[32760]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:38.428 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:38 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3064064694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:38.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:38.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:38.429 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:38 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:38.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:38 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3064064694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:38.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:38.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:38.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:38 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:38 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3064064694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:38.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:38 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:39.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:39 smithi027 ceph-mon[29197]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:39.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:39 smithi027 ceph-mon[29197]: Deploying daemon mds.h on smithi139 2024-04-25T01:22:39.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:39 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] up:boot 2024-04-25T01:22:39.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:39 smithi027 ceph-mon[29197]: fsmap 7 up:standby 2024-04-25T01:22:39.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:39 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:22:39.472 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:39 smithi139 ceph-mon[32760]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:39.472 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:39 smithi139 ceph-mon[32760]: Deploying daemon mds.h on smithi139 2024-04-25T01:22:39.472 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:39 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] up:boot 2024-04-25T01:22:39.472 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:39 smithi139 ceph-mon[32760]: fsmap 7 up:standby 2024-04-25T01:22:39.472 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:39 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:22:39.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:39.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:39 smithi153 ceph-mon[32621]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:39.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:39 smithi153 ceph-mon[32621]: Deploying daemon mds.h on smithi139 2024-04-25T01:22:39.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:39 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] up:boot 2024-04-25T01:22:39.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:39 smithi153 ceph-mon[32621]: fsmap 7 up:standby 2024-04-25T01:22:39.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:39 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:22:39.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 -- 172.21.15.27:0/1344540415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 msgr2=0x7f2e04100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/1344540415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 0x7f2e04100ac0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009900 tx=0x7f2df402a410 comp rx=0 tx=0).stop 2024-04-25T01:22:39.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 -- 172.21.15.27:0/1344540415 shutdown_connections 2024-04-25T01:22:39.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/1344540415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e04102250 0x7f2e04108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/1344540415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04101890 0x7f2e04101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/1344540415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 0x7f2e04100ac0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 -- 172.21.15.27:0/1344540415 >> 172.21.15.27:0/1344540415 conn(0x7f2e040fbc30 msgr2=0x7f2e040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 -- 172.21.15.27:0/1344540415 shutdown_connections 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.626+0000 7f2e0a74f640 1 -- 172.21.15.27:0/1344540415 wait complete. 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e0a74f640 1 Processor -- start 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e0a74f640 1 -- start start 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e0a74f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 0x7f2e04197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e0a74f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04101890 0x7f2e04198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e0a74f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e04102250 0x7f2e0419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e0a74f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0410c2f0 con 0x7f2e041006a0 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e0a74f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0410c170 con 0x7f2e04102250 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e0a74f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0410c470 con 0x7f2e04101890 2024-04-25T01:22:39.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.627+0000 7f2e03fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 0x7f2e04197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e03fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 0x7f2e04197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52204/0 (socket says 172.21.15.27:52204) 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e03fff640 1 -- 172.21.15.27:0/2557827905 learned_addr learned my addr 172.21.15.27:0/2557827905 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e08cec640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e04102250 0x7f2e0419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e037fe640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04101890 0x7f2e04198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e03fff640 1 -- 172.21.15.27:0/2557827905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04101890 msgr2=0x7f2e04198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e03fff640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04101890 0x7f2e04198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e03fff640 1 -- 172.21.15.27:0/2557827905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e04102250 msgr2=0x7f2e0419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e03fff640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e04102250 0x7f2e0419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e03fff640 1 -- 172.21.15.27:0/2557827905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0419f6f0 con 0x7f2e041006a0 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e037fe640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04101890 0x7f2e04198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e03fff640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 0x7f2e04197b40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2df4031610 tx=0x7f2df40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e017fa640 1 -- 172.21.15.27:0/2557827905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df403f030 con 0x7f2e041006a0 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e0419f980 con 0x7f2e041006a0 2024-04-25T01:22:39.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e017fa640 1 -- 172.21.15.27:0/2557827905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2df4003ce0 con 0x7f2e041006a0 2024-04-25T01:22:39.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e017fa640 1 -- 172.21.15.27:0/2557827905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4035070 con 0x7f2e041006a0 2024-04-25T01:22:39.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.628+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e0419fe70 con 0x7f2e041006a0 2024-04-25T01:22:39.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.629+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e041a0040 con 0x7f2e041006a0 2024-04-25T01:22:39.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.630+0000 7f2e017fa640 1 -- 172.21.15.27:0/2557827905 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2df4042030 con 0x7f2e041006a0 2024-04-25T01:22:39.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.631+0000 7f2e017fa640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2dec0a2b90 0x7f2dec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.631+0000 7f2e037fe640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2dec0a2b90 0x7f2dec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.631+0000 7f2e017fa640 1 -- 172.21.15.27:0/2557827905 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2df40ef300 con 0x7f2e041006a0 2024-04-25T01:22:39.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.632+0000 7f2e037fe640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2dec0a2b90 0x7f2dec0a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2e04198fe0 tx=0x7f2df8005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.636+0000 7f2e017fa640 1 -- 172.21.15.27:0/2557827905 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2df40bc700 con 0x7f2e041006a0 2024-04-25T01:22:39.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.861+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e041a0330 con 0x7f2e041006a0 2024-04-25T01:22:39.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.862+0000 7f2e017fa640 1 -- 172.21.15.27:0/2557827905 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f2df40bc000 con 0x7f2e041006a0 2024-04-25T01:22:39.842 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:39.842 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T01:22:39.843 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2dec0a2b90 msgr2=0x7f2dec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2dec0a2b90 0x7f2dec0a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2e04198fe0 tx=0x7f2df8005d30 comp rx=0 tx=0).stop 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 msgr2=0x7f2e04197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 0x7f2e04197b40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2df4031610 tx=0x7f2df40043b0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 shutdown_connections 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2dec0a2b90 0x7f2dec0a5050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e04102250 0x7f2e0419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04101890 0x7f2e04198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 --2- 172.21.15.27:0/2557827905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e041006a0 0x7f2e04197b40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 >> 172.21.15.27:0/2557827905 conn(0x7f2e040fbc30 msgr2=0x7f2e040fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 shutdown_connections 2024-04-25T01:22:39.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:39.864+0000 7f2e0a74f640 1 -- 172.21.15.27:0/2557827905 wait complete. 2024-04-25T01:22:40.137 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 systemd[1]: Starting Ceph mds.h for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:40.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:40 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2557827905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:40.447 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:40 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2557827905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:40.448 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 podman[62782]: 2024-04-25 01:22:40.142681063 +0000 UTC m=+0.018262250 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:40.448 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 podman[62782]: 2024-04-25 01:22:40.248939982 +0000 UTC m=+0.124521185 container create e5e6102d99a620215e55b8d410fb9fceafedccafbd7a198f5bfb45ca39033370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T01:22:40.448 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 podman[62782]: 2024-04-25 01:22:40.467994404 +0000 UTC m=+0.343575564 container init e5e6102d99a620215e55b8d410fb9fceafedccafbd7a198f5bfb45ca39033370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T01:22:40.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:40 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2557827905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:40.793 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 podman[62782]: 2024-04-25 01:22:40.470710667 +0000 UTC m=+0.346291834 container start e5e6102d99a620215e55b8d410fb9fceafedccafbd7a198f5bfb45ca39033370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T01:22:40.793 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 ceph-mds[62801]: -- 172.21.15.139:0/1045669639 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d82697a40 con 0x560d8266a800 2024-04-25T01:22:40.793 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h[62797]: starting mds.h at 2024-04-25T01:22:40.793 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 bash[62782]: e5e6102d99a620215e55b8d410fb9fceafedccafbd7a198f5bfb45ca39033370 2024-04-25T01:22:40.793 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:22:40 smithi139 systemd[1]: Started Ceph mds.h for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:41.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:41 smithi027 ceph-mon[29197]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:41.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:41.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:41.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:41 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:41.472 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:41 smithi153 ceph-mon[32621]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:41.472 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.472 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.473 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.473 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:41.473 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:41.473 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:41 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:41.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:41 smithi139 ceph-mon[32760]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:41.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:41.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:41.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:41.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:41 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:41.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 -- 172.21.15.27:0/656644919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb41029a0 msgr2=0x7f4fb4102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/656644919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb41029a0 0x7f4fb4102e20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4009900 tx=0x7f4fa402a410 comp rx=0 tx=0).stop 2024-04-25T01:22:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 -- 172.21.15.27:0/656644919 shutdown_connections 2024-04-25T01:22:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/656644919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fb4103360 0x7f4fb4109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/656644919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb41029a0 0x7f4fb4102e20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/656644919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fb41017b0 0x7f4fb4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 -- 172.21.15.27:0/656644919 >> 172.21.15.27:0/656644919 conn(0x7f4fb40fcd40 msgr2=0x7f4fb40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 -- 172.21.15.27:0/656644919 shutdown_connections 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.990+0000 7f4fbb56a640 1 -- 172.21.15.27:0/656644919 wait complete. 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fbb56a640 1 Processor -- start 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fbb56a640 1 -- start start 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fbb56a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fb41017b0 0x7f4fb4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fbb56a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fb41029a0 0x7f4fb4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fbb56a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb4103360 0x7f4fb419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fbb56a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb410d430 con 0x7f4fb4103360 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fbb56a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb410d2b0 con 0x7f4fb41017b0 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fbb56a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb410d5b0 con 0x7f4fb41029a0 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9b07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb4103360 0x7f4fb419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9b07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb4103360 0x7f4fb419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52226/0 (socket says 172.21.15.27:52226) 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9b07640 1 -- 172.21.15.27:0/2313947680 learned_addr learned my addr 172.21.15.27:0/2313947680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb8b05640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fb41029a0 0x7f4fb4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9306640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fb41017b0 0x7f4fb4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9b07640 1 -- 172.21.15.27:0/2313947680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fb41029a0 msgr2=0x7f4fb4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9b07640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fb41029a0 0x7f4fb4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9b07640 1 -- 172.21.15.27:0/2313947680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fb41017b0 msgr2=0x7f4fb4197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9b07640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fb41017b0 0x7f4fb4197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9b07640 1 -- 172.21.15.27:0/2313947680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb419f6a0 con 0x7f4fb4103360 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.991+0000 7f4fb9306640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fb41017b0 0x7f4fb4197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.992+0000 7f4fb9b07640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb4103360 0x7f4fb419f030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00eaa0 tx=0x7f4fac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.992+0000 7f4faa7fc640 1 -- 172.21.15.27:0/2313947680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac00cdb0 con 0x7f4fb4103360 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.992+0000 7f4faa7fc640 1 -- 172.21.15.27:0/2313947680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fac005020 con 0x7f4fb4103360 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.992+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb419f990 con 0x7f4fb4103360 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.992+0000 7f4faa7fc640 1 -- 172.21.15.27:0/2313947680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac0116b0 con 0x7f4fb4103360 2024-04-25T01:22:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.992+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb419feb0 con 0x7f4fb4103360 2024-04-25T01:22:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.993+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb4108ad0 con 0x7f4fb4103360 2024-04-25T01:22:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.994+0000 7f4faa7fc640 1 -- 172.21.15.27:0/2313947680 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4fac005560 con 0x7f4fb4103360 2024-04-25T01:22:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.995+0000 7f4faa7fc640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4fa00a2b90 0x7f4fa00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.995+0000 7f4fb9306640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4fa00a2b90 0x7f4fa00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.995+0000 7f4faa7fc640 1 -- 172.21.15.27:0/2313947680 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4fac0cee10 con 0x7f4fb4103360 2024-04-25T01:22:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.995+0000 7f4fb9306640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4fa00a2b90 0x7f4fa00a5050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4102800 tx=0x7f4fb000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:41.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:41.999+0000 7f4faa7fc640 1 -- 172.21.15.27:0/2313947680 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4fac016030 con 0x7f4fb4103360 2024-04-25T01:22:42.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.222+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fb402d250 con 0x7f4fb4103360 2024-04-25T01:22:42.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.222+0000 7f4faa7fc640 1 -- 172.21.15.27:0/2313947680 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7f4fac09c210 con 0x7f4fb4103360 2024-04-25T01:22:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:42.204 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T01:22:42.204 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 9 2024-04-25T01:22:42.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4fa00a2b90 msgr2=0x7f4fa00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4fa00a2b90 0x7f4fa00a5050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4102800 tx=0x7f4fb000a560 comp rx=0 tx=0).stop 2024-04-25T01:22:42.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb4103360 msgr2=0x7f4fb419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:42.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb4103360 0x7f4fb419f030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00eaa0 tx=0x7f4fac00ef60 comp rx=0 tx=0).stop 2024-04-25T01:22:42.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 shutdown_connections 2024-04-25T01:22:42.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4fa00a2b90 0x7f4fa00a5050 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb4103360 0x7f4fb419f030 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fb41029a0 0x7f4fb4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 --2- 172.21.15.27:0/2313947680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4fb41017b0 0x7f4fb4197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:42.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.224+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 >> 172.21.15.27:0/2313947680 conn(0x7f4fb40fcd40 msgr2=0x7f4fb4105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:42.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.225+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 shutdown_connections 2024-04-25T01:22:42.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:42.225+0000 7f4fbb56a640 1 -- 172.21.15.27:0/2313947680 wait complete. 2024-04-25T01:22:42.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:42 smithi027 ceph-mon[29197]: Deploying daemon mds.i on smithi153 2024-04-25T01:22:42.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:42 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.139:6836/1472553553,v1:172.21.15.139:6837/1472553553] up:boot 2024-04-25T01:22:42.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:42 smithi027 ceph-mon[29197]: fsmap 8 up:standby 2024-04-25T01:22:42.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:42 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:22:42.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:42 smithi153 ceph-mon[32621]: Deploying daemon mds.i on smithi153 2024-04-25T01:22:42.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:42 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.139:6836/1472553553,v1:172.21.15.139:6837/1472553553] up:boot 2024-04-25T01:22:42.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:42 smithi153 ceph-mon[32621]: fsmap 8 up:standby 2024-04-25T01:22:42.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:42 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:22:42.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:42 smithi139 ceph-mon[32760]: Deploying daemon mds.i on smithi153 2024-04-25T01:22:42.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:42 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.139:6836/1472553553,v1:172.21.15.139:6837/1472553553] up:boot 2024-04-25T01:22:42.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:42 smithi139 ceph-mon[32760]: fsmap 8 up:standby 2024-04-25T01:22:42.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:42 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:22:42.949 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:42 smithi153 systemd[1]: Starting Ceph mds.i for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:43.220 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:43 smithi153 ceph-mon[32621]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:43.220 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:43 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2313947680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:43.221 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:42 smithi153 podman[64000]: 2024-04-25 01:22:42.871147261 +0000 UTC m=+0.018221576 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:43.221 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:42 smithi153 podman[64000]: 2024-04-25 01:22:42.973883679 +0000 UTC m=+0.120957988 container create 4b77951ab54e6b431f164bd567c6857fe3ade63e551310cf25f03cbdae7d6df9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T01:22:43.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:43 smithi027 ceph-mon[29197]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:43.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:43 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2313947680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:43.527 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:43 smithi153 podman[64000]: 2024-04-25 01:22:43.240339033 +0000 UTC m=+0.387413344 container init 4b77951ab54e6b431f164bd567c6857fe3ade63e551310cf25f03cbdae7d6df9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:22:43.527 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:43 smithi153 podman[64000]: 2024-04-25 01:22:43.243105741 +0000 UTC m=+0.390180059 container start 4b77951ab54e6b431f164bd567c6857fe3ade63e551310cf25f03cbdae7d6df9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T01:22:43.527 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:43 smithi153 ceph-mds[64019]: -- 172.21.15.153:0/1233331183 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ec1df1a40 con 0x560ec1dc5000 2024-04-25T01:22:43.527 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i[64015]: starting mds.i at 2024-04-25T01:22:43.527 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:43 smithi153 bash[64000]: 4b77951ab54e6b431f164bd567c6857fe3ade63e551310cf25f03cbdae7d6df9 2024-04-25T01:22:43.527 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:22:43 smithi153 systemd[1]: Started Ceph mds.i for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:43.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:43 smithi139 ceph-mon[32760]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:43.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:43 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2313947680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:44.218 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 -- 172.21.15.27:0/1772329284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd8103360 msgr2=0x7fddd8109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 --2- 172.21.15.27:0/1772329284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd8103360 0x7fddd8109bf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009ca0 tx=0x7fddd002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 -- 172.21.15.27:0/1772329284 shutdown_connections 2024-04-25T01:22:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 --2- 172.21.15.27:0/1772329284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd8103360 0x7fddd8109bf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 --2- 172.21.15.27:0/1772329284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddd81029a0 0x7fddd8102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 --2- 172.21.15.27:0/1772329284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddd81017b0 0x7fddd8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 -- 172.21.15.27:0/1772329284 >> 172.21.15.27:0/1772329284 conn(0x7fddd80fcd40 msgr2=0x7fddd80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 -- 172.21.15.27:0/1772329284 shutdown_connections 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.339+0000 7fdddeeda640 1 -- 172.21.15.27:0/1772329284 wait complete. 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddeeda640 1 Processor -- start 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddeeda640 1 -- start start 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddeeda640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddd81017b0 0x7fddd8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddeeda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd81029a0 0x7fddd8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddeeda640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddd8103360 0x7fddd819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddeeda640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fddd810d430 con 0x7fddd81029a0 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddeeda640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fddd810d2b0 con 0x7fddd8103360 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddeeda640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fddd810d5b0 con 0x7fddd81017b0 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd81029a0 0x7fddd8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd81029a0 0x7fddd8197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52246/0 (socket says 172.21.15.27:52246) 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fddd7fff640 1 -- 172.21.15.27:0/2874262375 learned_addr learned my addr 172.21.15.27:0/2874262375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddd477640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddd8103360 0x7fddd819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddcc76640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddd81017b0 0x7fddd8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fddd7fff640 1 -- 172.21.15.27:0/2874262375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddd81017b0 msgr2=0x7fddd8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fddd7fff640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddd81017b0 0x7fddd8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fddd7fff640 1 -- 172.21.15.27:0/2874262375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddd8103360 msgr2=0x7fddd819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fddd7fff640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddd8103360 0x7fddd819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fddd7fff640 1 -- 172.21.15.27:0/2874262375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd819f6c0 con 0x7fddd81029a0 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.340+0000 7fdddcc76640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddd81017b0 0x7fddd8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.341+0000 7fddd7fff640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd81029a0 0x7fddd8197fe0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fddc400d770 tx=0x7fddc400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.341+0000 7fddd5ffb640 1 -- 172.21.15.27:0/2874262375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4017070 con 0x7fddd81029a0 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.341+0000 7fddd5ffb640 1 -- 172.21.15.27:0/2874262375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddc4005500 con 0x7fddd81029a0 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.341+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd819f9b0 con 0x7fddd81029a0 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.341+0000 7fddd5ffb640 1 -- 172.21.15.27:0/2874262375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4004e00 con 0x7fddd81029a0 2024-04-25T01:22:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.341+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd819ff00 con 0x7fddd81029a0 2024-04-25T01:22:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.342+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd8108ba0 con 0x7fddd81029a0 2024-04-25T01:22:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.343+0000 7fddd5ffb640 1 -- 172.21.15.27:0/2874262375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fddc4004fa0 con 0x7fddd81029a0 2024-04-25T01:22:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.343+0000 7fddd5ffb640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fddc80a2c10 0x7fddc80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.343+0000 7fdddcc76640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fddc80a2c10 0x7fddc80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.344+0000 7fddd5ffb640 1 -- 172.21.15.27:0/2874262375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fddc40ce570 con 0x7fddd81029a0 2024-04-25T01:22:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.344+0000 7fdddcc76640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fddc80a2c10 0x7fddc80a50d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fddd8102800 tx=0x7fddcc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.348+0000 7fddd5ffb640 1 -- 172.21.15.27:0/2874262375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fddc409b970 con 0x7fddd81029a0 2024-04-25T01:22:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.585+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddd802d170 con 0x7fddd81029a0 2024-04-25T01:22:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.586+0000 7fddd5ffb640 1 -- 172.21.15.27:0/2874262375 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7fddc409b270 con 0x7fddd81029a0 2024-04-25T01:22:44.568 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:44.568 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:22:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 10 2024-04-25T01:22:44.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fddc80a2c10 msgr2=0x7fddc80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fddc80a2c10 0x7fddc80a50d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fddd8102800 tx=0x7fddcc005f90 comp rx=0 tx=0).stop 2024-04-25T01:22:44.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd81029a0 msgr2=0x7fddd8197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd81029a0 0x7fddd8197fe0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fddc400d770 tx=0x7fddc400dc30 comp rx=0 tx=0).stop 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 shutdown_connections 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fddc80a2c10 0x7fddc80a50d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddd8103360 0x7fddd819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd81029a0 0x7fddd8197fe0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 --2- 172.21.15.27:0/2874262375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fddd81017b0 0x7fddd8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.588+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 >> 172.21.15.27:0/2874262375 conn(0x7fddd80fcd40 msgr2=0x7fddd8105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.589+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 shutdown_connections 2024-04-25T01:22:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:44.589+0000 7fdddeeda640 1 -- 172.21.15.27:0/2874262375 wait complete. 2024-04-25T01:22:44.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:44.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:44.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:44.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:44.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: Deploying daemon mds.g on smithi027 2024-04-25T01:22:44.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] up:boot 2024-04-25T01:22:44.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: fsmap 9 up:standby 2024-04-25T01:22:44.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:44 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:22:44.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:44.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:44.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:44.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:44.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: Deploying daemon mds.g on smithi027 2024-04-25T01:22:44.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] up:boot 2024-04-25T01:22:44.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: fsmap 9 up:standby 2024-04-25T01:22:44.778 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:44 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:22:44.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:44.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:44.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:44.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:44.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:44.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: Deploying daemon mds.g on smithi027 2024-04-25T01:22:44.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] up:boot 2024-04-25T01:22:44.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: fsmap 9 up:standby 2024-04-25T01:22:44.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:44 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:22:45.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:45 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2874262375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:45.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:45 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2874262375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:45.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:45 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2874262375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:46.446 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 01:22:46 smithi027 systemd[1]: Started Ceph mds.g for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:46.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 -- 172.21.15.27:0/3977892670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4741009b0 msgr2=0x7fc474100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/3977892670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4741009b0 0x7fc474100e10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc470009900 tx=0x7fc47002a410 comp rx=0 tx=0).stop 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 -- 172.21.15.27:0/3977892670 shutdown_connections 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/3977892670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc474101350 0x7fc4741017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/3977892670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4741009b0 0x7fc474100e10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/3977892670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474100050 0x7fc474100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 -- 172.21.15.27:0/3977892670 >> 172.21.15.27:0/3977892670 conn(0x7fc4740fbbf0 msgr2=0x7fc4740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 -- 172.21.15.27:0/3977892670 shutdown_connections 2024-04-25T01:22:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.706+0000 7fc47c1b0640 1 -- 172.21.15.27:0/3977892670 wait complete. 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47c1b0640 1 Processor -- start 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47c1b0640 1 -- start start 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47c1b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474100050 0x7fc474199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47c1b0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4741009b0 0x7fc47419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47c1b0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474101350 0x7fc4741a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47c1b0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc47410f640 con 0x7fc474100050 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47c1b0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc47410f4c0 con 0x7fc474101350 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47c1b0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc47410f7c0 con 0x7fc4741009b0 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc479f4c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474100050 0x7fc474199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc479f4c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474100050 0x7fc474199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52300/0 (socket says 172.21.15.27:52300) 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc479f4c640 1 -- 172.21.15.27:0/656689811 learned_addr learned my addr 172.21.15.27:0/656689811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:46.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47974b640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4741009b0 0x7fc47419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc479f4c640 1 -- 172.21.15.27:0/656689811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4741009b0 msgr2=0x7fc47419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47a74d640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474101350 0x7fc4741a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc479f4c640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4741009b0 0x7fc47419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc479f4c640 1 -- 172.21.15.27:0/656689811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474101350 msgr2=0x7fc4741a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc479f4c640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474101350 0x7fc4741a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc479f4c640 1 -- 172.21.15.27:0/656689811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4741a18f0 con 0x7fc474100050 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.707+0000 7fc47974b640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4741009b0 0x7fc47419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.708+0000 7fc479f4c640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474100050 0x7fc474199cb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc46400d950 tx=0x7fc46400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.708+0000 7fc46affd640 1 -- 172.21.15.27:0/656689811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc464017030 con 0x7fc474100050 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.708+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4741a1be0 con 0x7fc474100050 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.708+0000 7fc46affd640 1 -- 172.21.15.27:0/656689811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc464004020 con 0x7fc474100050 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.708+0000 7fc46affd640 1 -- 172.21.15.27:0/656689811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4640047a0 con 0x7fc474100050 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.708+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4741a2100 con 0x7fc474100050 2024-04-25T01:22:46.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.709+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc474068590 con 0x7fc474100050 2024-04-25T01:22:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.710+0000 7fc46affd640 1 -- 172.21.15.27:0/656689811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc4640041d0 con 0x7fc474100050 2024-04-25T01:22:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.711+0000 7fc46affd640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc4600a2b90 0x7fc4600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.711+0000 7fc47974b640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc4600a2b90 0x7fc4600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.711+0000 7fc46affd640 1 -- 172.21.15.27:0/656689811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc4640ce770 con 0x7fc474100050 2024-04-25T01:22:46.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.712+0000 7fc47974b640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc4600a2b90 0x7fc4600a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc470000c00 tx=0x7fc4700095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.715+0000 7fc46affd640 1 -- 172.21.15.27:0/656689811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc474068590 con 0x7fc474100050 2024-04-25T01:22:46.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:46 smithi153 ceph-mon[32621]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:46.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:46.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:46.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:46 smithi139 ceph-mon[32760]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:46.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:46.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:46.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.941+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc474062370 con 0x7fc474100050 2024-04-25T01:22:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.941+0000 7fc46affd640 1 -- 172.21.15.27:0/656689811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7fc46409bad0 con 0x7fc474100050 2024-04-25T01:22:46.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:46.923 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:22:46.924 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 11 2024-04-25T01:22:46.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc4600a2b90 msgr2=0x7fc4600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc4600a2b90 0x7fc4600a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc470000c00 tx=0x7fc4700095f0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474100050 msgr2=0x7fc474199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474100050 0x7fc474199cb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc46400d950 tx=0x7fc46400de10 comp rx=0 tx=0).stop 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 shutdown_connections 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc4600a2b90 0x7fc4600a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc474101350 0x7fc4741a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4741009b0 0x7fc47419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 --2- 172.21.15.27:0/656689811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474100050 0x7fc474199cb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 >> 172.21.15.27:0/656689811 conn(0x7fc4740fbbf0 msgr2=0x7fc4740fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 shutdown_connections 2024-04-25T01:22:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:46.944+0000 7fc47c1b0640 1 -- 172.21.15.27:0/656689811 wait complete. 2024-04-25T01:22:46.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:46 smithi027 ceph-mon[29197]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:46.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:46.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:47.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:47.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:47.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.27:6838/1204805980,v1:172.21.15.27:6839/1204805980] up:boot 2024-04-25T01:22:47.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: fsmap 10 up:standby 2024-04-25T01:22:47.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:22:47.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:47.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:47.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: Deploying daemon mds.a on smithi027 2024-04-25T01:22:47.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:47 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/656689811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.27:6838/1204805980,v1:172.21.15.27:6839/1204805980] up:boot 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: fsmap 10 up:standby 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: Deploying daemon mds.a on smithi027 2024-04-25T01:22:47.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:47 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/656689811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:47.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:47.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:47.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.27:6838/1204805980,v1:172.21.15.27:6839/1204805980] up:boot 2024-04-25T01:22:47.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: fsmap 10 up:standby 2024-04-25T01:22:47.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:22:47.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:47.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:47.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: Deploying daemon mds.a on smithi027 2024-04-25T01:22:47.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:47 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/656689811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:48.575 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:48 smithi027 ceph-mon[29197]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:48.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:48 smithi153 ceph-mon[32621]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:48.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:48 smithi139 ceph-mon[32760]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:48.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:48.946 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:48 smithi027 systemd[1]: Starting Ceph mds.a for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:49.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 -- 172.21.15.27:0/1054878236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401029a0 msgr2=0x7f6140102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 --2- 172.21.15.27:0/1054878236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401029a0 0x7f6140102e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6134009900 tx=0x7f613402a410 comp rx=0 tx=0).stop 2024-04-25T01:22:49.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 -- 172.21.15.27:0/1054878236 shutdown_connections 2024-04-25T01:22:49.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 --2- 172.21.15.27:0/1054878236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140103360 0x7f6140109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 --2- 172.21.15.27:0/1054878236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401029a0 0x7f6140102e20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 --2- 172.21.15.27:0/1054878236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61401017b0 0x7f6140101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 -- 172.21.15.27:0/1054878236 >> 172.21.15.27:0/1054878236 conn(0x7f61400fcd40 msgr2=0x7f61400ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 -- 172.21.15.27:0/1054878236 shutdown_connections 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.063+0000 7f6145f22640 1 -- 172.21.15.27:0/1054878236 wait complete. 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f6145f22640 1 Processor -- start 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f6145f22640 1 -- start start 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f6145f22640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401017b0 0x7f6140197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f6145f22640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61401029a0 0x7f6140197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f6145f22640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140103360 0x7f614019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f6145f22640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f614010d480 con 0x7f61401017b0 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f6145f22640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f614010d300 con 0x7f61401029a0 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f6145f22640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f614010d600 con 0x7f6140103360 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401017b0 0x7f6140197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401017b0 0x7f6140197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54744/0 (socket says 172.21.15.27:54744) 2024-04-25T01:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613f7fe640 1 -- 172.21.15.27:0/4033184012 learned_addr learned my addr 172.21.15.27:0/4033184012 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613effd640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61401029a0 0x7f6140197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613ffff640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140103360 0x7f614019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613f7fe640 1 -- 172.21.15.27:0/4033184012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140103360 msgr2=0x7f614019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613f7fe640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140103360 0x7f614019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613f7fe640 1 -- 172.21.15.27:0/4033184012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61401029a0 msgr2=0x7f6140197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613f7fe640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61401029a0 0x7f6140197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613f7fe640 1 -- 172.21.15.27:0/4033184012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614019f6c0 con 0x7f61401017b0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613ffff640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140103360 0x7f614019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.064+0000 7f613effd640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61401029a0 0x7f6140197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.065+0000 7f613f7fe640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401017b0 0x7f6140197aa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f613000d770 tx=0x7f613000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.065+0000 7f613cff9640 1 -- 172.21.15.27:0/4033184012 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6130017070 con 0x7f61401017b0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.065+0000 7f613cff9640 1 -- 172.21.15.27:0/4033184012 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6130009c30 con 0x7f61401017b0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.065+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614019f9b0 con 0x7f61401017b0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.065+0000 7f613cff9640 1 -- 172.21.15.27:0/4033184012 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6130003dd0 con 0x7f61401017b0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.065+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f614019ff20 con 0x7f61401017b0 2024-04-25T01:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.066+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61400689b0 con 0x7f61401017b0 2024-04-25T01:22:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.067+0000 7f613cff9640 1 -- 172.21.15.27:0/4033184012 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6130003fb0 con 0x7f61401017b0 2024-04-25T01:22:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.067+0000 7f613cff9640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f61280a2c10 0x7f61280a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.068+0000 7f613effd640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f61280a2c10 0x7f61280a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.068+0000 7f613cff9640 1 -- 172.21.15.27:0/4033184012 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f61300ce5e0 con 0x7f61401017b0 2024-04-25T01:22:49.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.068+0000 7f613effd640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f61280a2c10 0x7f61280a50d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6134006010 tx=0x7f6134000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:49.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.072+0000 7f613cff9640 1 -- 172.21.15.27:0/4033184012 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6130009de0 con 0x7f61401017b0 2024-04-25T01:22:49.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.298+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6140108ad0 con 0x7f61401017b0 2024-04-25T01:22:49.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.298+0000 7f613cff9640 1 -- 172.21.15.27:0/4033184012 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7f613009b9e0 con 0x7f61401017b0 2024-04-25T01:22:49.295 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:49.295 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:22:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 11 2024-04-25T01:22:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.300+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f61280a2c10 msgr2=0x7f61280a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.300+0000 7f6145f22640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f61280a2c10 0x7f61280a50d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6134006010 tx=0x7f6134000bc0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.300+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401017b0 msgr2=0x7f6140197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.300+0000 7f6145f22640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401017b0 0x7f6140197aa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f613000d770 tx=0x7f613000dc30 comp rx=0 tx=0).stop 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.301+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 shutdown_connections 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.301+0000 7f6145f22640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f61280a2c10 0x7f61280a50d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.301+0000 7f6145f22640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140103360 0x7f614019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.301+0000 7f6145f22640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61401029a0 0x7f6140197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.301+0000 7f6145f22640 1 --2- 172.21.15.27:0/4033184012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61401017b0 0x7f6140197aa0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.301+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 >> 172.21.15.27:0/4033184012 conn(0x7f61400fcd40 msgr2=0x7f6140105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.301+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 shutdown_connections 2024-04-25T01:22:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:49.301+0000 7f6145f22640 1 -- 172.21.15.27:0/4033184012 wait complete. 2024-04-25T01:22:49.297 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:49 smithi027 podman[70908]: 2024-04-25 01:22:48.911639134 +0000 UTC m=+0.018584246 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:49.298 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:49 smithi027 podman[70908]: 2024-04-25 01:22:49.208972574 +0000 UTC m=+0.315917676 container create ff31018298463f9cc3b50c0a3fa27d5e1af3ec23b3d61cef4a3b1c2703f45ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:22:49.551 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:49 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4033184012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:49.551 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:49 smithi027 podman[70908]: 2024-04-25 01:22:49.453436493 +0000 UTC m=+0.560381597 container init ff31018298463f9cc3b50c0a3fa27d5e1af3ec23b3d61cef4a3b1c2703f45ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:22:49.551 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:49 smithi027 podman[70908]: 2024-04-25 01:22:49.45610664 +0000 UTC m=+0.563051752 container start ff31018298463f9cc3b50c0a3fa27d5e1af3ec23b3d61cef4a3b1c2703f45ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T01:22:49.551 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:49 smithi027 ceph-mds[70968]: -- 172.21.15.27:0/3839118014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e964f63a40 con 0x55e964f36c00 2024-04-25T01:22:49.551 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:49 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a[70964]: starting mds.a at 2024-04-25T01:22:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:49 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4033184012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:49.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:49 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4033184012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:49.946 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:49 smithi027 bash[70908]: ff31018298463f9cc3b50c0a3fa27d5e1af3ec23b3d61cef4a3b1c2703f45ddb 2024-04-25T01:22:49.946 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:22:49 smithi027 systemd[1]: Started Ceph mds.a for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:50.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.27:6840/1912988733,v1:172.21.15.27:6841/1912988733] up:boot 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: fsmap 11 up:standby 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:50.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:50 smithi153 ceph-mon[32621]: Deploying daemon mds.b on smithi139 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.27:6840/1912988733,v1:172.21.15.27:6841/1912988733] up:boot 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: fsmap 11 up:standby 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:50.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:50.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:50.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:50 smithi139 ceph-mon[32760]: Deploying daemon mds.b on smithi139 2024-04-25T01:22:50.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.27:6840/1912988733,v1:172.21.15.27:6841/1912988733] up:boot 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: fsmap 11 up:standby 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:50.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:50 smithi027 ceph-mon[29197]: Deploying daemon mds.b on smithi139 2024-04-25T01:22:51.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 -- 172.21.15.27:0/2284077774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3501006a0 msgr2=0x7fe350100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 --2- 172.21.15.27:0/2284077774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3501006a0 0x7fe350100ac0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009900 tx=0x7fe34c02a410 comp rx=0 tx=0).stop 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 -- 172.21.15.27:0/2284077774 shutdown_connections 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 --2- 172.21.15.27:0/2284077774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe350102250 0x7fe350108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 --2- 172.21.15.27:0/2284077774 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe350101890 0x7fe350101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 --2- 172.21.15.27:0/2284077774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3501006a0 0x7fe350100ac0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 -- 172.21.15.27:0/2284077774 >> 172.21.15.27:0/2284077774 conn(0x7fe3500fbc30 msgr2=0x7fe3500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 -- 172.21.15.27:0/2284077774 shutdown_connections 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.434+0000 7fe357e38640 1 -- 172.21.15.27:0/2284077774 wait complete. 2024-04-25T01:22:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe357e38640 1 Processor -- start 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe357e38640 1 -- start start 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe357e38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3501006a0 0x7fe350199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe357e38640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe350101890 0x7fe35019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe357e38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe350102250 0x7fe3501a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe357e38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe35010c370 con 0x7fe350102250 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe357e38640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe35010c1f0 con 0x7fe350101890 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe357e38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe35010c4f0 con 0x7fe3501006a0 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3563d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe350102250 0x7fe3501a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3563d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe350102250 0x7fe3501a1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54796/0 (socket says 172.21.15.27:54796) 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3563d5640 1 -- 172.21.15.27:0/1735493975 learned_addr learned my addr 172.21.15.27:0/1735493975 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe355bd4640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3501006a0 0x7fe350199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3553d3640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe350101890 0x7fe35019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3563d5640 1 -- 172.21.15.27:0/1735493975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3501006a0 msgr2=0x7fe350199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3563d5640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3501006a0 0x7fe350199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3563d5640 1 -- 172.21.15.27:0/1735493975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe350101890 msgr2=0x7fe35019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3563d5640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe350101890 0x7fe35019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.435+0000 7fe3563d5640 1 -- 172.21.15.27:0/1735493975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3501a1910 con 0x7fe350102250 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.436+0000 7fe3563d5640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe350102250 0x7fe3501a1270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe34800cad0 tx=0x7fe34800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.436+0000 7fe346ffd640 1 -- 172.21.15.27:0/1735493975 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34800ae20 con 0x7fe350102250 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.436+0000 7fe346ffd640 1 -- 172.21.15.27:0/1735493975 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe348005020 con 0x7fe350102250 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.436+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3501a1c00 con 0x7fe350102250 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.436+0000 7fe346ffd640 1 -- 172.21.15.27:0/1735493975 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34800f6f0 con 0x7fe350102250 2024-04-25T01:22:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.436+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3501a2170 con 0x7fe350102250 2024-04-25T01:22:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.437+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3501a23a0 con 0x7fe350102250 2024-04-25T01:22:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.438+0000 7fe346ffd640 1 -- 172.21.15.27:0/1735493975 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe3480051d0 con 0x7fe350102250 2024-04-25T01:22:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.439+0000 7fe346ffd640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe33c0a2c10 0x7fe33c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.439+0000 7fe355bd4640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe33c0a2c10 0x7fe33c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.439+0000 7fe346ffd640 1 -- 172.21.15.27:0/1735493975 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe3480cee50 con 0x7fe350102250 2024-04-25T01:22:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.439+0000 7fe355bd4640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe33c0a2c10 0x7fe33c0a50d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe34c02aa00 tx=0x7fe34c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:51.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.444+0000 7fe346ffd640 1 -- 172.21.15.27:0/1735493975 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe348014030 con 0x7fe350102250 2024-04-25T01:22:51.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.671+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3501a2570 con 0x7fe350102250 2024-04-25T01:22:51.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.671+0000 7fe346ffd640 1 -- 172.21.15.27:0/1735493975 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7fe34809bc00 con 0x7fe350102250 2024-04-25T01:22:51.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:51.653 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:22:51.653 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 12 2024-04-25T01:22:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.673+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe33c0a2c10 msgr2=0x7fe33c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.673+0000 7fe357e38640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe33c0a2c10 0x7fe33c0a50d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe34c02aa00 tx=0x7fe34c0315b0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.673+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe350102250 msgr2=0x7fe3501a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.673+0000 7fe357e38640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe350102250 0x7fe3501a1270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe34800cad0 tx=0x7fe34800cf90 comp rx=0 tx=0).stop 2024-04-25T01:22:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.674+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 shutdown_connections 2024-04-25T01:22:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.674+0000 7fe357e38640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe33c0a2c10 0x7fe33c0a50d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.674+0000 7fe357e38640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe350102250 0x7fe3501a1270 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.674+0000 7fe357e38640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe350101890 0x7fe35019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.674+0000 7fe357e38640 1 --2- 172.21.15.27:0/1735493975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3501006a0 0x7fe350199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.674+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 >> 172.21.15.27:0/1735493975 conn(0x7fe3500fbc30 msgr2=0x7fe3500fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.674+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 shutdown_connections 2024-04-25T01:22:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:51.674+0000 7fe357e38640 1 -- 172.21.15.27:0/1735493975 wait complete. 2024-04-25T01:22:52.043 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:51 smithi139 systemd[1]: Starting Ceph mds.b for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:22:52.377 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:52 smithi139 podman[63089]: 2024-04-25 01:22:52.031301526 +0000 UTC m=+0.018121516 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:22:52.377 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:52 smithi139 podman[63089]: 2024-04-25 01:22:52.144822977 +0000 UTC m=+0.131642953 container create ffb82643ef8d98b16d1c8b6cb78a688ec9f52505b91bb94dda1bdc99ca1b7ddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T01:22:52.378 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:52 smithi139 podman[63089]: 2024-04-25 01:22:52.397411799 +0000 UTC m=+0.384231772 container init ffb82643ef8d98b16d1c8b6cb78a688ec9f52505b91bb94dda1bdc99ca1b7ddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:22:52.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:52 smithi153 ceph-mon[32621]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:52.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:52 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1735493975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:52.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:52 smithi139 ceph-mon[32760]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:52.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:52 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1735493975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:52.793 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:52 smithi139 podman[63089]: 2024-04-25 01:22:52.400209366 +0000 UTC m=+0.387029348 container start ffb82643ef8d98b16d1c8b6cb78a688ec9f52505b91bb94dda1bdc99ca1b7ddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T01:22:52.793 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:52 smithi139 ceph-mds[63108]: -- 172.21.15.139:0/1876689766 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c6de7a5a40 con 0x55c6de779000 2024-04-25T01:22:52.793 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:52 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b[63104]: starting mds.b at 2024-04-25T01:22:52.793 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:52 smithi139 bash[63089]: ffb82643ef8d98b16d1c8b6cb78a688ec9f52505b91bb94dda1bdc99ca1b7ddf 2024-04-25T01:22:52.793 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:22:52 smithi139 systemd[1]: Started Ceph mds.b for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:22:52.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:52 smithi027 ceph-mon[29197]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:52.947 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:52 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1735493975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:53.666 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:22:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.791+0000 7fbd59542640 1 -- 172.21.15.27:0/1628265928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd54103360 msgr2=0x7fbd54109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.791+0000 7fbd59542640 1 --2- 172.21.15.27:0/1628265928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd54103360 0x7fbd54109bf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009ca0 tx=0x7fbd4402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 -- 172.21.15.27:0/1628265928 shutdown_connections 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 --2- 172.21.15.27:0/1628265928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd54103360 0x7fbd54109bf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 --2- 172.21.15.27:0/1628265928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd541029a0 0x7fbd54102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 --2- 172.21.15.27:0/1628265928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd541017b0 0x7fbd54101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 -- 172.21.15.27:0/1628265928 >> 172.21.15.27:0/1628265928 conn(0x7fbd540fcd40 msgr2=0x7fbd540ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 -- 172.21.15.27:0/1628265928 shutdown_connections 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 -- 172.21.15.27:0/1628265928 wait complete. 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 Processor -- start 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 -- start start 2024-04-25T01:22:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd541017b0 0x7fbd540754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd541029a0 0x7fbd54073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd54103360 0x7fbd54074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5410d4b0 con 0x7fbd541029a0 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5410d330 con 0x7fbd54103360 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.792+0000 7fbd59542640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5410d630 con 0x7fbd541017b0 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd541029a0 0x7fbd54073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd541029a0 0x7fbd54073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54804/0 (socket says 172.21.15.27:54804) 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 -- 172.21.15.27:0/741283171 learned_addr learned my addr 172.21.15.27:0/741283171 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd52ffd640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd541017b0 0x7fbd540754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd537fe640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd54103360 0x7fbd54074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 -- 172.21.15.27:0/741283171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd541017b0 msgr2=0x7fbd540754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd541017b0 0x7fbd540754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 -- 172.21.15.27:0/741283171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd54103360 msgr2=0x7fbd54074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd54103360 0x7fbd54074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 -- 172.21.15.27:0/741283171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd54074900 con 0x7fbd541029a0 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd52ffd640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd541017b0 0x7fbd540754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd537fe640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd54103360 0x7fbd54074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd527fc640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd541029a0 0x7fbd54073b30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800b4a0 tx=0x7fbd4800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd3bfff640 1 -- 172.21.15.27:0/741283171 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48014070 con 0x7fbd541029a0 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd541a1c80 con 0x7fbd541029a0 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd3bfff640 1 -- 172.21.15.27:0/741283171 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd48010af0 con 0x7fbd541029a0 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd3bfff640 1 -- 172.21.15.27:0/741283171 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48017d10 con 0x7fbd541029a0 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.793+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd541a21d0 con 0x7fbd541029a0 2024-04-25T01:22:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.794+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd54108a70 con 0x7fbd541029a0 2024-04-25T01:22:53.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] up:boot 2024-04-25T01:22:53.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: fsmap 12 up:standby 2024-04-25T01:22:53.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:22:53.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.780 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:53 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:53.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.795+0000 7fbd3bfff640 1 -- 172.21.15.27:0/741283171 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbd48017070 con 0x7fbd541029a0 2024-04-25T01:22:53.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.796+0000 7fbd3bfff640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:53.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.796+0000 7fbd52ffd640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:53.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.796+0000 7fbd3bfff640 1 -- 172.21.15.27:0/741283171 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbd480cdff0 con 0x7fbd541029a0 2024-04-25T01:22:53.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.796+0000 7fbd52ffd640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbd54102800 tx=0x7fbd4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:53.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:53.800+0000 7fbd3bfff640 1 -- 172.21.15.27:0/741283171 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd4809b3f0 con 0x7fbd541029a0 2024-04-25T01:22:53.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] up:boot 2024-04-25T01:22:53.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: fsmap 12 up:standby 2024-04-25T01:22:53.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:22:53.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:53 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:53.838 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] up:boot 2024-04-25T01:22:53.838 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: fsmap 12 up:standby 2024-04-25T01:22:53.838 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:22:53.838 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:53.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:53 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.026+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd54062340 con 0x7fbd541029a0 2024-04-25T01:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.027+0000 7fbd3bfff640 1 -- 172.21.15.27:0/741283171 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7fbd4809acf0 con 0x7fbd541029a0 2024-04-25T01:22:54.008 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:54.008 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24521,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T01:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 13 2024-04-25T01:22:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd3c0a2b90 msgr2=0x7fbd3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbd54102800 tx=0x7fbd4c005f90 comp rx=0 tx=0).stop 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd541029a0 msgr2=0x7fbd54073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd541029a0 0x7fbd54073b30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800b4a0 tx=0x7fbd4800b960 comp rx=0 tx=0).stop 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 shutdown_connections 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd54103360 0x7fbd54074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd541029a0 0x7fbd54073b30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 --2- 172.21.15.27:0/741283171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd541017b0 0x7fbd540754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 >> 172.21.15.27:0/741283171 conn(0x7fbd540fcd40 msgr2=0x7fbd54105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 shutdown_connections 2024-04-25T01:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.029+0000 7fbd59542640 1 -- 172.21.15.27:0/741283171 wait complete. 2024-04-25T01:22:54.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:22:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 -- 172.21.15.27:0/3641584835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 msgr2=0x7f9df8100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 --2- 172.21.15.27:0/3641584835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 0x7f9df8100ab0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009900 tx=0x7f9dec02a410 comp rx=0 tx=0).stop 2024-04-25T01:22:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 -- 172.21.15.27:0/3641584835 shutdown_connections 2024-04-25T01:22:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 --2- 172.21.15.27:0/3641584835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9df8102240 0x7f9df8108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 --2- 172.21.15.27:0/3641584835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df8101880 0x7f9df8101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 --2- 172.21.15.27:0/3641584835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 0x7f9df8100ab0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 -- 172.21.15.27:0/3641584835 >> 172.21.15.27:0/3641584835 conn(0x7f9df80fbc40 msgr2=0x7f9df80fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 -- 172.21.15.27:0/3641584835 shutdown_connections 2024-04-25T01:22:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.152+0000 7f9dfce88640 1 -- 172.21.15.27:0/3641584835 wait complete. 2024-04-25T01:22:54.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9dfce88640 1 Processor -- start 2024-04-25T01:22:54.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9dfce88640 1 -- start start 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9dfce88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 0x7f9df8197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9dfce88640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df8101880 0x7f9df8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9dfce88640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9df8102240 0x7f9df819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9dfce88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9df810c310 con 0x7f9df8100690 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9dfce88640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9df810c190 con 0x7f9df8102240 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9dfce88640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9df810c490 con 0x7f9df8101880 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9df659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 0x7f9df8197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9df659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 0x7f9df8197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54822/0 (socket says 172.21.15.27:54822) 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9df659c640 1 -- 172.21.15.27:0/2606897257 learned_addr learned my addr 172.21.15.27:0/2606897257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9df5d9b640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df8101880 0x7f9df8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.153+0000 7f9df6d9d640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9df8102240 0x7f9df819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9df659c640 1 -- 172.21.15.27:0/2606897257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df8101880 msgr2=0x7f9df8198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9df659c640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df8101880 0x7f9df8198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9df659c640 1 -- 172.21.15.27:0/2606897257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9df8102240 msgr2=0x7f9df819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9df659c640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9df8102240 0x7f9df819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9df659c640 1 -- 172.21.15.27:0/2606897257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df819f6e0 con 0x7f9df8100690 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9df659c640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 0x7f9df8197b30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9dec031610 tx=0x7f9dec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9de77fe640 1 -- 172.21.15.27:0/2606897257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec03f030 con 0x7f9df8100690 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df819f970 con 0x7f9df8100690 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9de77fe640 1 -- 172.21.15.27:0/2606897257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dec003ad0 con 0x7f9df8100690 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9de77fe640 1 -- 172.21.15.27:0/2606897257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec035070 con 0x7f9df8100690 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.154+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df819feb0 con 0x7f9df8100690 2024-04-25T01:22:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.155+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df81a0080 con 0x7f9df8100690 2024-04-25T01:22:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.156+0000 7f9de77fe640 1 -- 172.21.15.27:0/2606897257 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9dec042030 con 0x7f9df8100690 2024-04-25T01:22:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.157+0000 7f9de77fe640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9de00a2b90 0x7f9de00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.157+0000 7f9df5d9b640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9de00a2b90 0x7f9de00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.157+0000 7f9de77fe640 1 -- 172.21.15.27:0/2606897257 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9dec0ef400 con 0x7f9df8100690 2024-04-25T01:22:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.158+0000 7f9df5d9b640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9de00a2b90 0x7f9de00a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9df8198fd0 tx=0x7f9de8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.161+0000 7f9de77fe640 1 -- 172.21.15.27:0/2606897257 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9dec0bc1b0 con 0x7f9df8100690 2024-04-25T01:22:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.354+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9df81a0250 con 0x7f9df8100690 2024-04-25T01:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.355+0000 7f9de77fe640 1 -- 172.21.15.27:0/2606897257 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18452 (secure 0 0 0) 0x7f9dec0c04b0 con 0x7f9df8100690 2024-04-25T01:22:54.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:54.337 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":81,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:22:05.817831+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:22:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.357+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9de00a2b90 msgr2=0x7f9de00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.357+0000 7f9dfce88640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9de00a2b90 0x7f9de00a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9df8198fd0 tx=0x7f9de8005f90 comp rx=0 tx=0).stop 2024-04-25T01:22:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.357+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 msgr2=0x7f9df8197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.357+0000 7f9dfce88640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 0x7f9df8197b30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9dec031610 tx=0x7f9dec004270 comp rx=0 tx=0).stop 2024-04-25T01:22:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.357+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 shutdown_connections 2024-04-25T01:22:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.358+0000 7f9dfce88640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9de00a2b90 0x7f9de00a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.358+0000 7f9dfce88640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9df8102240 0x7f9df819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.358+0000 7f9dfce88640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df8101880 0x7f9df8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.358+0000 7f9dfce88640 1 --2- 172.21.15.27:0/2606897257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df8100690 0x7f9df8197b30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.358+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 >> 172.21.15.27:0/2606897257 conn(0x7f9df80fbc40 msgr2=0x7f9df80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.358+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 shutdown_connections 2024-04-25T01:22:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.358+0000 7f9dfce88640 1 -- 172.21.15.27:0/2606897257 wait complete. 2024-04-25T01:22:54.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:22:54.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:22:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 -- 172.21.15.27:0/2665103255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0102250 msgr2=0x7facc0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 --2- 172.21.15.27:0/2665103255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0102250 0x7facc0108ae0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7facb8009ca0 tx=0x7facb802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 -- 172.21.15.27:0/2665103255 shutdown_connections 2024-04-25T01:22:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 --2- 172.21.15.27:0/2665103255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0102250 0x7facc0108ae0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 --2- 172.21.15.27:0/2665103255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0101890 0x7facc0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 --2- 172.21.15.27:0/2665103255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facc01006a0 0x7facc0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 -- 172.21.15.27:0/2665103255 >> 172.21.15.27:0/2665103255 conn(0x7facc00fbc30 msgr2=0x7facc00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 -- 172.21.15.27:0/2665103255 shutdown_connections 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.481+0000 7facc5445640 1 -- 172.21.15.27:0/2665103255 wait complete. 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facc5445640 1 Processor -- start 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facc5445640 1 -- start start 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facc5445640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc01006a0 0x7facc0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facc5445640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0101890 0x7facc0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facc5445640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facc0102250 0x7facc019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facc5445640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7facc010d440 con 0x7facc0101890 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facc5445640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7facc010d2c0 con 0x7facc0102250 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facc5445640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7facc010d5c0 con 0x7facc01006a0 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facbe7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0101890 0x7facc0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facbe7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0101890 0x7facc0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54844/0 (socket says 172.21.15.27:54844) 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facbe7fc640 1 -- 172.21.15.27:0/2504172437 learned_addr learned my addr 172.21.15.27:0/2504172437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:54.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.482+0000 7facbf7fe640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facc0102250 0x7facc019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facbeffd640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc01006a0 0x7facc0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facbf7fe640 1 -- 172.21.15.27:0/2504172437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc01006a0 msgr2=0x7facc0197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facbf7fe640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc01006a0 0x7facc0197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facbf7fe640 1 -- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0101890 msgr2=0x7facc0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facbf7fe640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0101890 0x7facc0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facbf7fe640 1 -- 172.21.15.27:0/2504172437 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc019f6b0 con 0x7facc0102250 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facbe7fc640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0101890 0x7facc0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facbf7fe640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facc0102250 0x7facc019f040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7facb802aeb0 tx=0x7facb8003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7faca7fff640 1 -- 172.21.15.27:0/2504172437 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb8031650 con 0x7facc0102250 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc019f940 con 0x7facc0102250 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7faca7fff640 1 -- 172.21.15.27:0/2504172437 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facb80317f0 con 0x7facc0102250 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7faca7fff640 1 -- 172.21.15.27:0/2504172437 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb802fad0 con 0x7facc0102250 2024-04-25T01:22:54.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.483+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc019fe90 con 0x7facc0102250 2024-04-25T01:22:54.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.484+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc01079c0 con 0x7facc0102250 2024-04-25T01:22:54.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.485+0000 7faca7fff640 1 -- 172.21.15.27:0/2504172437 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7facb80045b0 con 0x7facc0102250 2024-04-25T01:22:54.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.486+0000 7faca7fff640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faca80a2b90 0x7faca80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.486+0000 7faca7fff640 1 -- 172.21.15.27:0/2504172437 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7facb80f01c0 con 0x7facc0102250 2024-04-25T01:22:54.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.486+0000 7facbeffd640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faca80a2b90 0x7faca80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.487+0000 7facbeffd640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faca80a2b90 0x7faca80a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7facc01016f0 tx=0x7facb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.490+0000 7faca7fff640 1 -- 172.21.15.27:0/2504172437 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7facb80bd520 con 0x7facc0102250 2024-04-25T01:22:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.680+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7facc002d300 con 0x7facc0102250 2024-04-25T01:22:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.680+0000 7faca7fff640 1 -- 172.21.15.27:0/2504172437 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7facb80bce20 con 0x7facc0102250 2024-04-25T01:22:54.662 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:22:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faca80a2b90 msgr2=0x7faca80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faca80a2b90 0x7faca80a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7facc01016f0 tx=0x7facb4005f90 comp rx=0 tx=0).stop 2024-04-25T01:22:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facc0102250 msgr2=0x7facc019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facc0102250 0x7facc019f040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7facb802aeb0 tx=0x7facb8003f10 comp rx=0 tx=0).stop 2024-04-25T01:22:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 shutdown_connections 2024-04-25T01:22:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faca80a2b90 0x7faca80a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facc0102250 0x7facc019f040 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facc0101890 0x7facc0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 --2- 172.21.15.27:0/2504172437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc01006a0 0x7facc0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 >> 172.21.15.27:0/2504172437 conn(0x7facc00fbc30 msgr2=0x7facc0104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.682+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 shutdown_connections 2024-04-25T01:22:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.683+0000 7facc5445640 1 -- 172.21.15.27:0/2504172437 wait complete. 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/741283171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2606897257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:54.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:54.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:54 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:54.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:22:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 -- 172.21.15.27:0/3462582854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8102200 msgr2=0x7f1ed81045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/3462582854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8102200 0x7f1ed81045f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009900 tx=0x7f1ec802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 -- 172.21.15.27:0/3462582854 shutdown_connections 2024-04-25T01:22:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/3462582854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed8107460 0x7f1ed81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/3462582854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ed8104b30 0x7f1ed8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/3462582854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8102200 0x7f1ed81045f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 -- 172.21.15.27:0/3462582854 >> 172.21.15.27:0/3462582854 conn(0x7f1ed80fbbf0 msgr2=0x7f1ed80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 -- 172.21.15.27:0/3462582854 shutdown_connections 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.798+0000 7f1edd6a5640 1 -- 172.21.15.27:0/3462582854 wait complete. 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1edd6a5640 1 Processor -- start 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1edd6a5640 1 -- start start 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1edd6a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ed8102200 0x7f1ed8101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1edd6a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed8104b30 0x7f1ed81001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1edd6a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8107460 0x7f1ed81006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1edd6a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed810d420 con 0x7f1ed8107460 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1edd6a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed810d2a0 con 0x7f1ed8102200 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1edd6a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed810d5a0 con 0x7f1ed8104b30 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1ed77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8107460 0x7f1ed81006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1ed6ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ed8102200 0x7f1ed8101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1ecffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed8104b30 0x7f1ed81001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1ed6ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ed8102200 0x7f1ed8101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:33358/0 (socket says 172.21.15.27:33358) 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1ed6ffd640 1 -- 172.21.15.27:0/955234909 learned_addr learned my addr 172.21.15.27:0/955234909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.799+0000 7f1ecffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed8104b30 0x7f1ed81001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:34746/0 (socket says 172.21.15.27:34746) 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ecffff640 1 -- 172.21.15.27:0/955234909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ed8102200 msgr2=0x7f1ed8101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ecffff640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ed8102200 0x7f1ed8101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ecffff640 1 -- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8107460 msgr2=0x7f1ed81006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ecffff640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8107460 0x7f1ed81006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ecffff640 1 -- 172.21.15.27:0/955234909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed8100f80 con 0x7f1ed8104b30 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ed77fe640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8107460 0x7f1ed81006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ecffff640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed8104b30 0x7f1ed81001b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e9c0 tx=0x7f1ec000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ed4ff9640 1 -- 172.21.15.27:0/955234909 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000ccc0 con 0x7f1ed8104b30 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ed4ff9640 1 -- 172.21.15.27:0/955234909 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ec0010910 con 0x7f1ed8104b30 2024-04-25T01:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1ed4ff9640 1 -- 172.21.15.27:0/955234909 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00188c0 con 0x7f1ed8104b30 2024-04-25T01:22:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed81a1bc0 con 0x7f1ed8104b30 2024-04-25T01:22:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.800+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed81a2160 con 0x7f1ed8104b30 2024-04-25T01:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.801+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed80fd6e0 con 0x7f1ed8104b30 2024-04-25T01:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.802+0000 7f1ed4ff9640 1 -- 172.21.15.27:0/955234909 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1ec0010ac0 con 0x7f1ed8104b30 2024-04-25T01:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.803+0000 7f1ed4ff9640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1ec40a2c10 0x7f1ec40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.803+0000 7f1ed6ffd640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1ec40a2c10 0x7f1ec40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.803+0000 7f1ed4ff9640 1 -- 172.21.15.27:0/955234909 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1ec00cf2c0 con 0x7f1ed8104b30 2024-04-25T01:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.804+0000 7f1ed6ffd640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1ec40a2c10 0x7f1ec40a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8031440 tx=0x7f1ec80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.806+0000 7f1ed4ff9640 1 -- 172.21.15.27:0/955234909 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1ec009bfd0 con 0x7f1ed8104b30 2024-04-25T01:22:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.995+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1ed802d3c0 con 0x7f1ed8104b30 2024-04-25T01:22:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.996+0000 7f1ed4ff9640 1 -- 172.21.15.27:0/955234909 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f1ec00a02d0 con 0x7f1ed8104b30 2024-04-25T01:22:54.977 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:22:54.977 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-04-25T01:22:54.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:22:54.978 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-04-25T01:22:54.978 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-04-25T01:22:54.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:22:54.978 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1ec40a2c10 msgr2=0x7f1ec40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1ec40a2c10 0x7f1ec40a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8031440 tx=0x7f1ec80315b0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed8104b30 msgr2=0x7f1ed81001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed8104b30 0x7f1ed81001b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e9c0 tx=0x7f1ec000ee80 comp rx=0 tx=0).stop 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 shutdown_connections 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f1ec40a2c10 0x7f1ec40a50d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ed8107460 0x7f1ed81006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed8104b30 0x7f1ed81001b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 --2- 172.21.15.27:0/955234909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ed8102200 0x7f1ed8101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 >> 172.21.15.27:0/955234909 conn(0x7f1ed80fbbf0 msgr2=0x7f1ed8102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 shutdown_connections 2024-04-25T01:22:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:54.998+0000 7f1edd6a5640 1 -- 172.21.15.27:0/955234909 wait complete. 2024-04-25T01:22:54.990 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:22:54.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:22:55.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/741283171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2606897257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:55.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:54 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:55.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/741283171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:22:55.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2606897257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:22:55.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:55.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:55.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:54 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 -- 172.21.15.27:0/1259939942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea101074c0 msgr2=0x7fea101098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 --2- 172.21.15.27:0/1259939942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea101074c0 0x7fea101098b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fea08009ca0 tx=0x7fea0802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 -- 172.21.15.27:0/1259939942 shutdown_connections 2024-04-25T01:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 --2- 172.21.15.27:0/1259939942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea101074c0 0x7fea101098b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 --2- 172.21.15.27:0/1259939942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea10104b90 0x7fea10106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 --2- 172.21.15.27:0/1259939942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea10102260 0x7fea10104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 -- 172.21.15.27:0/1259939942 >> 172.21.15.27:0/1259939942 conn(0x7fea100fbbf0 msgr2=0x7fea100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 -- 172.21.15.27:0/1259939942 shutdown_connections 2024-04-25T01:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.116+0000 7fea15eb3640 1 -- 172.21.15.27:0/1259939942 wait complete. 2024-04-25T01:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea15eb3640 1 Processor -- start 2024-04-25T01:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea15eb3640 1 -- start start 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea15eb3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea10102260 0x7fea10197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea15eb3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea10104b90 0x7fea10198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea15eb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea101074c0 0x7fea1019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea15eb3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea1010d420 con 0x7fea10102260 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea15eb3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fea1010d2a0 con 0x7fea10104b90 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea15eb3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea1010d5a0 con 0x7fea101074c0 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.117+0000 7fea0f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea10102260 0x7fea10197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea10102260 0x7fea10197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54884/0 (socket says 172.21.15.27:54884) 2024-04-25T01:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0f7fe640 1 -- 172.21.15.27:0/3241933525 learned_addr learned my addr 172.21.15.27:0/3241933525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0ffff640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea101074c0 0x7fea1019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0effd640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea10104b90 0x7fea10198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0f7fe640 1 -- 172.21.15.27:0/3241933525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea101074c0 msgr2=0x7fea1019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0f7fe640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea101074c0 0x7fea1019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0f7fe640 1 -- 172.21.15.27:0/3241933525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea10104b90 msgr2=0x7fea10198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0f7fe640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea10104b90 0x7fea10198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0f7fe640 1 -- 172.21.15.27:0/3241933525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1019f6b0 con 0x7fea10102260 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0f7fe640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea10102260 0x7fea10197b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fea0000e9f0 tx=0x7fea0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0cff9640 1 -- 172.21.15.27:0/3241933525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0000cd80 con 0x7fea10102260 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0cff9640 1 -- 172.21.15.27:0/3241933525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea00005500 con 0x7fea10102260 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1019f9a0 con 0x7fea10102260 2024-04-25T01:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea0cff9640 1 -- 172.21.15.27:0/3241933525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00010690 con 0x7fea10102260 2024-04-25T01:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.118+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1019fea0 con 0x7fea10102260 2024-04-25T01:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.119+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea100fd6e0 con 0x7fea10102260 2024-04-25T01:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.121+0000 7fea0cff9640 1 -- 172.21.15.27:0/3241933525 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fea000046c0 con 0x7fea10102260 2024-04-25T01:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.122+0000 7fea0cff9640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe9f80a2b90 0x7fe9f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.122+0000 7fea0cff9640 1 -- 172.21.15.27:0/3241933525 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fea000cf1b0 con 0x7fea10102260 2024-04-25T01:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.122+0000 7fea0effd640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe9f80a2b90 0x7fe9f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.122+0000 7fea0effd640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe9f80a2b90 0x7fe9f80a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fea10198fa0 tx=0x7fea04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.125+0000 7fea0cff9640 1 -- 172.21.15.27:0/3241933525 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fea00053740 con 0x7fea10102260 2024-04-25T01:22:55.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.325+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fea1002d2f0 con 0x7fea10102260 2024-04-25T01:22:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.816+0000 7fea0cff9640 1 -- 172.21.15.27:0/3241933525 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fea0009c5b0 con 0x7fea10102260 2024-04-25T01:22:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe9f80a2b90 msgr2=0x7fe9f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe9f80a2b90 0x7fe9f80a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fea10198fa0 tx=0x7fea04005f90 comp rx=0 tx=0).stop 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea10102260 msgr2=0x7fea10197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea10102260 0x7fea10197b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fea0000e9f0 tx=0x7fea0000edf0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 shutdown_connections 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe9f80a2b90 0x7fe9f80a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea101074c0 0x7fea1019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea10104b90 0x7fea10198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 --2- 172.21.15.27:0/3241933525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea10102260 0x7fea10197b00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 >> 172.21.15.27:0/3241933525 conn(0x7fea100fbbf0 msgr2=0x7fea10102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 shutdown_connections 2024-04-25T01:22:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.819+0000 7fea15eb3640 1 -- 172.21.15.27:0/3241933525 wait complete. 2024-04-25T01:22:55.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:55 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2504172437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:55.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:55 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:55.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:55 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/955234909' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:22:55.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T01:22:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.943+0000 7ff8b5989640 1 -- 172.21.15.27:0/4074567036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0100680 msgr2=0x7ff8b0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.943+0000 7ff8b5989640 1 --2- 172.21.15.27:0/4074567036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0100680 0x7ff8b0100aa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0009900 tx=0x7ff8a002a410 comp rx=0 tx=0).stop 2024-04-25T01:22:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 -- 172.21.15.27:0/4074567036 shutdown_connections 2024-04-25T01:22:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 --2- 172.21.15.27:0/4074567036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8b0102230 0x7ff8b0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 --2- 172.21.15.27:0/4074567036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8b0101870 0x7ff8b0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 --2- 172.21.15.27:0/4074567036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0100680 0x7ff8b0100aa0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 -- 172.21.15.27:0/4074567036 >> 172.21.15.27:0/4074567036 conn(0x7ff8b00fbbf0 msgr2=0x7ff8b00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 -- 172.21.15.27:0/4074567036 shutdown_connections 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 -- 172.21.15.27:0/4074567036 wait complete. 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 Processor -- start 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 -- start start 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8b0100680 0x7ff8b0069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8b0101870 0x7ff8b019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0102230 0x7ff8b019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b010c2d0 con 0x7ff8b0102230 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b010c150 con 0x7ff8b0100680 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8b5989640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b010c450 con 0x7ff8b0101870 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8af7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0102230 0x7ff8b019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.944+0000 7ff8af7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0102230 0x7ff8b019ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54894/0 (socket says 172.21.15.27:54894) 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8af7fe640 1 -- 172.21.15.27:0/3068543888 learned_addr learned my addr 172.21.15.27:0/3068543888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8aeffd640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8b0100680 0x7ff8b0069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8a7fff640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8b0101870 0x7ff8b019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8af7fe640 1 -- 172.21.15.27:0/3068543888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8b0101870 msgr2=0x7ff8b019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8af7fe640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8b0101870 0x7ff8b019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8af7fe640 1 -- 172.21.15.27:0/3068543888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8b0100680 msgr2=0x7ff8b0069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8af7fe640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8b0100680 0x7ff8b0069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8af7fe640 1 -- 172.21.15.27:0/3068543888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b019f770 con 0x7ff8b0102230 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8af7fe640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0102230 0x7ff8b019ee60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800eaa0 tx=0x7ff8a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8b4987640 1 -- 172.21.15.27:0/3068543888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800cdb0 con 0x7ff8b0102230 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8b4987640 1 -- 172.21.15.27:0/3068543888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a8005020 con 0x7ff8b0102230 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b019fa00 con 0x7ff8b0102230 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8b4987640 1 -- 172.21.15.27:0/3068543888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80116b0 con 0x7ff8b0102230 2024-04-25T01:22:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.945+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b019ff50 con 0x7ff8b0102230 2024-04-25T01:22:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.946+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b0107a70 con 0x7ff8b0102230 2024-04-25T01:22:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.947+0000 7ff8b4987640 1 -- 172.21.15.27:0/3068543888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff8a8005560 con 0x7ff8b0102230 2024-04-25T01:22:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.948+0000 7ff8b4987640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff89c0a2c10 0x7ff89c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.948+0000 7ff8aeffd640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff89c0a2c10 0x7ff89c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:55.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.948+0000 7ff8b4987640 1 -- 172.21.15.27:0/3068543888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7ff8a80cef30 con 0x7ff8b0102230 2024-04-25T01:22:55.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.948+0000 7ff8aeffd640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff89c0a2c10 0x7ff89c0a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0005ab0 tx=0x7ff8a0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:55.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:55.953+0000 7ff8b4987640 1 -- 172.21.15.27:0/3068543888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff8a8016030 con 0x7ff8b0102230 2024-04-25T01:22:56.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:55 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2504172437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:56.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:55 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:56.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:55 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/955234909' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:22:56.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:55 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2504172437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:22:56.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:55 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:56.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:55 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/955234909' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:22:56.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.152+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7ff8b002d170 con 0x7ff8b0102230 2024-04-25T01:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.818+0000 7ff8b4987640 1 -- 172.21.15.27:0/3068543888 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7ff8a809c1c0 con 0x7ff8b0102230 2024-04-25T01:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-04-25T01:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff89c0a2c10 msgr2=0x7ff89c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff89c0a2c10 0x7ff89c0a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0005ab0 tx=0x7ff8a0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0102230 msgr2=0x7ff8b019ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0102230 0x7ff8b019ee60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800eaa0 tx=0x7ff8a800ef60 comp rx=0 tx=0).stop 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 shutdown_connections 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff89c0a2c10 0x7ff89c0a50d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b0102230 0x7ff8b019ee60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8b0101870 0x7ff8b019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 --2- 172.21.15.27:0/3068543888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8b0100680 0x7ff8b0069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 >> 172.21.15.27:0/3068543888 conn(0x7ff8b00fbbf0 msgr2=0x7ff8b0104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.821+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 shutdown_connections 2024-04-25T01:22:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.822+0000 7ff8b5989640 1 -- 172.21.15.27:0/3068543888 wait complete. 2024-04-25T01:22:56.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:22:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/4195048877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8100680 msgr2=0x7ff7f8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/4195048877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8100680 0x7ff7f8100aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009900 tx=0x7ff7e802a410 comp rx=0 tx=0).stop 2024-04-25T01:22:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/4195048877 shutdown_connections 2024-04-25T01:22:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/4195048877 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f8102230 0x7ff7f8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/4195048877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7f8101870 0x7ff7f8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/4195048877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8100680 0x7ff7f8100aa0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/4195048877 >> 172.21.15.27:0/4195048877 conn(0x7ff7f80fbbf0 msgr2=0x7ff7f80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/4195048877 shutdown_connections 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.946+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/4195048877 wait complete. 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7fe4bd640 1 Processor -- start 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7fe4bd640 1 -- start start 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7fe4bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7f8100680 0x7ff7f81045a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7fe4bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101870 0x7ff7f8102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7fe4bd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f8102230 0x7ff7f8103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7fe4bd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f810c320 con 0x7ff7f8101870 2024-04-25T01:22:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7fe4bd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f810c1a0 con 0x7ff7f8102230 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7fe4bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f810c4a0 con 0x7ff7f8100680 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101870 0x7ff7f8102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101870 0x7ff7f8102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54910/0 (socket says 172.21.15.27:54910) 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7f77fe640 1 -- 172.21.15.27:0/2233941469 learned_addr learned my addr 172.21.15.27:0/2233941469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.947+0000 7ff7f7fff640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7f8100680 0x7ff7f81045a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7fca5a640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f8102230 0x7ff7f8103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f77fe640 1 -- 172.21.15.27:0/2233941469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7f8100680 msgr2=0x7ff7f81045a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f77fe640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7f8100680 0x7ff7f81045a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f77fe640 1 -- 172.21.15.27:0/2233941469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f8102230 msgr2=0x7ff7f8103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f77fe640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f8102230 0x7ff7f8103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f77fe640 1 -- 172.21.15.27:0/2233941469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f81039b0 con 0x7ff7f8101870 2024-04-25T01:22:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7fca5a640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f8102230 0x7ff7f8103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:22:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f77fe640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101870 0x7ff7f8102c50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000c920 tx=0x7ff7e000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f57fa640 1 -- 172.21.15.27:0/2233941469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0007c70 con 0x7ff7f8101870 2024-04-25T01:22:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f81a1bd0 con 0x7ff7f8101870 2024-04-25T01:22:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f57fa640 1 -- 172.21.15.27:0/2233941469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7e0007e10 con 0x7ff7f8101870 2024-04-25T01:22:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f81a2170 con 0x7ff7f8101870 2024-04-25T01:22:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.948+0000 7ff7f57fa640 1 -- 172.21.15.27:0/2233941469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0018890 con 0x7ff7f8101870 2024-04-25T01:22:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.949+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f81a2400 con 0x7ff7f8101870 2024-04-25T01:22:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.951+0000 7ff7f57fa640 1 -- 172.21.15.27:0/2233941469 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff7e0018b10 con 0x7ff7f8101870 2024-04-25T01:22:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.952+0000 7ff7f57fa640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff7e40a2c10 0x7ff7e40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.952+0000 7ff7f7fff640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff7e40a2c10 0x7ff7e40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.952+0000 7ff7f57fa640 1 -- 172.21.15.27:0/2233941469 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7ff7e00cf330 con 0x7ff7f8101870 2024-04-25T01:22:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.952+0000 7ff7f7fff640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff7e40a2c10 0x7ff7e40a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff7e802aa00 tx=0x7ff7e80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:56.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:56.955+0000 7ff7f57fa640 1 -- 172.21.15.27:0/2233941469 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff7e0014030 con 0x7ff7f8101870 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3241933525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3241933525' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3068543888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:57.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:57.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:56 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3241933525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:22:57.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:57.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3241933525' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:22:57.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:22:57.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3068543888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:22:57.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.043 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:57.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:57.044 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:56 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.178+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff7f81a25d0 con 0x7ff7f8101870 2024-04-25T01:22:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3241933525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:22:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3241933525' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:22:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:22:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3068543888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:57.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:56 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:22:57.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.820+0000 7ff7f57fa640 1 -- 172.21.15.27:0/2233941469 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7ff7e009c430 con 0x7ff7f8101870 2024-04-25T01:22:57.801 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff7e40a2c10 msgr2=0x7ff7e40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff7e40a2c10 0x7ff7e40a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff7e802aa00 tx=0x7ff7e80315b0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101870 msgr2=0x7ff7f8102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101870 0x7ff7f8102c50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000c920 tx=0x7ff7e000cde0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 shutdown_connections 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff7e40a2c10 0x7ff7e40a50d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f8102230 0x7ff7f8103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8101870 0x7ff7f8102c50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 --2- 172.21.15.27:0/2233941469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7f8100680 0x7ff7f81045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 >> 172.21.15.27:0/2233941469 conn(0x7ff7f80fbbf0 msgr2=0x7ff7f80fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 shutdown_connections 2024-04-25T01:22:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.822+0000 7ff7fe4bd640 1 -- 172.21.15.27:0/2233941469 wait complete. 2024-04-25T01:22:57.825 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T01:22:57.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T01:22:57.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 -- 172.21.15.27:0/1931835088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac103340 msgr2=0x7f99ac109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 --2- 172.21.15.27:0/1931835088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac103340 0x7f99ac109bd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009ca0 tx=0x7f99a402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 -- 172.21.15.27:0/1931835088 shutdown_connections 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 --2- 172.21.15.27:0/1931835088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac103340 0x7f99ac109bd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 --2- 172.21.15.27:0/1931835088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99ac102980 0x7f99ac102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 --2- 172.21.15.27:0/1931835088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99ac101790 0x7f99ac101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 -- 172.21.15.27:0/1931835088 >> 172.21.15.27:0/1931835088 conn(0x7f99ac0fcd20 msgr2=0x7f99ac0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 -- 172.21.15.27:0/1931835088 shutdown_connections 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 -- 172.21.15.27:0/1931835088 wait complete. 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 Processor -- start 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.960+0000 7f99b425d640 1 -- start start 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b425d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99ac101790 0x7f99ac1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b425d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac102980 0x7f99ac197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b425d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99ac103340 0x7f99ac19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b425d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac10d3f0 con 0x7f99ac102980 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b425d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac10d270 con 0x7f99ac101790 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b425d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac10d570 con 0x7f99ac103340 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac102980 0x7f99ac197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac102980 0x7f99ac197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49972/0 (socket says 172.21.15.27:49972) 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 -- 172.21.15.27:0/3258668702 learned_addr learned my addr 172.21.15.27:0/3258668702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b1ff9640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99ac101790 0x7f99ac1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b27fa640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99ac103340 0x7f99ac19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 -- 172.21.15.27:0/3258668702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99ac103340 msgr2=0x7f99ac19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99ac103340 0x7f99ac19ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 -- 172.21.15.27:0/3258668702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99ac101790 msgr2=0x7f99ac1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99ac101790 0x7f99ac1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 -- 172.21.15.27:0/3258668702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ac19f5d0 con 0x7f99ac102980 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b17f8640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac102980 0x7f99ac197ef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f999c00c890 tx=0x7f999c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:57.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99a2ffd640 1 -- 172.21.15.27:0/3258668702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c004220 con 0x7f99ac102980 2024-04-25T01:22:57.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ac19f8c0 con 0x7f99ac102980 2024-04-25T01:22:57.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99a2ffd640 1 -- 172.21.15.27:0/3258668702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f999c0043c0 con 0x7f99ac102980 2024-04-25T01:22:57.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99a2ffd640 1 -- 172.21.15.27:0/3258668702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c004a00 con 0x7f99ac102980 2024-04-25T01:22:57.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.961+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ac19fe10 con 0x7f99ac102980 2024-04-25T01:22:57.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.962+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ac108a50 con 0x7f99ac102980 2024-04-25T01:22:57.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.964+0000 7f99a2ffd640 1 -- 172.21.15.27:0/3258668702 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f999c0079b0 con 0x7f99ac102980 2024-04-25T01:22:57.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.964+0000 7f99a2ffd640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f99980a2b90 0x7f99980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.964+0000 7f99b1ff9640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f99980a2b90 0x7f99980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.964+0000 7f99a2ffd640 1 -- 172.21.15.27:0/3258668702 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f999c0cefc0 con 0x7f99ac102980 2024-04-25T01:22:57.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.965+0000 7f99b1ff9640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f99980a2b90 0x7f99980a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f99ac1027e0 tx=0x7f99a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:57.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:57.968+0000 7f99a2ffd640 1 -- 172.21.15.27:0/3258668702 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f999c014030 con 0x7f99ac102980 2024-04-25T01:22:58.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.166+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f99ac02d170 con 0x7f99ac102980 2024-04-25T01:22:58.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:57 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3068543888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T01:22:58.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:57 smithi027 ceph-mon[29197]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:22:58.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:57 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2233941469' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:22:58.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:57 smithi027 ceph-mon[29197]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:22:58.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:57 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3068543888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T01:22:58.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:57 smithi153 ceph-mon[32621]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:22:58.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:57 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2233941469' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:22:58.197 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:57 smithi153 ceph-mon[32621]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:22:58.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:57 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3068543888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T01:22:58.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:57 smithi139 ceph-mon[32760]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:22:58.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:57 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2233941469' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:22:58.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:57 smithi139 ceph-mon[32760]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:22:58.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.821+0000 7f99a2ffd640 1 -- 172.21.15.27:0/3258668702 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f999c09c070 con 0x7f99ac102980 2024-04-25T01:22:58.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f99980a2b90 msgr2=0x7f99980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f99980a2b90 0x7f99980a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f99ac1027e0 tx=0x7f99a8005f90 comp rx=0 tx=0).stop 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac102980 msgr2=0x7f99ac197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac102980 0x7f99ac197ef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f999c00c890 tx=0x7f999c00cd50 comp rx=0 tx=0).stop 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 shutdown_connections 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f99980a2b90 0x7f99980a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99ac103340 0x7f99ac19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99ac102980 0x7f99ac197ef0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 --2- 172.21.15.27:0/3258668702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99ac101790 0x7f99ac1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 >> 172.21.15.27:0/3258668702 conn(0x7f99ac0fcd20 msgr2=0x7f99ac105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 shutdown_connections 2024-04-25T01:22:58.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.825+0000 7f99b425d640 1 -- 172.21.15.27:0/3258668702 wait complete. 2024-04-25T01:22:58.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 -- 172.21.15.27:0/3838951263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528107460 msgr2=0x7fe5281098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 --2- 172.21.15.27:0/3838951263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528107460 0x7fe5281098b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe520009ca0 tx=0x7fe52002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 -- 172.21.15.27:0/3838951263 shutdown_connections 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 --2- 172.21.15.27:0/3838951263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528107460 0x7fe5281098b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 --2- 172.21.15.27:0/3838951263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528104b30 0x7fe528106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 --2- 172.21.15.27:0/3838951263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe528102200 0x7fe5281045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 -- 172.21.15.27:0/3838951263 >> 172.21.15.27:0/3838951263 conn(0x7fe5280fbbf0 msgr2=0x7fe5280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 -- 172.21.15.27:0/3838951263 shutdown_connections 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.957+0000 7fe52ea37640 1 -- 172.21.15.27:0/3838951263 wait complete. 2024-04-25T01:22:58.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52ea37640 1 Processor -- start 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52ea37640 1 -- start start 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52ea37640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102200 0x7fe5281979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52ea37640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528104b30 0x7fe528197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52ea37640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe528107460 0x7fe52819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52ea37640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810d420 con 0x7fe528104b30 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52ea37640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810d2a0 con 0x7fe528107460 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52ea37640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810d5a0 con 0x7fe528102200 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528104b30 0x7fe528197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528104b30 0x7fe528197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49996/0 (socket says 172.21.15.27:49996) 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe5277fe640 1 -- 172.21.15.27:0/2336522918 learned_addr learned my addr 172.21.15.27:0/2336522918 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe527fff640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102200 0x7fe5281979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.958+0000 7fe52cfd4640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe528107460 0x7fe52819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5277fe640 1 -- 172.21.15.27:0/2336522918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102200 msgr2=0x7fe5281979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5277fe640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102200 0x7fe5281979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5277fe640 1 -- 172.21.15.27:0/2336522918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe528107460 msgr2=0x7fe52819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5277fe640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe528107460 0x7fe52819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5277fe640 1 -- 172.21.15.27:0/2336522918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52819f5f0 con 0x7fe528104b30 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5277fe640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528104b30 0x7fe528197f10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe51800e9f0 tx=0x7fe51800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5257fa640 1 -- 172.21.15.27:0/2336522918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51800ccc0 con 0x7fe528104b30 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5257fa640 1 -- 172.21.15.27:0/2336522918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe518005500 con 0x7fe528104b30 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe52819f8e0 con 0x7fe528104b30 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe5257fa640 1 -- 172.21.15.27:0/2336522918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518010690 con 0x7fe528104b30 2024-04-25T01:22:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.959+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe52819fe30 con 0x7fe528104b30 2024-04-25T01:22:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.960+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5280fd6e0 con 0x7fe528104b30 2024-04-25T01:22:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.961+0000 7fe5257fa640 1 -- 172.21.15.27:0/2336522918 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe5180046c0 con 0x7fe528104b30 2024-04-25T01:22:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.962+0000 7fe5257fa640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe5140a2c10 0x7fe5140a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.962+0000 7fe527fff640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe5140a2c10 0x7fe5140a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.962+0000 7fe5257fa640 1 -- 172.21.15.27:0/2336522918 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fe5180cf3f0 con 0x7fe528104b30 2024-04-25T01:22:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.962+0000 7fe527fff640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe5140a2c10 0x7fe5140a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe51000abe0 tx=0x7fe51000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:58.966+0000 7fe5257fa640 1 -- 172.21.15.27:0/2336522918 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe51809c320 con 0x7fe528104b30 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: pgmap v289: 97 pgs: 10 creating+peering, 4 active+clean, 83 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2233941469' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3258668702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3258668702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T01:22:59.129 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:58 smithi153 ceph-mon[32621]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:22:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:22:59.151+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe528062370 con 0x7fe528104b30 2024-04-25T01:22:59.194 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: pgmap v289: 97 pgs: 10 creating+peering, 4 active+clean, 83 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:59.195 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:22:59.195 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2233941469' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:22:59.195 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:22:59.195 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:22:59.195 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:22:59.195 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3258668702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T01:22:59.195 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3258668702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T01:22:59.195 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:58 smithi139 ceph-mon[32760]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: pgmap v289: 97 pgs: 10 creating+peering, 4 active+clean, 83 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2233941469' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3258668702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3258668702' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T01:22:59.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:58 smithi027 ceph-mon[29197]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:23:00.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:59 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] up:active 2024-04-25T01:23:00.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:59 smithi027 ceph-mon[29197]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:00.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:59 smithi027 ceph-mon[29197]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:00.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:22:59 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2336522918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:23:00.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:59 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] up:active 2024-04-25T01:23:00.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:59 smithi153 ceph-mon[32621]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:00.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:59 smithi153 ceph-mon[32621]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:00.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:22:59 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2336522918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:23:00.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:59 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] up:active 2024-04-25T01:23:00.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:59 smithi139 ceph-mon[32760]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:00.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:59 smithi139 ceph-mon[32760]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:00.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:22:59 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2336522918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:23:00.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.845+0000 7fe5257fa640 1 -- 172.21.15.27:0/2336522918 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fe518086070 con 0x7fe528104b30 2024-04-25T01:23:00.827 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data_ec' created 2024-04-25T01:23:00.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.847+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe5140a2c10 msgr2=0x7fe5140a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.847+0000 7fe52ea37640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe5140a2c10 0x7fe5140a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe51000abe0 tx=0x7fe51000a2d0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.847+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528104b30 msgr2=0x7fe528197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.847+0000 7fe52ea37640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528104b30 0x7fe528197f10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe51800e9f0 tx=0x7fe51800edf0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.848+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 shutdown_connections 2024-04-25T01:23:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.848+0000 7fe52ea37640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe5140a2c10 0x7fe5140a50d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.848+0000 7fe52ea37640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe528107460 0x7fe52819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.848+0000 7fe52ea37640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe528104b30 0x7fe528197f10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.848+0000 7fe52ea37640 1 --2- 172.21.15.27:0/2336522918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe528102200 0x7fe5281979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.848+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 >> 172.21.15.27:0/2336522918 conn(0x7fe5280fbbf0 msgr2=0x7fe528102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.848+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 shutdown_connections 2024-04-25T01:23:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.848+0000 7fe52ea37640 1 -- 172.21.15.27:0/2336522918 wait complete. 2024-04-25T01:23:00.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T01:23:00.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 -- 172.21.15.27:0/3198886238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900102250 msgr2=0x7fb900108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 --2- 172.21.15.27:0/3198886238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900102250 0x7fb900108ae0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009ca0 tx=0x7fb8f802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 -- 172.21.15.27:0/3198886238 shutdown_connections 2024-04-25T01:23:00.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 --2- 172.21.15.27:0/3198886238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900102250 0x7fb900108ae0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 --2- 172.21.15.27:0/3198886238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb900101890 0x7fb900101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 --2- 172.21.15.27:0/3198886238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9001006a0 0x7fb900100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 -- 172.21.15.27:0/3198886238 >> 172.21.15.27:0/3198886238 conn(0x7fb9000fbc30 msgr2=0x7fb9000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 -- 172.21.15.27:0/3198886238 shutdown_connections 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 -- 172.21.15.27:0/3198886238 wait complete. 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.976+0000 7fb905509640 1 Processor -- start 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb905509640 1 -- start start 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb905509640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9001006a0 0x7fb9001968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb905509640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900101890 0x7fb900196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb905509640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb900102250 0x7fb90019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb905509640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010c300 con 0x7fb900101890 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb905509640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010c180 con 0x7fb9001006a0 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb905509640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010c480 con 0x7fb900102250 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900101890 0x7fb900196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900101890 0x7fb900196df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50018/0 (socket says 172.21.15.27:50018) 2024-04-25T01:23:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 -- 172.21.15.27:0/2089919329 learned_addr learned my addr 172.21.15.27:0/2089919329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8feffd640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9001006a0 0x7fb9001968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8ff7fe640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb900102250 0x7fb90019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 -- 172.21.15.27:0/2089919329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb900102250 msgr2=0x7fb90019de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb900102250 0x7fb90019de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 -- 172.21.15.27:0/2089919329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9001006a0 msgr2=0x7fb9001968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9001006a0 0x7fb9001968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 -- 172.21.15.27:0/2089919329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90019e4d0 con 0x7fb900101890 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8ff7fe640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb900102250 0x7fb90019de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.977+0000 7fb8f67fc640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900101890 0x7fb900196df0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8007b50 tx=0x7fb8e800e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.978+0000 7fb8fcff9640 1 -- 172.21.15.27:0/2089919329 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8007c40 con 0x7fb900101890 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.978+0000 7fb8fcff9640 1 -- 172.21.15.27:0/2089919329 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8e8007de0 con 0x7fb900101890 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.978+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb900069da0 con 0x7fb900101890 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.978+0000 7fb8fcff9640 1 -- 172.21.15.27:0/2089919329 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800ae60 con 0x7fb900101890 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.978+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb90006a340 con 0x7fb900101890 2024-04-25T01:23:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.979+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9000689b0 con 0x7fb900101890 2024-04-25T01:23:00.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.980+0000 7fb8fcff9640 1 -- 172.21.15.27:0/2089919329 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb8e800a440 con 0x7fb900101890 2024-04-25T01:23:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.980+0000 7fb8fcff9640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb8ec0a2c10 0x7fb8ec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.981+0000 7fb8feffd640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb8ec0a2c10 0x7fb8ec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.981+0000 7fb8fcff9640 1 -- 172.21.15.27:0/2089919329 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fb8e80d0750 con 0x7fb900101890 2024-04-25T01:23:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.981+0000 7fb8feffd640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb8ec0a2c10 0x7fb8ec0a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb9001016f0 tx=0x7fb8f000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:00.985+0000 7fb8fcff9640 1 -- 172.21.15.27:0/2089919329 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb8e8023060 con 0x7fb900101890 2024-04-25T01:23:01.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.170+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fb9001073d0 con 0x7fb900101890 2024-04-25T01:23:01.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:00 smithi027 ceph-mon[29197]: pgmap v292: 97 pgs: 26 creating+peering, 55 active+clean, 16 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-25T01:23:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:00 smithi027 ceph-mon[29197]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T01:23:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:00 smithi027 ceph-mon[29197]: Cluster is now healthy 2024-04-25T01:23:01.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:00 smithi027 ceph-mon[29197]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:23:01.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:00 smithi153 ceph-mon[32621]: pgmap v292: 97 pgs: 26 creating+peering, 55 active+clean, 16 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-25T01:23:01.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:00 smithi153 ceph-mon[32621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T01:23:01.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:00 smithi153 ceph-mon[32621]: Cluster is now healthy 2024-04-25T01:23:01.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:00 smithi153 ceph-mon[32621]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:23:01.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:00 smithi139 ceph-mon[32760]: pgmap v292: 97 pgs: 26 creating+peering, 55 active+clean, 16 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-25T01:23:01.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:00 smithi139 ceph-mon[32760]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T01:23:01.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:00 smithi139 ceph-mon[32760]: Cluster is now healthy 2024-04-25T01:23:01.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:00 smithi139 ceph-mon[32760]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:23:01.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.848+0000 7fb8fcff9640 1 -- 172.21.15.27:0/2089919329 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fb9001073d0 con 0x7fb900101890 2024-04-25T01:23:01.828 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T01:23:01.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb8ec0a2c10 msgr2=0x7fb8ec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb8ec0a2c10 0x7fb8ec0a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb9001016f0 tx=0x7fb8f000b390 comp rx=0 tx=0).stop 2024-04-25T01:23:01.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900101890 msgr2=0x7fb900196df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900101890 0x7fb900196df0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8007b50 tx=0x7fb8e800e420 comp rx=0 tx=0).stop 2024-04-25T01:23:01.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 shutdown_connections 2024-04-25T01:23:01.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb8ec0a2c10 0x7fb8ec0a50d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb900102250 0x7fb90019de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb900101890 0x7fb900196df0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 --2- 172.21.15.27:0/2089919329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9001006a0 0x7fb9001968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.851+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 >> 172.21.15.27:0/2089919329 conn(0x7fb9000fbc30 msgr2=0x7fb900104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:01.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.852+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 shutdown_connections 2024-04-25T01:23:01.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.852+0000 7fb905509640 1 -- 172.21.15.27:0/2089919329 wait complete. 2024-04-25T01:23:01.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T01:23:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.985+0000 7f38981c8640 1 -- 172.21.15.27:0/390611580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 msgr2=0x7f3890108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.985+0000 7f38981c8640 1 --2- 172.21.15.27:0/390611580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 0x7f3890108ac0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3888009ca0 tx=0x7f388802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.985+0000 7f38981c8640 1 -- 172.21.15.27:0/390611580 shutdown_connections 2024-04-25T01:23:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.985+0000 7f38981c8640 1 --2- 172.21.15.27:0/390611580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 0x7f3890108ac0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.985+0000 7f38981c8640 1 --2- 172.21.15.27:0/390611580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3890101870 0x7f3890101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.985+0000 7f38981c8640 1 --2- 172.21.15.27:0/390611580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3890100680 0x7f3890100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.985+0000 7f38981c8640 1 -- 172.21.15.27:0/390611580 >> 172.21.15.27:0/390611580 conn(0x7f38900fbbf0 msgr2=0x7f38900fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 -- 172.21.15.27:0/390611580 shutdown_connections 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 -- 172.21.15.27:0/390611580 wait complete. 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 Processor -- start 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 -- start start 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3890100680 0x7f3890069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3890101870 0x7f389019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 0x7f389019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f389010c300 con 0x7f3890102230 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f389010c180 con 0x7f3890100680 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f38981c8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f389010c480 con 0x7f3890101870 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f3896765640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 0x7f389019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f3896765640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 0x7f389019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50026/0 (socket says 172.21.15.27:50026) 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.986+0000 7f3896765640 1 -- 172.21.15.27:0/2631481512 learned_addr learned my addr 172.21.15.27:0/2631481512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3895f64640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3890100680 0x7f3890069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3895763640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3890101870 0x7f389019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3896765640 1 -- 172.21.15.27:0/2631481512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3890101870 msgr2=0x7f389019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3896765640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3890101870 0x7f389019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3896765640 1 -- 172.21.15.27:0/2631481512 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3890100680 msgr2=0x7f3890069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3896765640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3890100680 0x7f3890069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3896765640 1 -- 172.21.15.27:0/2631481512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389019f690 con 0x7f3890102230 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3896765640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 0x7f389019ed80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3888005e30 tx=0x7f3888031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3882ffd640 1 -- 172.21.15.27:0/2631481512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888003c30 con 0x7f3890102230 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3882ffd640 1 -- 172.21.15.27:0/2631481512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3888003dd0 con 0x7f3890102230 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f3882ffd640 1 -- 172.21.15.27:0/2631481512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388802fde0 con 0x7f3890102230 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389019f8c0 con 0x7f3890102230 2024-04-25T01:23:01.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.987+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389019fe10 con 0x7f3890102230 2024-04-25T01:23:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.988+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3890107940 con 0x7f3890102230 2024-04-25T01:23:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.990+0000 7f3882ffd640 1 -- 172.21.15.27:0/2631481512 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3888004050 con 0x7f3890102230 2024-04-25T01:23:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.991+0000 7f3882ffd640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f387c0a2b90 0x7f387c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.991+0000 7f3895f64640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f387c0a2b90 0x7f387c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.991+0000 7f3882ffd640 1 -- 172.21.15.27:0/2631481512 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f38880f07a0 con 0x7f3890102230 2024-04-25T01:23:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.991+0000 7f3895f64640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f387c0a2b90 0x7f387c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f38901016d0 tx=0x7f3884005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:01.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:01.994+0000 7f3882ffd640 1 -- 172.21.15.27:0/2631481512 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f38880bd480 con 0x7f3890102230 2024-04-25T01:23:02.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:01 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2336522918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:23:02.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:01 smithi027 ceph-mon[29197]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:23:02.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:01 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2089919329' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T01:23:02.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:01 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2089919329' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T01:23:02.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:01 smithi027 ceph-mon[29197]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:23:02.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.223+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f389002d290 con 0x7f3890102230 2024-04-25T01:23:02.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:01 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2336522918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:23:02.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:01 smithi153 ceph-mon[32621]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:23:02.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:01 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2089919329' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T01:23:02.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:01 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2089919329' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T01:23:02.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:01 smithi153 ceph-mon[32621]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:23:02.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:01 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2336522918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:23:02.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:01 smithi139 ceph-mon[32760]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:23:02.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:01 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2089919329' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T01:23:02.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:01 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2089919329' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T01:23:02.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:01 smithi139 ceph-mon[32760]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:23:02.793 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:23:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b[63104]: 2024-04-25T01:23:02.537+0000 7f4018d47700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:23:02.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.881+0000 7f3882ffd640 1 -- 172.21.15.27:0/2631481512 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f38880bcd80 con 0x7f3890102230 2024-04-25T01:23:02.862 INFO:teuthology.orchestra.run.smithi027.stderr:added data pool 4 to fsmap 2024-04-25T01:23:02.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f387c0a2b90 msgr2=0x7f387c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f387c0a2b90 0x7f387c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f38901016d0 tx=0x7f3884005f90 comp rx=0 tx=0).stop 2024-04-25T01:23:02.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 msgr2=0x7f389019ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 0x7f389019ed80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3888005e30 tx=0x7f3888031ee0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 shutdown_connections 2024-04-25T01:23:02.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f387c0a2b90 0x7f387c0a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890102230 0x7f389019ed80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3890101870 0x7f389019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 --2- 172.21.15.27:0/2631481512 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3890100680 0x7f3890069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 >> 172.21.15.27:0/2631481512 conn(0x7f38900fbbf0 msgr2=0x7f3890104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 shutdown_connections 2024-04-25T01:23:02.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:02.883+0000 7f38981c8640 1 -- 172.21.15.27:0/2631481512 wait complete. 2024-04-25T01:23:02.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 -- 172.21.15.27:0/2897683507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4102250 msgr2=0x7f8dd4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/2897683507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4102250 0x7f8dd4108ae0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009ca0 tx=0x7f8dcc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 -- 172.21.15.27:0/2897683507 shutdown_connections 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/2897683507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4102250 0x7f8dd4108ae0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/2897683507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dd4101890 0x7f8dd4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/2897683507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dd41006a0 0x7f8dd4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 -- 172.21.15.27:0/2897683507 >> 172.21.15.27:0/2897683507 conn(0x7f8dd40fbc30 msgr2=0x7f8dd40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 -- 172.21.15.27:0/2897683507 shutdown_connections 2024-04-25T01:23:02.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.001+0000 7f8dd8bee640 1 -- 172.21.15.27:0/2897683507 wait complete. 2024-04-25T01:23:02.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd8bee640 1 Processor -- start 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd8bee640 1 -- start start 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd8bee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dd41006a0 0x7f8dd4197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd8bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4101890 0x7f8dd4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd8bee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dd4102250 0x7f8dd419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd8bee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd410c300 con 0x7f8dd4101890 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd8bee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd410c180 con 0x7f8dd4102250 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd8bee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd410c480 con 0x7f8dd41006a0 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dcbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4101890 0x7f8dd4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dcbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4101890 0x7f8dd4197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50046/0 (socket says 172.21.15.27:50046) 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dcbfff640 1 -- 172.21.15.27:0/407647059 learned_addr learned my addr 172.21.15.27:0/407647059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd259c640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dd41006a0 0x7f8dd4197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dd2d9d640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dd4102250 0x7f8dd419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dcbfff640 1 -- 172.21.15.27:0/407647059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dd41006a0 msgr2=0x7f8dd4197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dcbfff640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dd41006a0 0x7f8dd4197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dcbfff640 1 -- 172.21.15.27:0/407647059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dd4102250 msgr2=0x7f8dd419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dcbfff640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dd4102250 0x7f8dd419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.002+0000 7f8dcbfff640 1 -- 172.21.15.27:0/407647059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd419f690 con 0x7f8dd4101890 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.003+0000 7f8dcbfff640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4101890 0x7f8dd4197fb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00c890 tx=0x7f8dbc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.003+0000 7f8dcb7fe640 1 -- 172.21.15.27:0/407647059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc004220 con 0x7f8dd4101890 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.003+0000 7f8dcb7fe640 1 -- 172.21.15.27:0/407647059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dbc0043c0 con 0x7f8dd4101890 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.003+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd419f980 con 0x7f8dd4101890 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.003+0000 7f8dcb7fe640 1 -- 172.21.15.27:0/407647059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc0049e0 con 0x7f8dd4101890 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.003+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd419fed0 con 0x7f8dd4101890 2024-04-25T01:23:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.004+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd41a0160 con 0x7f8dd4101890 2024-04-25T01:23:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.005+0000 7f8dcb7fe640 1 -- 172.21.15.27:0/407647059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8dbc0079b0 con 0x7f8dd4101890 2024-04-25T01:23:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.005+0000 7f8dcb7fe640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8dc00a2b90 0x7f8dc00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.006+0000 7f8dd259c640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8dc00a2b90 0x7f8dc00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.006+0000 7f8dcb7fe640 1 -- 172.21.15.27:0/407647059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8dbc0cf390 con 0x7f8dd4101890 2024-04-25T01:23:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.006+0000 7f8dd259c640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8dc00a2b90 0x7f8dc00a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4005fd0 tx=0x7f8dc4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.010+0000 7f8dcb7fe640 1 -- 172.21.15.27:0/407647059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8dbc014030 con 0x7f8dd4101890 2024-04-25T01:23:03.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:02 smithi027 ceph-mon[29197]: pgmap v295: 129 pgs: 26 creating+peering, 62 active+clean, 41 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-25T01:23:03.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:02 smithi027 ceph-mon[29197]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:23:03.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:02 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2631481512' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T01:23:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.230+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dd41a0330 con 0x7f8dd4101890 2024-04-25T01:23:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.231+0000 7f8dcb7fe640 1 -- 172.21.15.27:0/407647059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f8dbc09c0b0 con 0x7f8dd4101890 2024-04-25T01:23:03.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:23:02.224326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24521,"qdb_cluster":[24521]},"id":1}]} 2024-04-25T01:23:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8dc00a2b90 msgr2=0x7f8dc00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8dc00a2b90 0x7f8dc00a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4005fd0 tx=0x7f8dc4005850 comp rx=0 tx=0).stop 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4101890 msgr2=0x7f8dd4197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4101890 0x7f8dd4197fb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00c890 tx=0x7f8dbc00cd50 comp rx=0 tx=0).stop 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 shutdown_connections 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8dc00a2b90 0x7f8dc00a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8dd4102250 0x7f8dd419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd4101890 0x7f8dd4197fb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 --2- 172.21.15.27:0/407647059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8dd41006a0 0x7f8dd4197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 >> 172.21.15.27:0/407647059 conn(0x7f8dd40fbc30 msgr2=0x7f8dd40fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 shutdown_connections 2024-04-25T01:23:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.233+0000 7f8dd8bee640 1 -- 172.21.15.27:0/407647059 wait complete. 2024-04-25T01:23:03.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:03.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:02 smithi153 ceph-mon[32621]: pgmap v295: 129 pgs: 26 creating+peering, 62 active+clean, 41 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-25T01:23:03.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:02 smithi153 ceph-mon[32621]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:23:03.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:02 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2631481512' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T01:23:03.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:02 smithi139 ceph-mon[32760]: pgmap v295: 129 pgs: 26 creating+peering, 62 active+clean, 41 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-25T01:23:03.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:02 smithi139 ceph-mon[32760]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:23:03.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:02 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2631481512' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 -- 172.21.15.27:0/2752461617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc102250 msgr2=0x7fc2bc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 --2- 172.21.15.27:0/2752461617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc102250 0x7fc2bc108ae0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009ca0 tx=0x7fc2b402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 -- 172.21.15.27:0/2752461617 shutdown_connections 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 --2- 172.21.15.27:0/2752461617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc102250 0x7fc2bc108ae0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 --2- 172.21.15.27:0/2752461617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2bc101890 0x7fc2bc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 --2- 172.21.15.27:0/2752461617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2bc1006a0 0x7fc2bc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 -- 172.21.15.27:0/2752461617 >> 172.21.15.27:0/2752461617 conn(0x7fc2bc0fbc30 msgr2=0x7fc2bc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 -- 172.21.15.27:0/2752461617 shutdown_connections 2024-04-25T01:23:03.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 -- 172.21.15.27:0/2752461617 wait complete. 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.346+0000 7fc2c1729640 1 Processor -- start 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2c1729640 1 -- start start 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2c1729640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2bc1006a0 0x7fc2bc199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2c1729640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc101890 0x7fc2bc19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2c1729640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2bc102250 0x7fc2bc1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2c1729640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc10c320 con 0x7fc2bc101890 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2c1729640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc10c1a0 con 0x7fc2bc1006a0 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2c1729640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc10c4a0 con 0x7fc2bc102250 2024-04-25T01:23:03.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc101890 0x7fc2bc19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc101890 0x7fc2bc19a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50070/0 (socket says 172.21.15.27:50070) 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 -- 172.21.15.27:0/1451551165 learned_addr learned my addr 172.21.15.27:0/1451551165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2bb7fe640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2bc102250 0x7fc2bc1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2baffd640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2bc1006a0 0x7fc2bc199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 -- 172.21.15.27:0/1451551165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2bc102250 msgr2=0x7fc2bc1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2bc102250 0x7fc2bc1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 -- 172.21.15.27:0/1451551165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2bc1006a0 msgr2=0x7fc2bc199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2bc1006a0 0x7fc2bc199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 -- 172.21.15.27:0/1451551165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc1a17e0 con 0x7fc2bc101890 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2bb7fe640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2bc102250 0x7fc2bc1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2baffd640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2bc1006a0 0x7fc2bc199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2ba7fc640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc101890 0x7fc2bc19a100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000c920 tx=0x7fc2b000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2a3fff640 1 -- 172.21.15.27:0/1451551165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0007c70 con 0x7fc2bc101890 2024-04-25T01:23:03.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.347+0000 7fc2a3fff640 1 -- 172.21.15.27:0/1451551165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2b0007e10 con 0x7fc2bc101890 2024-04-25T01:23:03.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.348+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2bc1a1ad0 con 0x7fc2bc101890 2024-04-25T01:23:03.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.348+0000 7fc2a3fff640 1 -- 172.21.15.27:0/1451551165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0018890 con 0x7fc2bc101890 2024-04-25T01:23:03.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.348+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2bc1a2070 con 0x7fc2bc101890 2024-04-25T01:23:03.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.348+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2bc1a22a0 con 0x7fc2bc101890 2024-04-25T01:23:03.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.350+0000 7fc2a3fff640 1 -- 172.21.15.27:0/1451551165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc2b0018a70 con 0x7fc2bc101890 2024-04-25T01:23:03.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.350+0000 7fc2a3fff640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc2a40a2c10 0x7fc2a40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.351+0000 7fc2baffd640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc2a40a2c10 0x7fc2a40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.351+0000 7fc2a3fff640 1 -- 172.21.15.27:0/1451551165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc2b00cf3e0 con 0x7fc2bc101890 2024-04-25T01:23:03.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.351+0000 7fc2baffd640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc2a40a2c10 0x7fc2a40a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0058c0 tx=0x7fc2ac005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.355+0000 7fc2a3fff640 1 -- 172.21.15.27:0/1451551165 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc2b0014030 con 0x7fc2bc101890 2024-04-25T01:23:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.542+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2bc1a2510 con 0x7fc2bc101890 2024-04-25T01:23:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.543+0000 7fc2a3fff640 1 -- 172.21.15.27:0/1451551165 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7fc2b009c100 con 0x7fc2bc101890 2024-04-25T01:23:03.524 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:03.524 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.545+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc2a40a2c10 msgr2=0x7fc2a40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.545+0000 7fc2c1729640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc2a40a2c10 0x7fc2a40a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0058c0 tx=0x7fc2ac005d50 comp rx=0 tx=0).stop 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.545+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc101890 msgr2=0x7fc2bc19a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.545+0000 7fc2c1729640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc101890 0x7fc2bc19a100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000c920 tx=0x7fc2b000cde0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.546+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 shutdown_connections 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.546+0000 7fc2c1729640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc2a40a2c10 0x7fc2a40a50d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.546+0000 7fc2c1729640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2bc102250 0x7fc2bc1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.546+0000 7fc2c1729640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc101890 0x7fc2bc19a100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.546+0000 7fc2c1729640 1 --2- 172.21.15.27:0/1451551165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2bc1006a0 0x7fc2bc199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.546+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 >> 172.21.15.27:0/1451551165 conn(0x7fc2bc0fbc30 msgr2=0x7fc2bc0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.546+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 shutdown_connections 2024-04-25T01:23:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.546+0000 7fc2c1729640 1 -- 172.21.15.27:0/1451551165 wait complete. 2024-04-25T01:23:03.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.658+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2563364717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0102230 msgr2=0x7fcdc0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.658+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2563364717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0102230 0x7fcdc0108ac0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009ca0 tx=0x7fcdbc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.658+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2563364717 shutdown_connections 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.658+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2563364717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0102230 0x7fcdc0108ac0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.658+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2563364717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc0101870 0x7fcdc0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.658+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2563364717 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdc0100680 0x7fcdc0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.658+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2563364717 >> 172.21.15.27:0/2563364717 conn(0x7fcdc00fbbf0 msgr2=0x7fcdc00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2563364717 shutdown_connections 2024-04-25T01:23:03.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2563364717 wait complete. 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 Processor -- start 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 -- start start 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdc0100680 0x7fcdc0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0101870 0x7fcdc0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc0102230 0x7fcdc019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc010c300 con 0x7fcdc0101870 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc010c180 con 0x7fcdc0100680 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc81dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc010c480 con 0x7fcdc0102230 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc5777640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0101870 0x7fcdc0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc5777640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0101870 0x7fcdc0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50094/0 (socket says 172.21.15.27:50094) 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.659+0000 7fcdc5777640 1 -- 172.21.15.27:0/2821129204 learned_addr learned my addr 172.21.15.27:0/2821129204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc5f78640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdc0100680 0x7fcdc0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc6779640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc0102230 0x7fcdc019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc5777640 1 -- 172.21.15.27:0/2821129204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc0102230 msgr2=0x7fcdc019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc5777640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc0102230 0x7fcdc019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc5777640 1 -- 172.21.15.27:0/2821129204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdc0100680 msgr2=0x7fcdc0197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc5777640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdc0100680 0x7fcdc0197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc5777640 1 -- 172.21.15.27:0/2821129204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc019f6b0 con 0x7fcdc0101870 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc5777640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0101870 0x7fcdc0197fd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000d770 tx=0x7fcdb000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdb6ffd640 1 -- 172.21.15.27:0/2821129204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0017070 con 0x7fcdc0101870 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdb6ffd640 1 -- 172.21.15.27:0/2821129204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdb0005500 con 0x7fcdc0101870 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc019f9a0 con 0x7fcdc0101870 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdb6ffd640 1 -- 172.21.15.27:0/2821129204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0004e00 con 0x7fcdc0101870 2024-04-25T01:23:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.660+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc019fef0 con 0x7fcdc0101870 2024-04-25T01:23:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.661+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc01a0120 con 0x7fcdc0101870 2024-04-25T01:23:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.662+0000 7fcdb6ffd640 1 -- 172.21.15.27:0/2821129204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcdb0004fa0 con 0x7fcdc0101870 2024-04-25T01:23:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.663+0000 7fcdb6ffd640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcdac0a2c10 0x7fcdac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.663+0000 7fcdc5f78640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcdac0a2c10 0x7fcdac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.663+0000 7fcdb6ffd640 1 -- 172.21.15.27:0/2821129204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcdb00cec50 con 0x7fcdc0101870 2024-04-25T01:23:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.663+0000 7fcdc5f78640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcdac0a2c10 0x7fcdac0a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcdc01016d0 tx=0x7fcdb800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.667+0000 7fcdb6ffd640 1 -- 172.21.15.27:0/2821129204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcdb00d4020 con 0x7fcdc0101870 2024-04-25T01:23:03.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.857+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcdc01a04b0 con 0x7fcdc0101870 2024-04-25T01:23:03.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.858+0000 7fcdb6ffd640 1 -- 172.21.15.27:0/2821129204 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7fcdc01a04b0 con 0x7fcdc0101870 2024-04-25T01:23:03.839 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:03.839 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcdac0a2c10 msgr2=0x7fcdac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcdac0a2c10 0x7fcdac0a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcdc01016d0 tx=0x7fcdb800b390 comp rx=0 tx=0).stop 2024-04-25T01:23:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0101870 msgr2=0x7fcdc0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0101870 0x7fcdc0197fd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000d770 tx=0x7fcdb000dc30 comp rx=0 tx=0).stop 2024-04-25T01:23:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 shutdown_connections 2024-04-25T01:23:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcdac0a2c10 0x7fcdac0a50d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc0102230 0x7fcdc019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdc0101870 0x7fcdc0197fd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 --2- 172.21.15.27:0/2821129204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcdc0100680 0x7fcdc0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 >> 172.21.15.27:0/2821129204 conn(0x7fcdc00fbbf0 msgr2=0x7fcdc00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 shutdown_connections 2024-04-25T01:23:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.861+0000 7fcdc81dc640 1 -- 172.21.15.27:0/2821129204 wait complete. 2024-04-25T01:23:03.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 -- 172.21.15.27:0/2630779414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 msgr2=0x7f818c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 --2- 172.21.15.27:0/2630779414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 0x7f818c108ae0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8184009ca0 tx=0x7f818402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 -- 172.21.15.27:0/2630779414 shutdown_connections 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 --2- 172.21.15.27:0/2630779414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 0x7f818c108ae0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 --2- 172.21.15.27:0/2630779414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f818c101890 0x7f818c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 --2- 172.21.15.27:0/2630779414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c1006a0 0x7f818c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 -- 172.21.15.27:0/2630779414 >> 172.21.15.27:0/2630779414 conn(0x7f818c0fbc30 msgr2=0x7f818c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 -- 172.21.15.27:0/2630779414 shutdown_connections 2024-04-25T01:23:03.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.987+0000 7f8192509640 1 -- 172.21.15.27:0/2630779414 wait complete. 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8192509640 1 Processor -- start 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8192509640 1 -- start start 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8192509640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f818c1006a0 0x7f818c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8192509640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c101890 0x7f818c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8192509640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 0x7f818c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8192509640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f818c10c320 con 0x7f818c102250 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8192509640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f818c10c1a0 con 0x7f818c1006a0 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8192509640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f818c10c4a0 con 0x7f818c101890 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8190aa6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 0x7f818c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8190aa6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 0x7f818c19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50108/0 (socket says 172.21.15.27:50108) 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8190aa6640 1 -- 172.21.15.27:0/870699131 learned_addr learned my addr 172.21.15.27:0/870699131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f818bfff640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f818c1006a0 0x7f818c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8190aa6640 1 -- 172.21.15.27:0/870699131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c101890 msgr2=0x7f818c19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8190aa6640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c101890 0x7f818c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8190aa6640 1 -- 172.21.15.27:0/870699131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f818c1006a0 msgr2=0x7f818c06a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8190aa6640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f818c1006a0 0x7f818c06a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f8190aa6640 1 -- 172.21.15.27:0/870699131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818c19f6e0 con 0x7f818c102250 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.988+0000 7f818bfff640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f818c1006a0 0x7f818c06a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.989+0000 7f8190aa6640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 0x7f818c19edd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8184005e30 tx=0x7f8184031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.989+0000 7f81897fa640 1 -- 172.21.15.27:0/870699131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8184003c30 con 0x7f818c102250 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.989+0000 7f81897fa640 1 -- 172.21.15.27:0/870699131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8184003dd0 con 0x7f818c102250 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.989+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818c19f910 con 0x7f818c102250 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.989+0000 7f81897fa640 1 -- 172.21.15.27:0/870699131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818402fde0 con 0x7f818c102250 2024-04-25T01:23:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.989+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818c19fe60 con 0x7f818c102250 2024-04-25T01:23:03.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.990+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f818c107a90 con 0x7f818c102250 2024-04-25T01:23:03.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.991+0000 7f81897fa640 1 -- 172.21.15.27:0/870699131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8184004050 con 0x7f818c102250 2024-04-25T01:23:03.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.992+0000 7f81897fa640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f81740a2b90 0x7f81740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:03.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.992+0000 7f818bfff640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f81740a2b90 0x7f81740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:03.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.992+0000 7f81897fa640 1 -- 172.21.15.27:0/870699131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f81840f0840 con 0x7f818c102250 2024-04-25T01:23:03.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.992+0000 7f818bfff640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f81740a2b90 0x7f81740a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f818c1016f0 tx=0x7f817c005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:03.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:03.997+0000 7f81897fa640 1 -- 172.21.15.27:0/870699131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f81840bd680 con 0x7f818c102250 2024-04-25T01:23:04.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:03 smithi027 ceph-mon[29197]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:23:04.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2631481512' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T01:23:04.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:03 smithi027 ceph-mon[29197]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:04.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/407647059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1451551165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2821129204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.218+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f818c02d350 con 0x7f818c102250 2024-04-25T01:23:04.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.219+0000 7f81897fa640 1 -- 172.21.15.27:0/870699131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f81840bcf80 con 0x7f818c102250 2024-04-25T01:23:04.199 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:04.199 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:23:02.224326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24521,"qdb_cluster":[24521]},"id":1}]} 2024-04-25T01:23:04.200 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-04-25T01:23:04.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f81740a2b90 msgr2=0x7f81740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f81740a2b90 0x7f81740a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f818c1016f0 tx=0x7f817c005e70 comp rx=0 tx=0).stop 2024-04-25T01:23:04.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 msgr2=0x7f818c19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 0x7f818c19edd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8184005e30 tx=0x7f8184031ee0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 shutdown_connections 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f81740a2b90 0x7f81740a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f818c102250 0x7f818c19edd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f818c101890 0x7f818c19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 --2- 172.21.15.27:0/870699131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f818c1006a0 0x7f818c06a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 >> 172.21.15.27:0/870699131 conn(0x7f818c0fbc30 msgr2=0x7f818c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 shutdown_connections 2024-04-25T01:23:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.221+0000 7f8192509640 1 -- 172.21.15.27:0/870699131 wait complete. 2024-04-25T01:23:04.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:04.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:03 smithi153 ceph-mon[32621]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:23:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2631481512' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T01:23:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:03 smithi153 ceph-mon[32621]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/407647059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1451551165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2821129204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:03 smithi139 ceph-mon[32760]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:23:04.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2631481512' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T01:23:04.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:03 smithi139 ceph-mon[32760]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:04.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/407647059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1451551165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2821129204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 -- 172.21.15.27:0/23975938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0107460 msgr2=0x7f36d01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 --2- 172.21.15.27:0/23975938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0107460 0x7f36d01098b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009ca0 tx=0x7f36c802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 -- 172.21.15.27:0/23975938 shutdown_connections 2024-04-25T01:23:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 --2- 172.21.15.27:0/23975938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0107460 0x7f36d01098b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 --2- 172.21.15.27:0/23975938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d0104b30 0x7f36d0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 --2- 172.21.15.27:0/23975938 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0102200 0x7f36d01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 -- 172.21.15.27:0/23975938 >> 172.21.15.27:0/23975938 conn(0x7f36d00fbbf0 msgr2=0x7f36d00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 -- 172.21.15.27:0/23975938 shutdown_connections 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 -- 172.21.15.27:0/23975938 wait complete. 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 Processor -- start 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.334+0000 7f36d64ca640 1 -- start start 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d64ca640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0102200 0x7f36d0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d64ca640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d0104b30 0x7f36d0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d64ca640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0107460 0x7f36d019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d64ca640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f36d010d420 con 0x7f36d0102200 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d64ca640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f36d010d2a0 con 0x7f36d0107460 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d64ca640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36d010d5a0 con 0x7f36d0104b30 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0102200 0x7f36d0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0102200 0x7f36d0197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50136/0 (socket says 172.21.15.27:50136) 2024-04-25T01:23:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 -- 172.21.15.27:0/1196084497 learned_addr learned my addr 172.21.15.27:0/1196084497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d0104b30 0x7f36d0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d4a67640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0107460 0x7f36d019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 -- 172.21.15.27:0/1196084497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d0104b30 msgr2=0x7f36d0197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d0104b30 0x7f36d0197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 -- 172.21.15.27:0/1196084497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0107460 msgr2=0x7f36d019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0107460 0x7f36d019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 -- 172.21.15.27:0/1196084497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d019f660 con 0x7f36d0102200 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d4a67640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0107460 0x7f36d019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cffff640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0102200 0x7f36d0197a40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f36c000d950 tx=0x7f36c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cd7fa640 1 -- 172.21.15.27:0/1196084497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c0017070 con 0x7f36d0102200 2024-04-25T01:23:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d019f950 con 0x7f36d0102200 2024-04-25T01:23:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cd7fa640 1 -- 172.21.15.27:0/1196084497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36c0009c30 con 0x7f36d0102200 2024-04-25T01:23:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36cd7fa640 1 -- 172.21.15.27:0/1196084497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c000f6b0 con 0x7f36d0102200 2024-04-25T01:23:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.335+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d019fea0 con 0x7f36d0102200 2024-04-25T01:23:04.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.336+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d00fd6e0 con 0x7f36d0102200 2024-04-25T01:23:04.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.337+0000 7f36cd7fa640 1 -- 172.21.15.27:0/1196084497 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f36c00109d0 con 0x7f36d0102200 2024-04-25T01:23:04.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.338+0000 7f36cd7fa640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36bc0a2b90 0x7f36bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.338+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36bc0a2b90 0x7f36bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.338+0000 7f36cd7fa640 1 -- 172.21.15.27:0/1196084497 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f36c00cefa0 con 0x7f36d0102200 2024-04-25T01:23:04.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.338+0000 7f36cf7fe640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36bc0a2b90 0x7f36bc0a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f36d0198f50 tx=0x7f36b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.343+0000 7f36cd7fa640 1 -- 172.21.15.27:0/1196084497 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f36c0009de0 con 0x7f36d0102200 2024-04-25T01:23:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.527+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36d00623e0 con 0x7f36d0102200 2024-04-25T01:23:04.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.529+0000 7f36cd7fa640 1 -- 172.21.15.27:0/1196084497 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7f36c009bcc0 con 0x7f36d0102200 2024-04-25T01:23:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:04.510 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36bc0a2b90 msgr2=0x7f36bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36bc0a2b90 0x7f36bc0a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f36d0198f50 tx=0x7f36b8005f90 comp rx=0 tx=0).stop 2024-04-25T01:23:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0102200 msgr2=0x7f36d0197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0102200 0x7f36d0197a40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f36c000d950 tx=0x7f36c000de10 comp rx=0 tx=0).stop 2024-04-25T01:23:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 shutdown_connections 2024-04-25T01:23:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f36bc0a2b90 0x7f36bc0a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36d0107460 0x7f36d019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36d0104b30 0x7f36d0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 --2- 172.21.15.27:0/1196084497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36d0102200 0x7f36d0197a40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 >> 172.21.15.27:0/1196084497 conn(0x7f36d00fbbf0 msgr2=0x7f36d0102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 shutdown_connections 2024-04-25T01:23:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.531+0000 7f36d64ca640 1 -- 172.21.15.27:0/1196084497 wait complete. 2024-04-25T01:23:04.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:22:57.180706+0000', 'modified': '2024-04-25T01:23:02.224326+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.139:6839/2167485871', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 2167485871}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 2167485871}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24521, 'qdb_cluster': [24521]} 2024-04-25T01:23:04.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:23:04.523 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:23:04.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.645+0000 7f3a8b99b640 1 -- 172.21.15.27:0/2236744021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a840ffff0 msgr2=0x7f3a84102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.645+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/2236744021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a840ffff0 0x7f3a84102420 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009900 tx=0x7f3a7c02a410 comp rx=0 tx=0).stop 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.645+0000 7f3a8b99b640 1 -- 172.21.15.27:0/2236744021 shutdown_connections 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.645+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/2236744021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a84105310 0x7f3a84105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.645+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/2236744021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a84102960 0x7f3a84104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.645+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/2236744021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a840ffff0 0x7f3a84102420 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 -- 172.21.15.27:0/2236744021 >> 172.21.15.27:0/2236744021 conn(0x7f3a840fbbf0 msgr2=0x7f3a840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 -- 172.21.15.27:0/2236744021 shutdown_connections 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 -- 172.21.15.27:0/2236744021 wait complete. 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 Processor -- start 2024-04-25T01:23:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 -- start start 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a840ffff0 0x7f3a8419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a84102960 0x7f3a8419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a84105310 0x7f3a841a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3a84113a40 con 0x7f3a84105310 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3a841138c0 con 0x7f3a840ffff0 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.646+0000 7f3a8b99b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a84113bc0 con 0x7f3a84102960 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a84105310 0x7f3a841a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a84105310 0x7f3a841a56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50150/0 (socket says 172.21.15.27:50150) 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 -- 172.21.15.27:0/690285720 learned_addr learned my addr 172.21.15.27:0/690285720 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:04.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a88f36640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a84102960 0x7f3a8419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89737640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a840ffff0 0x7f3a8419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 -- 172.21.15.27:0/690285720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a84102960 msgr2=0x7f3a8419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a84102960 0x7f3a8419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 -- 172.21.15.27:0/690285720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a840ffff0 msgr2=0x7f3a8419e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a840ffff0 0x7f3a8419e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 -- 172.21.15.27:0/690285720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a841a5d60 con 0x7f3a84105310 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89737640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a840ffff0 0x7f3a8419e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a89f38640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a84105310 0x7f3a841a56f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000cad0 tx=0x7f3a8000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a7a7fc640 1 -- 172.21.15.27:0/690285720 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8000ae20 con 0x7f3a84105310 2024-04-25T01:23:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a7a7fc640 1 -- 172.21.15.27:0/690285720 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a80005020 con 0x7f3a84105310 2024-04-25T01:23:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a7a7fc640 1 -- 172.21.15.27:0/690285720 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8000f6f0 con 0x7f3a84105310 2024-04-25T01:23:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a8b99b640 1 -- 172.21.15.27:0/690285720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a841a5ff0 con 0x7f3a84105310 2024-04-25T01:23:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.647+0000 7f3a8b99b640 1 -- 172.21.15.27:0/690285720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a841a64e0 con 0x7f3a84105310 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.650+0000 7f3a7a7fc640 1 -- 172.21.15.27:0/690285720 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3a800051d0 con 0x7f3a84105310 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.650+0000 7f3a8e783740 1 Processor -- start 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.650+0000 7f3a8e783740 1 -- start start 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2c4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 0x563c3a2cc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a8e783740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563c3a227f00 con 0x563c3a2c4e30 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a8e783740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563c3a1d0750 con 0x563c3a2c4440 2024-04-25T01:23:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a8e783740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563c3a223ec0 con 0x563c3a2c9fa0 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50154/0 (socket says 172.21.15.27:50154) 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 -- 172.21.15.27:0/2581790919 learned_addr learned my addr 172.21.15.27:0/2581790919 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a88f36640 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a88f36640 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 0x563c3a2cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 -- 172.21.15.27:0/2581790919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 msgr2=0x563c3a2cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 0x563c3a2cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 -- 172.21.15.27:0/2581790919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 msgr2=0x563c3a2c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 -- 172.21.15.27:0/2581790919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c39f3c2d0 con 0x563c3a2c4e30 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a88f36640 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 0x563c3a2cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a7a7fc640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3a700a2b90 0x7f3a700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a89737640 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c9930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c000c00 tx=0x7f3a7c032df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4829da0246dca4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a88f36640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3a700a2b90 0x7f3a700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a7a7fc640 1 -- 172.21.15.27:0/690285720 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3a800cf330 con 0x7f3a84105310 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.651+0000 7f3a6f7fe640 1 -- 172.21.15.27:0/2581790919 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c03f070 con 0x563c3a2c4e30 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a6f7fe640 1 -- 172.21.15.27:0/2581790919 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a7c0318b0 con 0x563c3a2c4e30 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a6f7fe640 1 -- 172.21.15.27:0/2581790919 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c035070 con 0x563c3a2c4e30 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 -- 172.21.15.27:0/2581790919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 msgr2=0x563c3a2c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a88f36640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3a700a2b90 0x7f3a700a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a74006fd0 tx=0x7f3a7400a230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c9930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c000c00 tx=0x7f3a7c032df0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 -- 172.21.15.27:0/2581790919 shutdown_connections 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 0x563c3a2cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c9930 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 --2- 172.21.15.27:0/2581790919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 -- 172.21.15.27:0/2581790919 >> 172.21.15.27:0/2581790919 conn(0x563c3a2bfab0 msgr2=0x563c3a2bfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 -- 172.21.15.27:0/2581790919 shutdown_connections 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.652+0000 7f3a8e783740 1 -- 172.21.15.27:0/2581790919 wait complete. 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 1 -- start start 2024-04-25T01:23:04.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 1 Processor -- start 2024-04-25T01:23:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c4e30 0x563c3a2cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563c3a2ccef0 con 0x563c3a2c9fa0 2024-04-25T01:23:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563c3a1e3cf0 con 0x563c3a2c4440 2024-04-25T01:23:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563c3a2cd070 con 0x563c3a2c4e30 2024-04-25T01:23:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a8e783740 10 client.0 _ll_register_callbacks cb 0x563c3a1e36e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:23:04.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89f38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89f38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50168/0 (socket says 172.21.15.27:50168) 2024-04-25T01:23:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89f38640 1 -- 172.21.15.27:0/3502580068 learned_addr learned my addr 172.21.15.27:0/3502580068 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89737640 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89f38640 1 -- 172.21.15.27:0/3502580068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c4e30 msgr2=0x563c3a2cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89f38640 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c4e30 0x563c3a2cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89f38640 1 -- 172.21.15.27:0/3502580068 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 msgr2=0x563c3a2ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89f38640 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.653+0000 7f3a89f38640 1 -- 172.21.15.27:0/3502580068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c3a0066b0 con 0x563c3a2c9fa0 2024-04-25T01:23:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a89f38640 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5d00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3a80098320 tx=0x7f3a8009c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24b326204877a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:23:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a6cff9640 1 -- 172.21.15.27:0/3502580068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8001d040 con 0x563c3a2c9fa0 2024-04-25T01:23:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a6cff9640 1 -- 172.21.15.27:0/3502580068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3a80086070 con 0x563c3a2c9fa0 2024-04-25T01:23:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 1 -- 172.21.15.27:0/3502580068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a7c0318b0 con 0x563c3a2c9fa0 2024-04-25T01:23:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a6cff9640 1 -- 172.21.15.27:0/3502580068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3a800a45c0 con 0x563c3a2c9fa0 2024-04-25T01:23:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 10 client.14772 fetch_fsmap learned FSMap version 17 2024-04-25T01:23:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 1 -- 172.21.15.27:0/3502580068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x563c3a2d8db0 con 0x563c3a2c9fa0 2024-04-25T01:23:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a6cff9640 1 -- 172.21.15.27:0/3502580068 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a7c0318b0 con 0x563c3a2c9fa0 2024-04-25T01:23:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a6cff9640 1 -- 172.21.15.27:0/3502580068 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f3a8009ada0 con 0x563c3a2c9fa0 2024-04-25T01:23:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 10 client.14772 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T01:23:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 10 client.14772 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:23:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 20 client.14772 populate_metadata read hostname 'smithi027' 2024-04-25T01:23:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c3a2db480 0x563c3a2dc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 4 client.14772 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:23:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.654+0000 7f3a8e783740 1 -- 172.21.15.27:0/3502580068 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c3a235440 con 0x563c3a2db480 2024-04-25T01:23:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.655+0000 7f3a89737640 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c3a2db480 0x563c3a2dc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.655+0000 7f3a89737640 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c3a2db480 0x563c3a2dc8f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=692ec36255815476 server_cookie=3aa5b9a341e71338 in_seq=0 out_seq=0 2024-04-25T01:23:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.655+0000 7f3a6cff9640 10 client.14772 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:23:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a6cff9640 1 -- 172.21.15.27:0/3502580068 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c3a235440 con 0x563c3a2db480 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a6cff9640 10 client.14772 handle_command_reply: tid=0 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 client.14772 shutdown 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 -- 172.21.15.27:0/3502580068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c9fa0 msgr2=0x563c3a2d5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5d00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3a80098320 tx=0x7f3a8009c190 comp rx=0 tx=0).stop 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 -- 172.21.15.27:0/3502580068 shutdown_connections 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c3a2db480 0x563c3a2dc8f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5d00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c4e30 0x563c3a2cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3502580068 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a6cff9640 0 client.14772 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 -- 172.21.15.27:0/3502580068 >> 172.21.15.27:0/3502580068 conn(0x563c3a2bfab0 msgr2=0x563c3a2beae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 -- 172.21.15.27:0/3502580068 shutdown_connections 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 1 -- 172.21.15.27:0/3502580068 wait complete. 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.656+0000 7f3a8e783740 20 client.14772 trim_cache size 0 max 16384 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a8e783740 1 Processor -- start 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a8e783740 1 -- start start 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c4440 0x563c3a2c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a8e783740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563c3a2dac30 con 0x563c3a2c4e30 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a8e783740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563c3a2db300 con 0x563c3a2c9fa0 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a8e783740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563c3a2db9d0 con 0x563c3a2c4440 2024-04-25T01:23:04.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50176/0 (socket says 172.21.15.27:50176) 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 -- 172.21.15.27:0/3806672684 learned_addr learned my addr 172.21.15.27:0/3806672684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 -- 172.21.15.27:0/3806672684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c4440 msgr2=0x563c3a2c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 --2- 172.21.15.27:0/3806672684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c4440 0x563c3a2c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a89f38640 1 --2- 172.21.15.27:0/3806672684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 -- 172.21.15.27:0/3806672684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c9fa0 msgr2=0x563c3a2d5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 --2- 172.21.15.27:0/3806672684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 -- 172.21.15.27:0/3806672684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c3a2c56c0 con 0x563c3a2c4e30 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a88f36640 1 --2- 172.21.15.27:0/3806672684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c6730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400aba0 tx=0x7f3a74006da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36e64d45b85aec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.675+0000 7f3a6effd640 1 -- 172.21.15.27:0/3806672684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74028070 con 0x563c3a2c4e30 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a6effd640 1 -- 172.21.15.27:0/3806672684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a740188f0 con 0x563c3a2c4e30 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a6effd640 1 -- 172.21.15.27:0/3806672684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a740238e0 con 0x563c3a2c4e30 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 -- 172.21.15.27:0/3806672684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 msgr2=0x563c3a2c6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3806672684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c6730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400aba0 tx=0x7f3a74006da0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 -- 172.21.15.27:0/3806672684 shutdown_connections 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3806672684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3806672684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2c6730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 --2- 172.21.15.27:0/3806672684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c4440 0x563c3a2c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 -- 172.21.15.27:0/3806672684 >> 172.21.15.27:0/3806672684 conn(0x563c3a2bfab0 msgr2=0x563c3a2ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 -- 172.21.15.27:0/3806672684 shutdown_connections 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 -- 172.21.15.27:0/3806672684 wait complete. 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 -- start start 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.676+0000 7f3a8e783740 1 Processor -- start 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2d2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a8e783740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a8e783740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563c3a2dc770 con 0x563c3a2c4e30 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a8e783740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563c3a2dc0a0 con 0x563c3a2c4440 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a8e783740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563c3a2dce40 con 0x563c3a2c9fa0 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a8e783740 10 client.0 _ll_register_callbacks cb 0x7f3a8009b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2d2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2d2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50186/0 (socket says 172.21.15.27:50186) 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 -- 172.21.15.27:0/979206322 learned_addr learned my addr 172.21.15.27:0/979206322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a89737640 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 -- 172.21.15.27:0/979206322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 msgr2=0x563c3a2d51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 -- 172.21.15.27:0/979206322 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 msgr2=0x563c3a2d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 -- 172.21.15.27:0/979206322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c3a2d5850 con 0x563c3a2c4e30 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a88f36640 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2d2390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3a74004810 tx=0x7f3a74018640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f84cf31327e6a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.677+0000 7f3a6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a6cff9640 1 -- 172.21.15.27:0/979206322 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74028050 con 0x563c3a2c4e30 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a6cff9640 1 -- 172.21.15.27:0/979206322 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3a74018cb0 con 0x563c3a2c4e30 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 1 -- 172.21.15.27:0/979206322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a740188f0 con 0x563c3a2c4e30 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a6cff9640 1 -- 172.21.15.27:0/979206322 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3a740059c0 con 0x563c3a2c4e30 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 10 client.14778 fetch_fsmap learned FSMap version 17 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 1 -- 172.21.15.27:0/979206322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x563c3a341cd0 con 0x563c3a2c4e30 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a6cff9640 1 -- 172.21.15.27:0/979206322 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a740188f0 con 0x563c3a2c4e30 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a6cff9640 1 -- 172.21.15.27:0/979206322 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f3a74015040 con 0x563c3a2c4e30 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 10 client.14778 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 10 client.14778 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 20 client.14778 populate_metadata read hostname 'smithi027' 2024-04-25T01:23:04.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c3a346630 0x563c3a342a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 4 client.14778 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a8e783740 1 -- 172.21.15.27:0/979206322 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "status"}) -- 0x563c3a22f790 con 0x563c3a346630 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.678+0000 7f3a89737640 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c3a346630 0x563c3a342a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a89737640 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c3a346630 0x563c3a342a70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e84f183420435b75 server_cookie=201cd5cc89be16da in_seq=0 out_seq=0 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a6cff9640 10 client.14778 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a6cff9640 1 -- 172.21.15.27:0/979206322 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x563c3a22f790 con 0x563c3a346630 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a6cff9640 10 client.14778 handle_command_reply: tid=0 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 client.14778 shutdown 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 -- 172.21.15.27:0/979206322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 msgr2=0x563c3a2d2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2d2390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3a74004810 tx=0x7f3a74018640 comp rx=0 tx=0).stop 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 -- 172.21.15.27:0/979206322 shutdown_connections 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c3a346630 0x563c3a342a70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c3a2c9fa0 0x563c3a2d51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c3a2c4e30 0x563c3a2d2390 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 --2- 172.21.15.27:0/979206322 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c3a2c4440 0x563c3a2d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a6cff9640 0 client.14778 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:23:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 -- 172.21.15.27:0/979206322 >> 172.21.15.27:0/979206322 conn(0x563c3a2bfab0 msgr2=0x563c3a2ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 -- 172.21.15.27:0/979206322 shutdown_connections 2024-04-25T01:23:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 1 -- 172.21.15.27:0/979206322 wait complete. 2024-04-25T01:23:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.679+0000 7f3a8e783740 20 client.14778 trim_cache size 0 max 16384 2024-04-25T01:23:04.690 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:23:04.690 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24521, 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.8568050659999997, 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 17, 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 89, 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.254487907 2024-04-25T01:23:04.691 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:23:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 -- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3a700a2b90 msgr2=0x7f3a700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3a700a2b90 0x7f3a700a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a74006fd0 tx=0x7f3a7400a230 comp rx=0 tx=0).stop 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 -- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a84105310 msgr2=0x7f3a841a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a84105310 0x7f3a841a56f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000cad0 tx=0x7f3a8000cf90 comp rx=0 tx=0).stop 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 -- 172.21.15.27:0/690285720 shutdown_connections 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3a700a2b90 0x7f3a700a5050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a84105310 0x7f3a841a56f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a84102960 0x7f3a8419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 --2- 172.21.15.27:0/690285720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a840ffff0 0x7f3a8419e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 -- 172.21.15.27:0/690285720 >> 172.21.15.27:0/690285720 conn(0x7f3a840fbbf0 msgr2=0x7f3a841033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 -- 172.21.15.27:0/690285720 shutdown_connections 2024-04-25T01:23:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:04.680+0000 7f3a8b99b640 1 -- 172.21.15.27:0/690285720 wait complete. 2024-04-25T01:23:04.693 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:23:04.694 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:23:04.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T01:23:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:05.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:04 smithi027 ceph-mon[29197]: pgmap v298: 129 pgs: 8 creating+peering, 113 active+clean, 8 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2024-04-25T01:23:05.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:04 smithi027 ceph-mon[29197]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:23:05.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:04 smithi027 ceph-mon[29197]: Cluster is now healthy 2024-04-25T01:23:05.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/870699131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:05.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1196084497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:05.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:04 smithi153 ceph-mon[32621]: pgmap v298: 129 pgs: 8 creating+peering, 113 active+clean, 8 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2024-04-25T01:23:05.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:04 smithi153 ceph-mon[32621]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:23:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:04 smithi153 ceph-mon[32621]: Cluster is now healthy 2024-04-25T01:23:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:04 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/870699131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:05.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:04 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1196084497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:05.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:04 smithi139 ceph-mon[32760]: pgmap v298: 129 pgs: 8 creating+peering, 113 active+clean, 8 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2024-04-25T01:23:05.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:04 smithi139 ceph-mon[32760]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:23:05.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:04 smithi139 ceph-mon[32760]: Cluster is now healthy 2024-04-25T01:23:05.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:04 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/870699131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:05.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:04 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1196084497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.624+0000 7f10cc39b700 1 -- 172.21.15.27:0/2794288934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c4107590 msgr2=0x7f10c410f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.624+0000 7f10cc39b700 1 --2- 172.21.15.27:0/2794288934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c4107590 0x7f10c410f160 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0099c0 tx=0x7f10bc009cd0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 -- 172.21.15.27:0/2794288934 shutdown_connections 2024-04-25T01:23:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 --2- 172.21.15.27:0/2794288934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c4107590 0x7f10c410f160 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 --2- 172.21.15.27:0/2794288934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10c406ac40 0x7f10c406b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 --2- 172.21.15.27:0/2794288934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10c406b5f0 0x7f10c4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 -- 172.21.15.27:0/2794288934 >> 172.21.15.27:0/2794288934 conn(0x7f10c40781d0 msgr2=0x7f10c40785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 -- 172.21.15.27:0/2794288934 shutdown_connections 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 -- 172.21.15.27:0/2794288934 wait complete. 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 Processor -- start 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.625+0000 7f10cc39b700 1 -- start start 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10cc39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c406ac40 0x7f10c419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10cc39b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10c406b5f0 0x7f10c419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10cc39b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10c4107590 0x7f10c4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10cc39b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f10c4112590 con 0x7f10c406ac40 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10cc39b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f10c4112410 con 0x7f10c406b5f0 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10cc39b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10c4112710 con 0x7f10c4107590 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c406ac40 0x7f10c419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c406ac40 0x7f10c419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50208/0 (socket says 172.21.15.27:50208) 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 -- 172.21.15.27:0/4181163772 learned_addr learned my addr 172.21.15.27:0/4181163772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca938700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10c4107590 0x7f10c4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10c9936700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10c406b5f0 0x7f10c419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 -- 172.21.15.27:0/4181163772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10c4107590 msgr2=0x7f10c4198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10c4107590 0x7f10c4198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 -- 172.21.15.27:0/4181163772 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10c406b5f0 msgr2=0x7f10c419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10c406b5f0 0x7f10c419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 -- 172.21.15.27:0/4181163772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c4199070 con 0x7f10c406ac40 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10c9936700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10c406b5f0 0x7f10c419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10ca137700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c406ac40 0x7f10c419e670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f10c0007e20 tx=0x7f10c000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.626+0000 7f10bb7fe700 1 -- 172.21.15.27:0/4181163772 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c0016020 con 0x7f10c406ac40 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.627+0000 7f10bb7fe700 1 -- 172.21.15.27:0/4181163772 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10c000ad40 con 0x7f10c406ac40 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.627+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c4199360 con 0x7f10c406ac40 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.627+0000 7f10bb7fe700 1 -- 172.21.15.27:0/4181163772 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c0011700 con 0x7f10c406ac40 2024-04-25T01:23:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.627+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c41a55f0 con 0x7f10c406ac40 2024-04-25T01:23:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.628+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c4103990 con 0x7f10c406ac40 2024-04-25T01:23:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.629+0000 7f10bb7fe700 1 -- 172.21.15.27:0/4181163772 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f10c0019020 con 0x7f10c406ac40 2024-04-25T01:23:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.629+0000 7f10bb7fe700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f10b00a2c20 0x7f10b00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.629+0000 7f10c9936700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f10b00a2c20 0x7f10b00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.630+0000 7f10bb7fe700 1 -- 172.21.15.27:0/4181163772 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f10c00cefe0 con 0x7f10c406ac40 2024-04-25T01:23:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.630+0000 7f10c9936700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f10b00a2c20 0x7f10b00a50e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10b4005e10 tx=0x7f10b4005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.634+0000 7f10bb7fe700 1 -- 172.21.15.27:0/4181163772 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f10c009bd00 con 0x7f10c406ac40 2024-04-25T01:23:06.870 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:23:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.889+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f10c402db30 con 0x7f10c406ac40 2024-04-25T01:23:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.889+0000 7f10bb7fe700 1 -- 172.21.15.27:0/4181163772 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f10c009b600 con 0x7f10c406ac40 2024-04-25T01:23:06.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f10b00a2c20 msgr2=0x7f10b00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f10b00a2c20 0x7f10b00a50e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10b4005e10 tx=0x7f10b4005d60 comp rx=0 tx=0).stop 2024-04-25T01:23:06.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c406ac40 msgr2=0x7f10c419e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:06.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c406ac40 0x7f10c419e670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f10c0007e20 tx=0x7f10c000a340 comp rx=0 tx=0).stop 2024-04-25T01:23:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 shutdown_connections 2024-04-25T01:23:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f10b00a2c20 0x7f10b00a50e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10c4107590 0x7f10c4198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c406ac40 0x7f10c419e670 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 --2- 172.21.15.27:0/4181163772 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10c406b5f0 0x7f10c419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 >> 172.21.15.27:0/4181163772 conn(0x7f10c40781d0 msgr2=0x7f10c40ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 shutdown_connections 2024-04-25T01:23:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:06.892+0000 7f10cc39b700 1 -- 172.21.15.27:0/4181163772 wait complete. 2024-04-25T01:23:07.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:06 smithi027 ceph-mon[29197]: pgmap v299: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-25T01:23:07.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:06 smithi153 ceph-mon[32621]: pgmap v299: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-25T01:23:07.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:06 smithi139 ceph-mon[32760]: pgmap v299: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:23:07.584 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T01:23:07.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:07.585 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-04-25T01:23:07.602 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:23:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:23:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:23:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-04-25T01:23:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:07.602 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:07.602 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42218sec preferred_lft 42218sec 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-04-25T01:23:07.603 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-04-25T01:23:07.604 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:23:07.604 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:23:07.604 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:23:07.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:23:07.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-04-25T01:23:07.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-04-25T01:23:07.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:23:07.604 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:23:07.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:23:07.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:07.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:07.789 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:23:07.814 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-04-25T01:23:07.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:07.815 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-04-25T01:23:07.893 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-04-25T01:23:07.893 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:23:07.893 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T01:23:07.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T01:23:07.893 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:23:07.893 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:23:07.894 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:23:07.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T01:23:07.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T01:23:07.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T01:23:07.894 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:23:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:23:08.098 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:07 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4181163772' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:23:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:08.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:08.241 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T01:23:08.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:08.256 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-04-25T01:23:08.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:07 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4181163772' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:23:08.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:07 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4181163772' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:23:08.311 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:23:08.311 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:23:08.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:23:08.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T01:23:08.312 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:23:08.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:23:08.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:08.432 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T01:23:08.433 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:23:08.433 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:23:08.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T01:23:08.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:23:08.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T01:23:08.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T01:23:08.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T01:23:08.433 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:23:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:23:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:08.559 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:23:08.559 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:23:08.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-04-25T01:23:08.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:23:08.559 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:23:08.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:23:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:08.637 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T01:23:08.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:23:08.637 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:23:08.653 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T01:23:08.653 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:23:08.653 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:23:08.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-25T01:23:08.775 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-25T01:23:08.831 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:23:08.832 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/3fef4672961732b15caee083f459d74d65cc388e30468e4f023d78323bfcc95c/merged 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/fda01ae58f9c57e1a10355b434ead700bb5fcdb87d436292424f08ebe9510233/merged 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/cba12d79119209a25e90228048211192c327c596eb105b6195402a7b3b4a4bf5/merged 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4a887ae14bfc59f354b5b136e2e4ef0687ee2465785522ddaf30e356464af995/merged 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/07df4e9ba80d2e23df9a8cebe1d7570bf7391b26e5accaf530c37764bd8e8987/merged 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/bfbfa273e8d6bf77c86ffd70d3c0b36915b6883af7579f9eae30761370d4fe7f/merged 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/8d8bca7a8c194c26b2fdac3eb35ce07d55a3434f9135405483d30e26ed59ebcf/merged 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4db3c633dbf9a4be1af9521556b7f3a2ec18884fbf8a1bf153c7d54e443b19e0/merged 2024-04-25T01:23:08.833 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/84d36ac5b7227f67c63337ce02c0985e265dd84aa1f654c2d0511df3af10e8b4/merged 2024-04-25T01:23:08.834 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/2c4b721c1899dad8983b0e9d6a4393340d3655022ebf127a032f2856be9055f8/merged 2024-04-25T01:23:08.834 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-25T01:23:08.834 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:23:08.834 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:23:08.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:08.834 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-25T01:23:08.886 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T01:23:08.886 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T01:23:08.928 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-25T01:23:08.953 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:23:09.010 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-25T01:23:09.010 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-25T01:23:09.010 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-25T01:23:09.010 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-25T01:23:09.010 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-25T01:23:09.010 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-25T01:23:09.010 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-25T01:23:09.011 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/3fef4672961732b15caee083f459d74d65cc388e30468e4f023d78323bfcc95c/merged 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/fda01ae58f9c57e1a10355b434ead700bb5fcdb87d436292424f08ebe9510233/merged 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/cba12d79119209a25e90228048211192c327c596eb105b6195402a7b3b4a4bf5/merged 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4a887ae14bfc59f354b5b136e2e4ef0687ee2465785522ddaf30e356464af995/merged 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/07df4e9ba80d2e23df9a8cebe1d7570bf7391b26e5accaf530c37764bd8e8987/merged 2024-04-25T01:23:09.012 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/bfbfa273e8d6bf77c86ffd70d3c0b36915b6883af7579f9eae30761370d4fe7f/merged 2024-04-25T01:23:09.013 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/8d8bca7a8c194c26b2fdac3eb35ce07d55a3434f9135405483d30e26ed59ebcf/merged 2024-04-25T01:23:09.013 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/4db3c633dbf9a4be1af9521556b7f3a2ec18884fbf8a1bf153c7d54e443b19e0/merged 2024-04-25T01:23:09.013 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/84d36ac5b7227f67c63337ce02c0985e265dd84aa1f654c2d0511df3af10e8b4/merged 2024-04-25T01:23:09.013 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/2c4b721c1899dad8983b0e9d6a4393340d3655022ebf127a032f2856be9055f8/merged 2024-04-25T01:23:09.013 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-25T01:23:09.013 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:23:09.013 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:23:09.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:09.014 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-25T01:23:09.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:08 smithi153 ceph-mon[32621]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-25T01:23:09.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:2024-04-25T01:23:09.048+0000 7fec15bc5480 -1 init, newargv = 0x5616017f6210 newargc=15 2024-04-25T01:23:09.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[73103]: starting ceph client 2024-04-25T01:23:09.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[73103]: starting fuse 2024-04-25T01:23:09.045 INFO:teuthology.orchestra.run.smithi027.stdout:115 2024-04-25T01:23:09.045 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-25T01:23:09.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-04-25T01:23:09.045 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-04-25T01:23:09.045 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-04-25T01:23:09.045 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-04-25T01:23:09.045 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-04-25T01:23:09.045 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T01:23:09.046 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T01:23:09.047 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-04-25T01:23:09.047 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-04-25T01:23:09.047 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-04-25T01:23:09.047 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T01:23:09.047 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-25T01:23:09.047 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.admin")) 2024-04-25T01:23:09.047 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-04-25T01:23:09.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:08 smithi027 ceph-mon[29197]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-25T01:23:09.165 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.admin.73103.asok 2024-04-25T01:23:09.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:09.176 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.73103.asok 2024-04-25T01:23:09.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:09.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.73103.asok status 2024-04-25T01:23:09.257 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:08 smithi139 ceph-mon[32760]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-25T01:23:09.313 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:23:09.313 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-04-25T01:23:09.313 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T01:23:09.313 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "admin", 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "73103", 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 14790, 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-04-25T01:23:09.314 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 14790 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3612715387 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3612715387 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.14790 192.168.0.1:0/3612715387", 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3612715387", 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-04-25T01:23:09.315 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 17, 2024-04-25T01:23:09.316 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 89, 2024-04-25T01:23:09.316 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-04-25T01:23:09.316 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-04-25T01:23:09.316 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-04-25T01:23:09.316 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:23:09.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:09.322 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-04-25T01:23:09.322 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:23:09.338 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-04-25T01:23:09.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:23:09.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:09.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:23:09.404 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T01:23:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:23:09.526 INFO:teuthology.orchestra.run.smithi027.stdout:# file: . 2024-04-25T01:23:09.526 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T01:23:09.526 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:23:09.532 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:23:09.532 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:09.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 -- 172.21.15.27:0/1576216570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 msgr2=0x7fb010101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 --2- 172.21.15.27:0/1576216570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 0x7fb010101d10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb000009900 tx=0x7fb00002a410 comp rx=0 tx=0).stop 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 -- 172.21.15.27:0/1576216570 shutdown_connections 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 --2- 172.21.15.27:0/1576216570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb010102250 0x7fb010108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 --2- 172.21.15.27:0/1576216570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 0x7fb010101d10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 --2- 172.21.15.27:0/1576216570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0101006a0 0x7fb010100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 -- 172.21.15.27:0/1576216570 >> 172.21.15.27:0/1576216570 conn(0x7fb0100fbc30 msgr2=0x7fb0100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 -- 172.21.15.27:0/1576216570 shutdown_connections 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.710+0000 7fb0174a3640 1 -- 172.21.15.27:0/1576216570 wait complete. 2024-04-25T01:23:09.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb0174a3640 1 Processor -- start 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb0174a3640 1 -- start start 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb0174a3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0101006a0 0x7fb01006a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb0174a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 0x7fb01019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb0174a3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb010102250 0x7fb01019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb0174a3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb01010c320 con 0x7fb010101890 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb0174a3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb01010c1a0 con 0x7fb010102250 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb0174a3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb01010c4a0 con 0x7fb0101006a0 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb014a3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 0x7fb01019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb014a3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 0x7fb01019c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60232/0 (socket says 172.21.15.27:60232) 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb014a3e640 1 -- 172.21.15.27:0/4058572971 learned_addr learned my addr 172.21.15.27:0/4058572971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb015a40640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb010102250 0x7fb01019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb014a3e640 1 -- 172.21.15.27:0/4058572971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0101006a0 msgr2=0x7fb01006a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb01523f640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0101006a0 0x7fb01006a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb014a3e640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0101006a0 0x7fb01006a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb014a3e640 1 -- 172.21.15.27:0/4058572971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb010102250 msgr2=0x7fb01019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.711+0000 7fb014a3e640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb010102250 0x7fb01019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.712+0000 7fb014a3e640 1 -- 172.21.15.27:0/4058572971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01019f720 con 0x7fb010101890 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.712+0000 7fb014a3e640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 0x7fb01019c7f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb000009230 tx=0x7fb000004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.712+0000 7fb0067fc640 1 -- 172.21.15.27:0/4058572971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00003f030 con 0x7fb010101890 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.712+0000 7fb0067fc640 1 -- 172.21.15.27:0/4058572971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb000003ce0 con 0x7fb010101890 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.712+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01019f950 con 0x7fb010101890 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.712+0000 7fb0067fc640 1 -- 172.21.15.27:0/4058572971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000035070 con 0x7fb010101890 2024-04-25T01:23:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.712+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb01019fe90 con 0x7fb010101890 2024-04-25T01:23:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.713+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb010107a40 con 0x7fb010101890 2024-04-25T01:23:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.714+0000 7fb0067fc640 1 -- 172.21.15.27:0/4058572971 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb000042030 con 0x7fb010101890 2024-04-25T01:23:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.715+0000 7fb0067fc640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faffc0a2c10 0x7faffc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.715+0000 7fb01523f640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faffc0a2c10 0x7faffc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.715+0000 7fb0067fc640 1 -- 172.21.15.27:0/4058572971 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb0000efc50 con 0x7fb010101890 2024-04-25T01:23:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.715+0000 7fb01523f640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faffc0a2c10 0x7faffc0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb00c005fd0 tx=0x7fb00c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:09.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.718+0000 7fb0067fc640 1 -- 172.21.15.27:0/4058572971 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0000bc910 con 0x7fb010101890 2024-04-25T01:23:09.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.908+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb01002d340 con 0x7fb010101890 2024-04-25T01:23:09.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.909+0000 7fb0067fc640 1 -- 172.21.15.27:0/4058572971 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7fb0000bc210 con 0x7fb010101890 2024-04-25T01:23:09.890 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:09.891 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:09.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faffc0a2c10 msgr2=0x7faffc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faffc0a2c10 0x7faffc0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb00c005fd0 tx=0x7fb00c005850 comp rx=0 tx=0).stop 2024-04-25T01:23:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 msgr2=0x7fb01019c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 0x7fb01019c7f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb000009230 tx=0x7fb000004360 comp rx=0 tx=0).stop 2024-04-25T01:23:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 shutdown_connections 2024-04-25T01:23:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faffc0a2c10 0x7faffc0a50d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb010102250 0x7fb01019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb010101890 0x7fb01019c7f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 --2- 172.21.15.27:0/4058572971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0101006a0 0x7fb01006a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:09.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 >> 172.21.15.27:0/4058572971 conn(0x7fb0100fbc30 msgr2=0x7fb010104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:09.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 shutdown_connections 2024-04-25T01:23:09.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:09.911+0000 7fb0174a3640 1 -- 172.21.15.27:0/4058572971 wait complete. 2024-04-25T01:23:09.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:23:09.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:23:10.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 -- 172.21.15.27:0/2713524099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c102250 msgr2=0x7f046c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 --2- 172.21.15.27:0/2713524099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c102250 0x7f046c108ae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0464009ca0 tx=0x7f046402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 -- 172.21.15.27:0/2713524099 shutdown_connections 2024-04-25T01:23:10.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 --2- 172.21.15.27:0/2713524099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c102250 0x7f046c108ae0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 --2- 172.21.15.27:0/2713524099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046c101890 0x7f046c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 --2- 172.21.15.27:0/2713524099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f046c1006a0 0x7f046c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 -- 172.21.15.27:0/2713524099 >> 172.21.15.27:0/2713524099 conn(0x7f046c0fbc30 msgr2=0x7f046c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 -- 172.21.15.27:0/2713524099 shutdown_connections 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 -- 172.21.15.27:0/2713524099 wait complete. 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.030+0000 7f047297d640 1 Processor -- start 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f047297d640 1 -- start start 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f047297d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f046c1006a0 0x7f046c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f047297d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c101890 0x7f046c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f047297d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046c102250 0x7f046c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f047297d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f046c10c300 con 0x7f046c101890 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f047297d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f046c10c180 con 0x7f046c1006a0 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f047297d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f046c10c480 con 0x7f046c102250 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c101890 0x7f046c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c101890 0x7f046c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60254/0 (socket says 172.21.15.27:60254) 2024-04-25T01:23:10.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 -- 172.21.15.27:0/1458081606 learned_addr learned my addr 172.21.15.27:0/1458081606 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046bfff640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f046c1006a0 0x7f046c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 -- 172.21.15.27:0/1458081606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046c102250 msgr2=0x7f046c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046c102250 0x7f046c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 -- 172.21.15.27:0/1458081606 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f046c1006a0 msgr2=0x7f046c1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f046c1006a0 0x7f046c1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 -- 172.21.15.27:0/1458081606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046c19f5f0 con 0x7f046c101890 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.031+0000 7f046b7fe640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c101890 0x7f046c197f10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f046000c920 tx=0x7f046000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.032+0000 7f04697fa640 1 -- 172.21.15.27:0/1458081606 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460007c70 con 0x7f046c101890 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.032+0000 7f04697fa640 1 -- 172.21.15.27:0/1458081606 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0460007e10 con 0x7f046c101890 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.032+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046c19f8e0 con 0x7f046c101890 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.032+0000 7f04697fa640 1 -- 172.21.15.27:0/1458081606 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460018890 con 0x7f046c101890 2024-04-25T01:23:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.032+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046c19fe30 con 0x7f046c101890 2024-04-25T01:23:10.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.033+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f046c1a00c0 con 0x7f046c101890 2024-04-25T01:23:10.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.034+0000 7f04697fa640 1 -- 172.21.15.27:0/1458081606 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0460018a70 con 0x7f046c101890 2024-04-25T01:23:10.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.035+0000 7f04697fa640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f04540a2b90 0x7f04540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.035+0000 7f046bfff640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f04540a2b90 0x7f04540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.035+0000 7f04697fa640 1 -- 172.21.15.27:0/1458081606 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f04600cf480 con 0x7f046c101890 2024-04-25T01:23:10.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.035+0000 7f046bfff640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f04540a2b90 0x7f04540a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f045c005fd0 tx=0x7f045c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.040+0000 7f04697fa640 1 -- 172.21.15.27:0/1458081606 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0460014030 con 0x7f046c101890 2024-04-25T01:23:10.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.222+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f046c1a0330 con 0x7f046c101890 2024-04-25T01:23:10.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.223+0000 7f04697fa640 1 -- 172.21.15.27:0/1458081606 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f046000de20 con 0x7f046c101890 2024-04-25T01:23:10.203 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:23:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f04540a2b90 msgr2=0x7f04540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f04540a2b90 0x7f04540a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f045c005fd0 tx=0x7f045c005850 comp rx=0 tx=0).stop 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c101890 msgr2=0x7f046c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c101890 0x7f046c197f10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f046000c920 tx=0x7f046000cde0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 shutdown_connections 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f04540a2b90 0x7f04540a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046c102250 0x7f046c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046c101890 0x7f046c197f10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 --2- 172.21.15.27:0/1458081606 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f046c1006a0 0x7f046c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 >> 172.21.15.27:0/1458081606 conn(0x7f046c0fbc30 msgr2=0x7f046c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 shutdown_connections 2024-04-25T01:23:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.225+0000 7f047297d640 1 -- 172.21.15.27:0/1458081606 wait complete. 2024-04-25T01:23:10.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:23:10.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.340+0000 7f4e2565d640 1 -- 172.21.15.27:0/693370299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20102980 msgr2=0x7f4e20102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.340+0000 7f4e2565d640 1 --2- 172.21.15.27:0/693370299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20102980 0x7f4e20102e00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009900 tx=0x7f4e0802a410 comp rx=0 tx=0).stop 2024-04-25T01:23:10.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.340+0000 7f4e2565d640 1 -- 172.21.15.27:0/693370299 shutdown_connections 2024-04-25T01:23:10.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.340+0000 7f4e2565d640 1 --2- 172.21.15.27:0/693370299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e20103340 0x7f4e20109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.340+0000 7f4e2565d640 1 --2- 172.21.15.27:0/693370299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20102980 0x7f4e20102e00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.340+0000 7f4e2565d640 1 --2- 172.21.15.27:0/693370299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e20101790 0x7f4e20101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.340+0000 7f4e2565d640 1 -- 172.21.15.27:0/693370299 >> 172.21.15.27:0/693370299 conn(0x7f4e200fcd20 msgr2=0x7f4e200ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.340+0000 7f4e2565d640 1 -- 172.21.15.27:0/693370299 shutdown_connections 2024-04-25T01:23:10.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 -- 172.21.15.27:0/693370299 wait complete. 2024-04-25T01:23:10.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 Processor -- start 2024-04-25T01:23:10.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 -- start start 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e20101790 0x7f4e20197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e20102980 0x7f4e20197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20103340 0x7f4e2019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2010d3f0 con 0x7f4e20103340 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2010d270 con 0x7f4e20101790 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.341+0000 7f4e2565d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2010d570 con 0x7f4e20102980 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20103340 0x7f4e2019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20103340 0x7f4e2019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60274/0 (socket says 172.21.15.27:60274) 2024-04-25T01:23:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1effd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e20101790 0x7f4e20197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 -- 172.21.15.27:0/1428400641 learned_addr learned my addr 172.21.15.27:0/1428400641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1e7fc640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e20102980 0x7f4e20197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 -- 172.21.15.27:0/1428400641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e20102980 msgr2=0x7f4e20197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e20102980 0x7f4e20197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 -- 172.21.15.27:0/1428400641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e20101790 msgr2=0x7f4e20197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e20101790 0x7f4e20197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 -- 172.21.15.27:0/1428400641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2019f6d0 con 0x7f4e20103340 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e1f7fe640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20103340 0x7f4e2019f060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000eb10 tx=0x7f4e1000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e17fff640 1 -- 172.21.15.27:0/1428400641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10004b20 con 0x7f4e20103340 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2019f9c0 con 0x7f4e20103340 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e17fff640 1 -- 172.21.15.27:0/1428400641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e10004cc0 con 0x7f4e20103340 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e17fff640 1 -- 172.21.15.27:0/1428400641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10010760 con 0x7f4e20103340 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.342+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2019fe90 con 0x7f4e20103340 2024-04-25T01:23:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.343+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e20108b80 con 0x7f4e20103340 2024-04-25T01:23:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.344+0000 7f4e17fff640 1 -- 172.21.15.27:0/1428400641 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4e10005020 con 0x7f4e20103340 2024-04-25T01:23:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.345+0000 7f4e17fff640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4e0c0a2b90 0x7f4e0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.345+0000 7f4e17fff640 1 -- 172.21.15.27:0/1428400641 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4e100cf8e0 con 0x7f4e20103340 2024-04-25T01:23:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.345+0000 7f4e1effd640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4e0c0a2b90 0x7f4e0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.346+0000 7f4e1effd640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4e0c0a2b90 0x7f4e0c0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4e201027e0 tx=0x7f4e1800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.350+0000 7f4e17fff640 1 -- 172.21.15.27:0/1428400641 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4e1009c600 con 0x7f4e20103340 2024-04-25T01:23:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.534+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4e2002d250 con 0x7f4e20103340 2024-04-25T01:23:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.534+0000 7f4e17fff640 1 -- 172.21.15.27:0/1428400641 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f4e1009bf00 con 0x7f4e20103340 2024-04-25T01:23:10.515 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4e0c0a2b90 msgr2=0x7f4e0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4e0c0a2b90 0x7f4e0c0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4e201027e0 tx=0x7f4e1800b390 comp rx=0 tx=0).stop 2024-04-25T01:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20103340 msgr2=0x7f4e2019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20103340 0x7f4e2019f060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000eb10 tx=0x7f4e1000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 shutdown_connections 2024-04-25T01:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4e0c0a2b90 0x7f4e0c0a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20103340 0x7f4e2019f060 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e20102980 0x7f4e20197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 --2- 172.21.15.27:0/1428400641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e20101790 0x7f4e20197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.536+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 >> 172.21.15.27:0/1428400641 conn(0x7f4e200fcd20 msgr2=0x7f4e20105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.537+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 shutdown_connections 2024-04-25T01:23:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.537+0000 7f4e2565d640 1 -- 172.21.15.27:0/1428400641 wait complete. 2024-04-25T01:23:10.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:23:10.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/1062114199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd041017a0 msgr2=0x7fdd04101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/1062114199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd041017a0 0x7fdd04101bc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009ca0 tx=0x7fdcf402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/1062114199 shutdown_connections 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/1062114199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd04103350 0x7fdd04109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/1062114199 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd04102990 0x7fdd04102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/1062114199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd041017a0 0x7fdd04101bc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/1062114199 >> 172.21.15.27:0/1062114199 conn(0x7fdd040fcd10 msgr2=0x7fdd040ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/1062114199 shutdown_connections 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/1062114199 wait complete. 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.652+0000 7fdd0c7a9640 1 Processor -- start 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0c7a9640 1 -- start start 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0c7a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd041017a0 0x7fdd04197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0c7a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd04102990 0x7fdd04198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0c7a9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd04103350 0x7fdd0419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0c7a9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0410d3f0 con 0x7fdd04102990 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0c7a9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0410d270 con 0x7fdd04103350 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0c7a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0410d570 con 0x7fdd041017a0 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd04102990 0x7fdd04198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd04102990 0x7fdd04198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60306/0 (socket says 172.21.15.27:60306) 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 -- 172.21.15.27:0/3491723348 learned_addr learned my addr 172.21.15.27:0/3491723348 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0ad46640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd04103350 0x7fdd0419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0a545640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd041017a0 0x7fdd04197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 -- 172.21.15.27:0/3491723348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd041017a0 msgr2=0x7fdd04197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd041017a0 0x7fdd04197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 -- 172.21.15.27:0/3491723348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd04103350 msgr2=0x7fdd0419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd04103350 0x7fdd0419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 -- 172.21.15.27:0/3491723348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0419f6f0 con 0x7fdd04102990 2024-04-25T01:23:10.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0a545640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd041017a0 0x7fdd04197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:10.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd09d44640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd04102990 0x7fdd04198010 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800e9f0 tx=0x7fdcf800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd037fe640 1 -- 172.21.15.27:0/3491723348 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf800cd80 con 0x7fdd04102990 2024-04-25T01:23:10.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd037fe640 1 -- 172.21.15.27:0/3491723348 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcf8010910 con 0x7fdd04102990 2024-04-25T01:23:10.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0419f9e0 con 0x7fdd04102990 2024-04-25T01:23:10.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.653+0000 7fdd037fe640 1 -- 172.21.15.27:0/3491723348 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8018990 con 0x7fdd04102990 2024-04-25T01:23:10.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.654+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0419ff80 con 0x7fdd04102990 2024-04-25T01:23:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.655+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd04108b30 con 0x7fdd04102990 2024-04-25T01:23:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.656+0000 7fdd037fe640 1 -- 172.21.15.27:0/3491723348 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdcf8010ac0 con 0x7fdd04102990 2024-04-25T01:23:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.657+0000 7fdd037fe640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdcec0a2c10 0x7fdcec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.657+0000 7fdd0a545640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdcec0a2c10 0x7fdcec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.657+0000 7fdd037fe640 1 -- 172.21.15.27:0/3491723348 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdcf80cf9c0 con 0x7fdd04102990 2024-04-25T01:23:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.657+0000 7fdd0a545640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdcec0a2c10 0x7fdcec0a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4006010 tx=0x7fdcf4031bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.660+0000 7fdd037fe640 1 -- 172.21.15.27:0/3491723348 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdcf809c6e0 con 0x7fdd04102990 2024-04-25T01:23:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.847+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdd0402d3c0 con 0x7fdd04102990 2024-04-25T01:23:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.847+0000 7fdd037fe640 1 -- 172.21.15.27:0/3491723348 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fdcf809bfe0 con 0x7fdd04102990 2024-04-25T01:23:10.828 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdcec0a2c10 msgr2=0x7fdcec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdcec0a2c10 0x7fdcec0a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4006010 tx=0x7fdcf4031bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd04102990 msgr2=0x7fdd04198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd04102990 0x7fdd04198010 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800e9f0 tx=0x7fdcf800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 shutdown_connections 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdcec0a2c10 0x7fdcec0a50d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd04103350 0x7fdd0419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd04102990 0x7fdd04198010 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 --2- 172.21.15.27:0/3491723348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd041017a0 0x7fdd04197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 >> 172.21.15.27:0/3491723348 conn(0x7fdd040fcd10 msgr2=0x7fdd04105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 shutdown_connections 2024-04-25T01:23:10.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.849+0000 7fdd0c7a9640 1 -- 172.21.15.27:0/3491723348 wait complete. 2024-04-25T01:23:10.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:23:10.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 -- 172.21.15.27:0/977286772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28103240 msgr2=0x7f4d28109ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 --2- 172.21.15.27:0/977286772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28103240 0x7f4d28109ad0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009ca0 tx=0x7f4d2002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 -- 172.21.15.27:0/977286772 shutdown_connections 2024-04-25T01:23:10.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 --2- 172.21.15.27:0/977286772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28103240 0x7f4d28109ad0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 --2- 172.21.15.27:0/977286772 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d28102880 0x7f4d28102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 --2- 172.21.15.27:0/977286772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d28101690 0x7f4d28101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 -- 172.21.15.27:0/977286772 >> 172.21.15.27:0/977286772 conn(0x7f4d280fcbe0 msgr2=0x7f4d280ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 -- 172.21.15.27:0/977286772 shutdown_connections 2024-04-25T01:23:10.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.965+0000 7f4d2d200640 1 -- 172.21.15.27:0/977286772 wait complete. 2024-04-25T01:23:10.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.966+0000 7f4d2d200640 1 Processor -- start 2024-04-25T01:23:10.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.966+0000 7f4d2d200640 1 -- start start 2024-04-25T01:23:10.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.966+0000 7f4d2d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28101690 0x7f4d28197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.966+0000 7f4d2d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d28102880 0x7f4d28197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.966+0000 7f4d2d200640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d28103240 0x7f4d2819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.966+0000 7f4d2d200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810d310 con 0x7f4d28101690 2024-04-25T01:23:10.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.966+0000 7f4d2d200640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810d190 con 0x7f4d28103240 2024-04-25T01:23:10.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.966+0000 7f4d2d200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810d490 con 0x7f4d28102880 2024-04-25T01:23:10.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28101690 0x7f4d28197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28101690 0x7f4d28197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60328/0 (socket says 172.21.15.27:60328) 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 -- 172.21.15.27:0/4224504090 learned_addr learned my addr 172.21.15.27:0/4224504090 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d2659c640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d28102880 0x7f4d28197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d2759e640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d28103240 0x7f4d2819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 -- 172.21.15.27:0/4224504090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d28102880 msgr2=0x7f4d28197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d28102880 0x7f4d28197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 -- 172.21.15.27:0/4224504090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d28103240 msgr2=0x7f4d2819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d28103240 0x7f4d2819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 -- 172.21.15.27:0/4224504090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2819f550 con 0x7f4d28101690 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d26d9d640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28101690 0x7f4d28197930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800d950 tx=0x7f4d1800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d0ffff640 1 -- 172.21.15.27:0/4224504090 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18017070 con 0x7f4d28101690 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d0ffff640 1 -- 172.21.15.27:0/4224504090 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d18009c30 con 0x7f4d28101690 2024-04-25T01:23:10.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2819f840 con 0x7f4d28101690 2024-04-25T01:23:10.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d0ffff640 1 -- 172.21.15.27:0/4224504090 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1800f600 con 0x7f4d28101690 2024-04-25T01:23:10.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.967+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d2819fd40 con 0x7f4d28101690 2024-04-25T01:23:10.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.968+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d281089b0 con 0x7f4d28101690 2024-04-25T01:23:10.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.970+0000 7f4d0ffff640 1 -- 172.21.15.27:0/4224504090 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4d1801a070 con 0x7f4d28101690 2024-04-25T01:23:10.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.970+0000 7f4d0ffff640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4d100a2b90 0x7f4d100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.970+0000 7f4d2659c640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4d100a2b90 0x7f4d100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.970+0000 7f4d0ffff640 1 -- 172.21.15.27:0/4224504090 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4d180cec50 con 0x7f4d28101690 2024-04-25T01:23:10.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.971+0000 7f4d2659c640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4d100a2b90 0x7f4d100a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4d28198e40 tx=0x7f4d1c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:10.974+0000 7f4d0ffff640 1 -- 172.21.15.27:0/4224504090 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d18009de0 con 0x7f4d28101690 2024-04-25T01:23:11.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.162+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4d28062340 con 0x7f4d28101690 2024-04-25T01:23:11.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.162+0000 7f4d0ffff640 1 -- 172.21.15.27:0/4224504090 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f4d1809b320 con 0x7f4d28101690 2024-04-25T01:23:11.143 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:23:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4d100a2b90 msgr2=0x7f4d100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4d100a2b90 0x7f4d100a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4d28198e40 tx=0x7f4d1c00b390 comp rx=0 tx=0).stop 2024-04-25T01:23:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28101690 msgr2=0x7f4d28197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28101690 0x7f4d28197930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800d950 tx=0x7f4d1800de10 comp rx=0 tx=0).stop 2024-04-25T01:23:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 shutdown_connections 2024-04-25T01:23:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4d100a2b90 0x7f4d100a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d28103240 0x7f4d2819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d28102880 0x7f4d28197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 --2- 172.21.15.27:0/4224504090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d28101690 0x7f4d28197930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 >> 172.21.15.27:0/4224504090 conn(0x7f4d280fcbe0 msgr2=0x7f4d28105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 shutdown_connections 2024-04-25T01:23:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.165+0000 7f4d2d200640 1 -- 172.21.15.27:0/4224504090 wait complete. 2024-04-25T01:23:11.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:10 smithi027 ceph-mon[29197]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-25T01:23:11.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:10 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4058572971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:11.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:10 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1458081606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.155 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:10 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1428400641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.156 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:10 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3491723348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:23:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 -- 172.21.15.27:0/1436457216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e81006a0 msgr2=0x7f73e8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1436457216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e81006a0 0x7f73e8100ac0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009900 tx=0x7f73d802a410 comp rx=0 tx=0).stop 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 -- 172.21.15.27:0/1436457216 shutdown_connections 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1436457216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e8102250 0x7f73e8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1436457216 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73e8101890 0x7f73e8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1436457216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e81006a0 0x7f73e8100ac0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 -- 172.21.15.27:0/1436457216 >> 172.21.15.27:0/1436457216 conn(0x7f73e80fbc30 msgr2=0x7f73e80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 -- 172.21.15.27:0/1436457216 shutdown_connections 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 -- 172.21.15.27:0/1436457216 wait complete. 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 Processor -- start 2024-04-25T01:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.280+0000 7f73ee57e640 1 -- start start 2024-04-25T01:23:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ee57e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73e81006a0 0x7f73e8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ee57e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e8101890 0x7f73e8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ee57e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e8102250 0x7f73e819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ee57e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f73e810d440 con 0x7f73e8102250 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ee57e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f73e810d2c0 con 0x7f73e81006a0 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ee57e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73e810d5c0 con 0x7f73e8101890 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e8102250 0x7f73e819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e8102250 0x7f73e819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60344/0 (socket says 172.21.15.27:60344) 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 -- 172.21.15.27:0/1970871215 learned_addr learned my addr 172.21.15.27:0/1970871215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73e7fff640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73e81006a0 0x7f73e8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73e77fe640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e8101890 0x7f73e8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 -- 172.21.15.27:0/1970871215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e8101890 msgr2=0x7f73e8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e8101890 0x7f73e8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 -- 172.21.15.27:0/1970871215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73e81006a0 msgr2=0x7f73e8197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73e81006a0 0x7f73e8197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 -- 172.21.15.27:0/1970871215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e819f6f0 con 0x7f73e8102250 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73e77fe640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e8101890 0x7f73e8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ecb1b640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e8102250 0x7f73e819f080 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f73e000eaa0 tx=0x7f73e000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73e57fa640 1 -- 172.21.15.27:0/1970871215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e000cdb0 con 0x7f73e8102250 2024-04-25T01:23:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73e57fa640 1 -- 172.21.15.27:0/1970871215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73e0005020 con 0x7f73e8102250 2024-04-25T01:23:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e819f9e0 con 0x7f73e8102250 2024-04-25T01:23:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73e57fa640 1 -- 172.21.15.27:0/1970871215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e00116b0 con 0x7f73e8102250 2024-04-25T01:23:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.281+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e819ff30 con 0x7f73e8102250 2024-04-25T01:23:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.283+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e8107a30 con 0x7f73e8102250 2024-04-25T01:23:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.284+0000 7f73e57fa640 1 -- 172.21.15.27:0/1970871215 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f73e0005560 con 0x7f73e8102250 2024-04-25T01:23:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.284+0000 7f73e57fa640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f73d40a2b90 0x7f73d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.285+0000 7f73e7fff640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f73d40a2b90 0x7f73d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.285+0000 7f73e57fa640 1 -- 172.21.15.27:0/1970871215 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f73e00cf4f0 con 0x7f73e8102250 2024-04-25T01:23:11.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.285+0000 7f73e7fff640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f73d40a2b90 0x7f73d40a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f73d8005ab0 tx=0x7f73d8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.289+0000 7f73e57fa640 1 -- 172.21.15.27:0/1970871215 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f73e0016030 con 0x7f73e8102250 2024-04-25T01:23:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:10 smithi153 ceph-mon[32621]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-25T01:23:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:10 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4058572971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:10 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1458081606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:10 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1428400641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:10 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3491723348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:10 smithi139 ceph-mon[32760]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-25T01:23:11.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:10 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4058572971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:11.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:10 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1458081606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:10 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1428400641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:10 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3491723348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:23:11.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.487+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f73e8062340 con 0x7f73e8102250 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.487+0000 7f73e57fa640 1 -- 172.21.15.27:0/1970871215 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+586 (secure 0 0 0) 0x7f73e009c210 con 0x7f73e8102250 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/12403185 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/354957039 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/3353153293 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/234792808 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3645132749 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/219526886 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/219526886 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2409557466 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4191124222 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:11.468 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/3353153293 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:listed 10 entries 2024-04-25T01:23:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.489+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f73d40a2b90 msgr2=0x7f73d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.489+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f73d40a2b90 0x7f73d40a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f73d8005ab0 tx=0x7f73d8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.489+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e8102250 msgr2=0x7f73e819f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.489+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e8102250 0x7f73e819f080 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f73e000eaa0 tx=0x7f73e000ef60 comp rx=0 tx=0).stop 2024-04-25T01:23:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.490+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 shutdown_connections 2024-04-25T01:23:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.490+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f73d40a2b90 0x7f73d40a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.490+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e8102250 0x7f73e819f080 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.490+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e8101890 0x7f73e8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.490+0000 7f73ee57e640 1 --2- 172.21.15.27:0/1970871215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73e81006a0 0x7f73e8197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.490+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 >> 172.21.15.27:0/1970871215 conn(0x7f73e80fbc30 msgr2=0x7f73e8104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.490+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 shutdown_connections 2024-04-25T01:23:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.490+0000 7f73ee57e640 1 -- 172.21.15.27:0/1970871215 wait complete. 2024-04-25T01:23:11.481 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:23:11.481 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:23:11.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.658+0000 7fa81c24d640 1 -- 172.21.15.27:0/2952444277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814101310 msgr2=0x7fa814101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.658+0000 7fa81c24d640 1 --2- 172.21.15.27:0/2952444277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814101310 0x7fa814101790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa810009ca0 tx=0x7fa81002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.658+0000 7fa81c24d640 1 -- 172.21.15.27:0/2952444277 shutdown_connections 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.658+0000 7fa81c24d640 1 --2- 172.21.15.27:0/2952444277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814101310 0x7fa814101790 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.658+0000 7fa81c24d640 1 --2- 172.21.15.27:0/2952444277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa814100970 0x7fa814100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.658+0000 7fa81c24d640 1 --2- 172.21.15.27:0/2952444277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa814100010 0x7fa814100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.658+0000 7fa81c24d640 1 -- 172.21.15.27:0/2952444277 >> 172.21.15.27:0/2952444277 conn(0x7fa8140fbc30 msgr2=0x7fa8140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 -- 172.21.15.27:0/2952444277 shutdown_connections 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 -- 172.21.15.27:0/2952444277 wait complete. 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 Processor -- start 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 -- start start 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa814100010 0x7fa814199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814100970 0x7fa81419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa814101310 0x7fa8141a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa81410f5f0 con 0x7fa814100970 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa81410f470 con 0x7fa814100010 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa81c24d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa81410f770 con 0x7fa814101310 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa8197e8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814100970 0x7fa81419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa8197e8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814100970 0x7fa81419a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60372/0 (socket says 172.21.15.27:60372) 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa8197e8640 1 -- 172.21.15.27:0/3975823526 learned_addr learned my addr 172.21.15.27:0/3975823526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.659+0000 7fa819fe9640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa814100010 0x7fa814199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa81a7ea640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa814101310 0x7fa8141a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa8197e8640 1 -- 172.21.15.27:0/3975823526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa814101310 msgr2=0x7fa8141a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa8197e8640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa814101310 0x7fa8141a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa8197e8640 1 -- 172.21.15.27:0/3975823526 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa814100010 msgr2=0x7fa814199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa8197e8640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa814100010 0x7fa814199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa8197e8640 1 -- 172.21.15.27:0/3975823526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8141a1860 con 0x7fa814100970 2024-04-25T01:23:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa81a7ea640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa814101310 0x7fa8141a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa8197e8640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814100970 0x7fa81419a190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa80400d950 tx=0x7fa80400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa80affd640 1 -- 172.21.15.27:0/3975823526 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa804017070 con 0x7fa814100970 2024-04-25T01:23:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa80affd640 1 -- 172.21.15.27:0/3975823526 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa804009c30 con 0x7fa814100970 2024-04-25T01:23:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8141a1b50 con 0x7fa814100970 2024-04-25T01:23:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa80affd640 1 -- 172.21.15.27:0/3975823526 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80400f6b0 con 0x7fa814100970 2024-04-25T01:23:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.660+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8141a20f0 con 0x7fa814100970 2024-04-25T01:23:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.661+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81410ad60 con 0x7fa814100970 2024-04-25T01:23:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.662+0000 7fa80affd640 1 -- 172.21.15.27:0/3975823526 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa8040109d0 con 0x7fa814100970 2024-04-25T01:23:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.663+0000 7fa80affd640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8000a2c10 0x7fa8000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.663+0000 7fa819fe9640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8000a2c10 0x7fa8000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.663+0000 7fa80affd640 1 -- 172.21.15.27:0/3975823526 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa8040ceed0 con 0x7fa814100970 2024-04-25T01:23:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.663+0000 7fa819fe9640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8000a2c10 0x7fa8000a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa81419b120 tx=0x7fa80c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.666+0000 7fa80affd640 1 -- 172.21.15.27:0/3975823526 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa804009de0 con 0x7fa814100970 2024-04-25T01:23:11.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.852+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa81402d330 con 0x7fa814100970 2024-04-25T01:23:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.853+0000 7fa80affd640 1 -- 172.21.15.27:0/3975823526 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7fa80409bc70 con 0x7fa814100970 2024-04-25T01:23:11.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:11.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8000a2c10 msgr2=0x7fa8000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8000a2c10 0x7fa8000a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa81419b120 tx=0x7fa80c00b390 comp rx=0 tx=0).stop 2024-04-25T01:23:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814100970 msgr2=0x7fa81419a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814100970 0x7fa81419a190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa80400d950 tx=0x7fa80400de10 comp rx=0 tx=0).stop 2024-04-25T01:23:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 shutdown_connections 2024-04-25T01:23:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa8000a2c10 0x7fa8000a50d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa814101310 0x7fa8141a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa814100970 0x7fa81419a190 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 --2- 172.21.15.27:0/3975823526 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa814100010 0x7fa814199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 >> 172.21.15.27:0/3975823526 conn(0x7fa8140fbc30 msgr2=0x7fa8140fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:11.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 shutdown_connections 2024-04-25T01:23:11.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.855+0000 7fa81c24d640 1 -- 172.21.15.27:0/3975823526 wait complete. 2024-04-25T01:23:11.847 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:23:11.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:23:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 -- 172.21.15.27:0/9840707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80103350 msgr2=0x7f7c80109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 --2- 172.21.15.27:0/9840707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80103350 0x7f7c80109be0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009ca0 tx=0x7f7c7802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 -- 172.21.15.27:0/9840707 shutdown_connections 2024-04-25T01:23:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 --2- 172.21.15.27:0/9840707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80103350 0x7f7c80109be0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 --2- 172.21.15.27:0/9840707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c80102990 0x7f7c80102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 --2- 172.21.15.27:0/9840707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c801017a0 0x7f7c80101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 -- 172.21.15.27:0/9840707 >> 172.21.15.27:0/9840707 conn(0x7f7c800fcd10 msgr2=0x7f7c800ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 -- 172.21.15.27:0/9840707 shutdown_connections 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 -- 172.21.15.27:0/9840707 wait complete. 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.972+0000 7f7c86f14640 1 Processor -- start 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c86f14640 1 -- start start 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c86f14640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c801017a0 0x7f7c80197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c86f14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80102990 0x7f7c80197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c86f14640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c80103350 0x7f7c8019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c86f14640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8010d400 con 0x7f7c80102990 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c86f14640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8010d280 con 0x7f7c80103350 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c86f14640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8010d580 con 0x7f7c801017a0 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80102990 0x7f7c80197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80102990 0x7f7c80197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60376/0 (socket says 172.21.15.27:60376) 2024-04-25T01:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 -- 172.21.15.27:0/1681295059 learned_addr learned my addr 172.21.15.27:0/1681295059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c84cb0640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c801017a0 0x7f7c80197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c854b1640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c80103350 0x7f7c8019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 -- 172.21.15.27:0/1681295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c801017a0 msgr2=0x7f7c80197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c801017a0 0x7f7c80197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 -- 172.21.15.27:0/1681295059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c80103350 msgr2=0x7f7c8019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c80103350 0x7f7c8019eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 -- 172.21.15.27:0/1681295059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8019f550 con 0x7f7c80102990 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c854b1640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c80103350 0x7f7c8019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.973+0000 7f7c7ffff640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80102990 0x7f7c80197e70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800c920 tx=0x7f7c6800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.974+0000 7f7c7dffb640 1 -- 172.21.15.27:0/1681295059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68007c70 con 0x7f7c80102990 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.974+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c8019f840 con 0x7f7c80102990 2024-04-25T01:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.974+0000 7f7c7dffb640 1 -- 172.21.15.27:0/1681295059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c68007e10 con 0x7f7c80102990 2024-04-25T01:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.974+0000 7f7c7dffb640 1 -- 172.21.15.27:0/1681295059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68018890 con 0x7f7c80102990 2024-04-25T01:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.974+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c8019fd90 con 0x7f7c80102990 2024-04-25T01:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.975+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c801089f0 con 0x7f7c80102990 2024-04-25T01:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.976+0000 7f7c7dffb640 1 -- 172.21.15.27:0/1681295059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7c68018a70 con 0x7f7c80102990 2024-04-25T01:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.976+0000 7f7c7dffb640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7c6c0a2b90 0x7f7c6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.977+0000 7f7c84cb0640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7c6c0a2b90 0x7f7c6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.977+0000 7f7c7dffb640 1 -- 172.21.15.27:0/1681295059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7c680cf3d0 con 0x7f7c80102990 2024-04-25T01:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.977+0000 7f7c84cb0640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7c6c0a2b90 0x7f7c6c0a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7c801027f0 tx=0x7f7c70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:11.981+0000 7f7c7dffb640 1 -- 172.21.15.27:0/1681295059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7c68014030 con 0x7f7c80102990 2024-04-25T01:23:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.166+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7c80062340 con 0x7f7c80102990 2024-04-25T01:23:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.166+0000 7f7c7dffb640 1 -- 172.21.15.27:0/1681295059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f7c6809c0f0 con 0x7f7c80102990 2024-04-25T01:23:12.147 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7c6c0a2b90 msgr2=0x7f7c6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7c6c0a2b90 0x7f7c6c0a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7c801027f0 tx=0x7f7c70005f90 comp rx=0 tx=0).stop 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80102990 msgr2=0x7f7c80197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80102990 0x7f7c80197e70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800c920 tx=0x7f7c6800cde0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 shutdown_connections 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7c6c0a2b90 0x7f7c6c0a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c80103350 0x7f7c8019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c80102990 0x7f7c80197e70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 --2- 172.21.15.27:0/1681295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c801017a0 0x7f7c80197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 >> 172.21.15.27:0/1681295059 conn(0x7f7c800fcd10 msgr2=0x7f7c80105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 shutdown_connections 2024-04-25T01:23:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.168+0000 7f7c86f14640 1 -- 172.21.15.27:0/1681295059 wait complete. 2024-04-25T01:23:12.158 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:11 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4224504090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:23:12.158 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:11 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1970871215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:23:12.158 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:11 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3975823526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:12.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:23:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 -- 172.21.15.27:0/665260114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8102200 msgr2=0x7f05b81045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 --2- 172.21.15.27:0/665260114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8102200 0x7f05b81045f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009ca0 tx=0x7f05a802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 -- 172.21.15.27:0/665260114 shutdown_connections 2024-04-25T01:23:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 --2- 172.21.15.27:0/665260114 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05b8107460 0x7f05b81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 --2- 172.21.15.27:0/665260114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8104b30 0x7f05b8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 --2- 172.21.15.27:0/665260114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8102200 0x7f05b81045f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 -- 172.21.15.27:0/665260114 >> 172.21.15.27:0/665260114 conn(0x7f05b80fbbf0 msgr2=0x7f05b80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 -- 172.21.15.27:0/665260114 shutdown_connections 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.284+0000 7f05bfaca640 1 -- 172.21.15.27:0/665260114 wait complete. 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bfaca640 1 Processor -- start 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bfaca640 1 -- start start 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bfaca640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102200 0x7f05b8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bfaca640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8104b30 0x7f05b81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bfaca640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05b8107460 0x7f05b819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bfaca640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f05b810d430 con 0x7f05b8104b30 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bfaca640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f05b810d2b0 con 0x7f05b8107460 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bfaca640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05b810d5b0 con 0x7f05b8102200 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bd065640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8104b30 0x7f05b81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bd065640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8104b30 0x7f05b81980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60394/0 (socket says 172.21.15.27:60394) 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bd065640 1 -- 172.21.15.27:0/66035738 learned_addr learned my addr 172.21.15.27:0/66035738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.285+0000 7f05bd866640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102200 0x7f05b8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05bd065640 1 -- 172.21.15.27:0/66035738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102200 msgr2=0x7f05b8197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05bd065640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102200 0x7f05b8197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05be067640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05b8107460 0x7f05b819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05bd065640 1 -- 172.21.15.27:0/66035738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05b8107460 msgr2=0x7f05b819f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05bd065640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05b8107460 0x7f05b819f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05bd065640 1 -- 172.21.15.27:0/66035738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b819f790 con 0x7f05b8104b30 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05bd065640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8104b30 0x7f05b81980b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00d740 tx=0x7f05ac00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05b6ffd640 1 -- 172.21.15.27:0/66035738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac017070 con 0x7f05b8104b30 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05b6ffd640 1 -- 172.21.15.27:0/66035738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05ac00de50 con 0x7f05b8104b30 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b819fa80 con 0x7f05b8104b30 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05b6ffd640 1 -- 172.21.15.27:0/66035738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac004e00 con 0x7f05b8104b30 2024-04-25T01:23:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.286+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b819ff80 con 0x7f05b8104b30 2024-04-25T01:23:12.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.287+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05b80fd6e0 con 0x7f05b8104b30 2024-04-25T01:23:12.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.289+0000 7f05b6ffd640 1 -- 172.21.15.27:0/66035738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f05ac004fa0 con 0x7f05b8104b30 2024-04-25T01:23:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.289+0000 7f05b6ffd640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f05a00a2b90 0x7f05a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.290+0000 7f05bd866640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f05a00a2b90 0x7f05a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.290+0000 7f05b6ffd640 1 -- 172.21.15.27:0/66035738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f05ac0cf010 con 0x7f05b8104b30 2024-04-25T01:23:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.290+0000 7f05bd866640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f05a00a2b90 0x7f05a00a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f05a8006010 tx=0x7f05a8000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.293+0000 7f05b6ffd640 1 -- 172.21.15.27:0/66035738 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f05ac09bd30 con 0x7f05b8104b30 2024-04-25T01:23:12.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:11 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4224504090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:23:12.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:11 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1970871215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:23:12.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:11 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3975823526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:12.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:11 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4224504090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:23:12.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:11 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1970871215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:23:12.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:11 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3975823526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:12.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.476+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f05b802d3c0 con 0x7f05b8104b30 2024-04-25T01:23:12.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.477+0000 7f05b6ffd640 1 -- 172.21.15.27:0/66035738 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f05ac09b630 con 0x7f05b8104b30 2024-04-25T01:23:12.457 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:23:12.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f05a00a2b90 msgr2=0x7f05a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f05a00a2b90 0x7f05a00a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f05a8006010 tx=0x7f05a8000f40 comp rx=0 tx=0).stop 2024-04-25T01:23:12.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8104b30 msgr2=0x7f05b81980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8104b30 0x7f05b81980b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00d740 tx=0x7f05ac00dc00 comp rx=0 tx=0).stop 2024-04-25T01:23:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 shutdown_connections 2024-04-25T01:23:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f05a00a2b90 0x7f05a00a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05b8107460 0x7f05b819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05b8104b30 0x7f05b81980b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 --2- 172.21.15.27:0/66035738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102200 0x7f05b8197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 >> 172.21.15.27:0/66035738 conn(0x7f05b80fbbf0 msgr2=0x7f05b8102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 shutdown_connections 2024-04-25T01:23:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.479+0000 7f05bfaca640 1 -- 172.21.15.27:0/66035738 wait complete. 2024-04-25T01:23:12.470 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:23:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.594+0000 7fa35184c640 1 -- 172.21.15.27:0/3463741009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 msgr2=0x7fa34c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.594+0000 7fa35184c640 1 --2- 172.21.15.27:0/3463741009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 0x7fa34c108ac0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009ca0 tx=0x7fa33c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.594+0000 7fa35184c640 1 -- 172.21.15.27:0/3463741009 shutdown_connections 2024-04-25T01:23:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.594+0000 7fa35184c640 1 --2- 172.21.15.27:0/3463741009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 0x7fa34c108ac0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.594+0000 7fa35184c640 1 --2- 172.21.15.27:0/3463741009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa34c101870 0x7fa34c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.594+0000 7fa35184c640 1 --2- 172.21.15.27:0/3463741009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c100680 0x7fa34c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 -- 172.21.15.27:0/3463741009 >> 172.21.15.27:0/3463741009 conn(0x7fa34c0fbbf0 msgr2=0x7fa34c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 -- 172.21.15.27:0/3463741009 shutdown_connections 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 -- 172.21.15.27:0/3463741009 wait complete. 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 Processor -- start 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 -- start start 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa34c100680 0x7fa34c195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c101870 0x7fa34c195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 0x7fa34c19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c10b1c0 con 0x7fa34c102230 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c10b040 con 0x7fa34c101870 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa35184c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c10b340 con 0x7fa34c100680 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.595+0000 7fa34b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 0x7fa34c19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 0x7fa34c19cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60406/0 (socket says 172.21.15.27:60406) 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34b7fe640 1 -- 172.21.15.27:0/130463742 learned_addr learned my addr 172.21.15.27:0/130463742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:12.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34a7fc640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c101870 0x7fa34c195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34affd640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa34c100680 0x7fa34c195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34b7fe640 1 -- 172.21.15.27:0/130463742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa34c100680 msgr2=0x7fa34c195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34b7fe640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa34c100680 0x7fa34c195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34b7fe640 1 -- 172.21.15.27:0/130463742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c101870 msgr2=0x7fa34c195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34b7fe640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c101870 0x7fa34c195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34b7fe640 1 -- 172.21.15.27:0/130463742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34c19d450 con 0x7fa34c102230 2024-04-25T01:23:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34affd640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa34c100680 0x7fa34c195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:12.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34a7fc640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c101870 0x7fa34c195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:12.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa34b7fe640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 0x7fa34c19cde0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa33c005e30 tx=0x7fa33c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa35084a640 1 -- 172.21.15.27:0/130463742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c003c30 con 0x7fa34c102230 2024-04-25T01:23:12.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa35084a640 1 -- 172.21.15.27:0/130463742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa33c003dd0 con 0x7fa34c102230 2024-04-25T01:23:12.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa34c19d6e0 con 0x7fa34c102230 2024-04-25T01:23:12.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa35084a640 1 -- 172.21.15.27:0/130463742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c02fde0 con 0x7fa34c102230 2024-04-25T01:23:12.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.596+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa34c19dc80 con 0x7fa34c102230 2024-04-25T01:23:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.597+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa34c1079a0 con 0x7fa34c102230 2024-04-25T01:23:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.598+0000 7fa35084a640 1 -- 172.21.15.27:0/130463742 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa33c004050 con 0x7fa34c102230 2024-04-25T01:23:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.599+0000 7fa35084a640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa3340a2c10 0x7fa3340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.599+0000 7fa34affd640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa3340a2c10 0x7fa3340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.599+0000 7fa35084a640 1 -- 172.21.15.27:0/130463742 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa33c0f06e0 con 0x7fa34c102230 2024-04-25T01:23:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.599+0000 7fa34affd640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa3340a2c10 0x7fa3340a50d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa344005d00 tx=0x7fa344005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.604+0000 7fa35084a640 1 -- 172.21.15.27:0/130463742 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa33c0bd400 con 0x7fa34c102230 2024-04-25T01:23:12.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.789+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa34c062340 con 0x7fa34c102230 2024-04-25T01:23:12.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.789+0000 7fa35084a640 1 -- 172.21.15.27:0/130463742 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fa33c0bcd00 con 0x7fa34c102230 2024-04-25T01:23:12.770 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:23:12.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa3340a2c10 msgr2=0x7fa3340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa3340a2c10 0x7fa3340a50d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa344005d00 tx=0x7fa344005c90 comp rx=0 tx=0).stop 2024-04-25T01:23:12.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 msgr2=0x7fa34c19cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 0x7fa34c19cde0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa33c005e30 tx=0x7fa33c031ee0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 shutdown_connections 2024-04-25T01:23:12.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa3340a2c10 0x7fa3340a50d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa34c102230 0x7fa34c19cde0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c101870 0x7fa34c195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 --2- 172.21.15.27:0/130463742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa34c100680 0x7fa34c195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.791+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 >> 172.21.15.27:0/130463742 conn(0x7fa34c0fbbf0 msgr2=0x7fa34c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.792+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 shutdown_connections 2024-04-25T01:23:12.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.792+0000 7fa35184c640 1 -- 172.21.15.27:0/130463742 wait complete. 2024-04-25T01:23:12.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:23:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.908+0000 7fe798607640 1 -- 172.21.15.27:0/1333702589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe790101920 msgr2=0x7fe790108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.908+0000 7fe798607640 1 --2- 172.21.15.27:0/1333702589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe790101920 0x7fe790108240 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe788009ca0 tx=0x7fe78802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.908+0000 7fe798607640 1 -- 172.21.15.27:0/1333702589 shutdown_connections 2024-04-25T01:23:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.908+0000 7fe798607640 1 --2- 172.21.15.27:0/1333702589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe790101920 0x7fe790108240 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.908+0000 7fe798607640 1 --2- 172.21.15.27:0/1333702589 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe790100f60 0x7fe7901013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.908+0000 7fe798607640 1 --2- 172.21.15.27:0/1333702589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7900ffe10 0x7fe790100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.908+0000 7fe798607640 1 -- 172.21.15.27:0/1333702589 >> 172.21.15.27:0/1333702589 conn(0x7fe7900fb380 msgr2=0x7fe7900fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.908+0000 7fe798607640 1 -- 172.21.15.27:0/1333702589 shutdown_connections 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 -- 172.21.15.27:0/1333702589 wait complete. 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 Processor -- start 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 -- start start 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7900ffe10 0x7fe790197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe790100f60 0x7fe790198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe790101920 0x7fe79019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe79010d470 con 0x7fe7900ffe10 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe79010d2f0 con 0x7fe790101920 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe798607640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe79010d5f0 con 0x7fe790100f60 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe7963a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7900ffe10 0x7fe790197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe7963a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7900ffe10 0x7fe790197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60424/0 (socket says 172.21.15.27:60424) 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.909+0000 7fe7963a3640 1 -- 172.21.15.27:0/2591752956 learned_addr learned my addr 172.21.15.27:0/2591752956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe796ba4640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe790101920 0x7fe79019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe795ba2640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe790100f60 0x7fe790198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7963a3640 1 -- 172.21.15.27:0/2591752956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe790100f60 msgr2=0x7fe790198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7963a3640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe790100f60 0x7fe790198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7963a3640 1 -- 172.21.15.27:0/2591752956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe790101920 msgr2=0x7fe79019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7963a3640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe790101920 0x7fe79019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7963a3640 1 -- 172.21.15.27:0/2591752956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79019f6e0 con 0x7fe7900ffe10 2024-04-25T01:23:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe795ba2640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe790100f60 0x7fe790198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7963a3640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7900ffe10 0x7fe790197ac0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c890 tx=0x7fe78c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7877fe640 1 -- 172.21.15.27:0/2591752956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c004220 con 0x7fe7900ffe10 2024-04-25T01:23:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7877fe640 1 -- 172.21.15.27:0/2591752956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe78c0043c0 con 0x7fe7900ffe10 2024-04-25T01:23:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79019f9d0 con 0x7fe7900ffe10 2024-04-25T01:23:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe7877fe640 1 -- 172.21.15.27:0/2591752956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0049e0 con 0x7fe7900ffe10 2024-04-25T01:23:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.910+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79019ff20 con 0x7fe7900ffe10 2024-04-25T01:23:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.911+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7901070c0 con 0x7fe7900ffe10 2024-04-25T01:23:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.912+0000 7fe7877fe640 1 -- 172.21.15.27:0/2591752956 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe78c0079b0 con 0x7fe7900ffe10 2024-04-25T01:23:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.913+0000 7fe7877fe640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe77c0a2b90 0x7fe77c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.913+0000 7fe795ba2640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe77c0a2b90 0x7fe77c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.913+0000 7fe7877fe640 1 -- 172.21.15.27:0/2591752956 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe78c0cf2e0 con 0x7fe7900ffe10 2024-04-25T01:23:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.913+0000 7fe795ba2640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe77c0a2b90 0x7fe77c0a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe790198fd0 tx=0x7fe78000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:12.917+0000 7fe7877fe640 1 -- 172.21.15.27:0/2591752956 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe78c014030 con 0x7fe7900ffe10 2024-04-25T01:23:13.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.100+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe790062340 con 0x7fe7900ffe10 2024-04-25T01:23:13.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.100+0000 7fe7877fe640 1 -- 172.21.15.27:0/2591752956 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fe78c09c000 con 0x7fe7900ffe10 2024-04-25T01:23:13.080 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe77c0a2b90 msgr2=0x7fe77c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe77c0a2b90 0x7fe77c0a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe790198fd0 tx=0x7fe78000a560 comp rx=0 tx=0).stop 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7900ffe10 msgr2=0x7fe790197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7900ffe10 0x7fe790197ac0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c890 tx=0x7fe78c00cd50 comp rx=0 tx=0).stop 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 shutdown_connections 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe77c0a2b90 0x7fe77c0a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe790101920 0x7fe79019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe790100f60 0x7fe790198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 --2- 172.21.15.27:0/2591752956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7900ffe10 0x7fe790197ac0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 >> 172.21.15.27:0/2591752956 conn(0x7fe7900fb380 msgr2=0x7fe790104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 shutdown_connections 2024-04-25T01:23:13.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.102+0000 7fe798607640 1 -- 172.21.15.27:0/2591752956 wait complete. 2024-04-25T01:23:13.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:12 smithi027 ceph-mon[29197]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 107 B/s wr, 0 op/s 2024-04-25T01:23:13.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:12 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1681295059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:12 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/66035738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:12 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/130463742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:23:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/2646003599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 msgr2=0x7f2e48106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/2646003599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 0x7f2e48106fe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009900 tx=0x7f2e3802a410 comp rx=0 tx=0).stop 2024-04-25T01:23:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/2646003599 shutdown_connections 2024-04-25T01:23:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/2646003599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e48107520 0x7f2e48109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/2646003599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 0x7f2e48106fe0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/2646003599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e481022c0 0x7f2e481046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/2646003599 >> 172.21.15.27:0/2646003599 conn(0x7f2e480fbc30 msgr2=0x7f2e480fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/2646003599 shutdown_connections 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/2646003599 wait complete. 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.217+0000 7f2e4e4e1640 1 Processor -- start 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4e4e1640 1 -- start start 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4e4e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e481022c0 0x7f2e48101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4e4e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 0x7f2e48100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4e4e1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e48107520 0x7f2e481007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4e4e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4810d470 con 0x7f2e48104bf0 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4e4e1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4810d2f0 con 0x7f2e48107520 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4e4e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4810d5f0 con 0x7f2e481022c0 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 0x7f2e48100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 0x7f2e48100270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60444/0 (socket says 172.21.15.27:60444) 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e477fe640 1 -- 172.21.15.27:0/849457094 learned_addr learned my addr 172.21.15.27:0/849457094 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e47fff640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e481022c0 0x7f2e48101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4ca7e640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e48107520 0x7f2e481007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e477fe640 1 -- 172.21.15.27:0/849457094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e481022c0 msgr2=0x7f2e48101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e477fe640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e481022c0 0x7f2e48101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e477fe640 1 -- 172.21.15.27:0/849457094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e48107520 msgr2=0x7f2e481007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e477fe640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e48107520 0x7f2e481007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e477fe640 1 -- 172.21.15.27:0/849457094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e48101040 con 0x7f2e48104bf0 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.218+0000 7f2e4ca7e640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e48107520 0x7f2e481007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.219+0000 7f2e477fe640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 0x7f2e48100270 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2e38031440 tx=0x7f2e38032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.219+0000 7f2e457fa640 1 -- 172.21.15.27:0/849457094 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3803f030 con 0x7f2e48104bf0 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.219+0000 7f2e457fa640 1 -- 172.21.15.27:0/849457094 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e38003ce0 con 0x7f2e48104bf0 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.219+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e481a1ba0 con 0x7f2e48104bf0 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.219+0000 7f2e457fa640 1 -- 172.21.15.27:0/849457094 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e38035070 con 0x7f2e48104bf0 2024-04-25T01:23:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.219+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e481a20f0 con 0x7f2e48104bf0 2024-04-25T01:23:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.220+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e480fd720 con 0x7f2e48104bf0 2024-04-25T01:23:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.221+0000 7f2e457fa640 1 -- 172.21.15.27:0/849457094 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2e38042030 con 0x7f2e48104bf0 2024-04-25T01:23:13.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.222+0000 7f2e457fa640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2e300a2b90 0x7f2e300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.222+0000 7f2e47fff640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2e300a2b90 0x7f2e300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.222+0000 7f2e457fa640 1 -- 172.21.15.27:0/849457094 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2e380ef860 con 0x7f2e48104bf0 2024-04-25T01:23:13.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.222+0000 7f2e47fff640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2e300a2b90 0x7f2e300a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0095f0 tx=0x7f2e3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.226+0000 7f2e457fa640 1 -- 172.21.15.27:0/849457094 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2e380bc580 con 0x7f2e48104bf0 2024-04-25T01:23:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:12 smithi153 ceph-mon[32621]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 107 B/s wr, 0 op/s 2024-04-25T01:23:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:12 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1681295059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:12 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/66035738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:12 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/130463742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:12 smithi139 ceph-mon[32760]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 107 B/s wr, 0 op/s 2024-04-25T01:23:13.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:12 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1681295059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:12 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/66035738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:12 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/130463742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:23:13.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.426+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2e4802d170 con 0x7f2e48104bf0 2024-04-25T01:23:13.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.426+0000 7f2e457fa640 1 -- 172.21.15.27:0/849457094 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+586 (secure 0 0 0) 0x7f2e380bbe80 con 0x7f2e48104bf0 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/12403185 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/354957039 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/3353153293 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/234792808 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3645132749 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/219526886 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/219526886 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2409557466 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4191124222 2024-04-26T01:15:18.102339+0000 2024-04-25T01:23:13.407 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/3353153293 2024-04-26T01:15:45.288446+0000 2024-04-25T01:23:13.408 INFO:teuthology.orchestra.run.smithi027.stderr:listed 10 entries 2024-04-25T01:23:13.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2e300a2b90 msgr2=0x7f2e300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2e300a2b90 0x7f2e300a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0095f0 tx=0x7f2e3c005f90 comp rx=0 tx=0).stop 2024-04-25T01:23:13.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 msgr2=0x7f2e48100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 0x7f2e48100270 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2e38031440 tx=0x7f2e38032ce0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 shutdown_connections 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2e300a2b90 0x7f2e300a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e48107520 0x7f2e481007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e48104bf0 0x7f2e48100270 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 --2- 172.21.15.27:0/849457094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e481022c0 0x7f2e48101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 >> 172.21.15.27:0/849457094 conn(0x7f2e480fbc30 msgr2=0x7f2e48102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 shutdown_connections 2024-04-25T01:23:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:13.428+0000 7f2e4e4e1640 1 -- 172.21.15.27:0/849457094 wait complete. 2024-04-25T01:23:13.419 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-04-25T01:23:13.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:13.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:23:13.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi027.stderr:ceph-fuse[73103]: fuse finished with error 0 and tester_r 0 2024-04-25T01:23:13.444 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:23:14.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:13 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2591752956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:23:14.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:13 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/849457094' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:23:14.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:13 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2591752956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:23:14.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:13 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/849457094' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:23:14.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:13 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2591752956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:23:14.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:13 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/849457094' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:23:15.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:14 smithi027 ceph-mon[29197]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:23:15.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:14 smithi153 ceph-mon[32621]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:23:15.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:14 smithi139 ceph-mon[32760]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:23:17.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:16 smithi027 ceph-mon[29197]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:23:17.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:16 smithi153 ceph-mon[32621]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:23:17.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:16 smithi139 ceph-mon[32760]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:23:19.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:18 smithi027 ceph-mon[29197]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:23:19.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:18 smithi153 ceph-mon[32621]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:23:19.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:18 smithi139 ceph-mon[32760]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:23:19.546 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:23:19.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:23:19.546 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T01:23:19.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 -- 172.21.15.27:0/1335056709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64101890 msgr2=0x7f2c64101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 --2- 172.21.15.27:0/1335056709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64101890 0x7f2c64101d10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009900 tx=0x7f2c4c02a410 comp rx=0 tx=0).stop 2024-04-25T01:23:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 -- 172.21.15.27:0/1335056709 shutdown_connections 2024-04-25T01:23:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 --2- 172.21.15.27:0/1335056709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c64102250 0x7f2c64108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 --2- 172.21.15.27:0/1335056709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64101890 0x7f2c64101d10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 --2- 172.21.15.27:0/1335056709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c641006a0 0x7f2c64100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 -- 172.21.15.27:0/1335056709 >> 172.21.15.27:0/1335056709 conn(0x7f2c640fbc30 msgr2=0x7f2c640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:19.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 -- 172.21.15.27:0/1335056709 shutdown_connections 2024-04-25T01:23:19.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.750+0000 7f2c68b90640 1 -- 172.21.15.27:0/1335056709 wait complete. 2024-04-25T01:23:19.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c68b90640 1 Processor -- start 2024-04-25T01:23:19.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c68b90640 1 -- start start 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c68b90640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c641006a0 0x7f2c64104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c68b90640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c64101890 0x7f2c64102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c68b90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64102250 0x7f2c641031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c68b90640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410c320 con 0x7f2c64102250 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c68b90640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410c1a0 con 0x7f2c641006a0 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c68b90640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410c4a0 con 0x7f2c64101890 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c62d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64102250 0x7f2c641031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c62d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64102250 0x7f2c641031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38450/0 (socket says 172.21.15.27:38450) 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.751+0000 7f2c62d9d640 1 -- 172.21.15.27:0/946883906 learned_addr learned my addr 172.21.15.27:0/946883906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c6259c640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c641006a0 0x7f2c64104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c5bfff640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c64101890 0x7f2c64102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c62d9d640 1 -- 172.21.15.27:0/946883906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c64101890 msgr2=0x7f2c64102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c62d9d640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c64101890 0x7f2c64102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c62d9d640 1 -- 172.21.15.27:0/946883906 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c641006a0 msgr2=0x7f2c64104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c62d9d640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c641006a0 0x7f2c64104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c62d9d640 1 -- 172.21.15.27:0/946883906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c64103a40 con 0x7f2c64102250 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c5bfff640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c64101890 0x7f2c64102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c62d9d640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64102250 0x7f2c641031b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00eaa0 tx=0x7f2c5c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c5b7fe640 1 -- 172.21.15.27:0/946883906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c00cdb0 con 0x7f2c64102250 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c641a1bd0 con 0x7f2c64102250 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c5b7fe640 1 -- 172.21.15.27:0/946883906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c5c005020 con 0x7f2c64102250 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c5b7fe640 1 -- 172.21.15.27:0/946883906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0116b0 con 0x7f2c64102250 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.752+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c641a2120 con 0x7f2c64102250 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.753+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c641a2350 con 0x7f2c64102250 2024-04-25T01:23:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.754+0000 7f2c5b7fe640 1 -- 172.21.15.27:0/946883906 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2c5c005560 con 0x7f2c64102250 2024-04-25T01:23:19.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.755+0000 7f2c5b7fe640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2c500a2b90 0x7f2c500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:19.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.755+0000 7f2c5b7fe640 1 -- 172.21.15.27:0/946883906 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2c5c0cf530 con 0x7f2c64102250 2024-04-25T01:23:19.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.755+0000 7f2c6259c640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2c500a2b90 0x7f2c500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:19.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.755+0000 7f2c6259c640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2c500a2b90 0x7f2c500a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2c54005d00 tx=0x7f2c54005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:19.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.760+0000 7f2c5b7fe640 1 -- 172.21.15.27:0/946883906 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2c5c016030 con 0x7f2c64102250 2024-04-25T01:23:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.945+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c641a26e0 con 0x7f2c64102250 2024-04-25T01:23:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.947+0000 7f2c5b7fe640 1 -- 172.21.15.27:0/946883906 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7f2c5c09c250 con 0x7f2c64102250 2024-04-25T01:23:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:19.928 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2c500a2b90 msgr2=0x7f2c500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2c500a2b90 0x7f2c500a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2c54005d00 tx=0x7f2c54005c50 comp rx=0 tx=0).stop 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64102250 msgr2=0x7f2c641031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64102250 0x7f2c641031b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00eaa0 tx=0x7f2c5c00ef60 comp rx=0 tx=0).stop 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 shutdown_connections 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2c500a2b90 0x7f2c500a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64102250 0x7f2c641031b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c64101890 0x7f2c64102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 --2- 172.21.15.27:0/946883906 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2c641006a0 0x7f2c64104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 >> 172.21.15.27:0/946883906 conn(0x7f2c640fbc30 msgr2=0x7f2c640fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 shutdown_connections 2024-04-25T01:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:19.949+0000 7f2c68b90640 1 -- 172.21.15.27:0/946883906 wait complete. 2024-04-25T01:23:19.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 -- 172.21.15.27:0/2167996043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 msgr2=0x7fe164108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 --2- 172.21.15.27:0/2167996043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 0x7fe164108ae0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009ca0 tx=0x7fe15c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 -- 172.21.15.27:0/2167996043 shutdown_connections 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 --2- 172.21.15.27:0/2167996043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 0x7fe164108ae0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 --2- 172.21.15.27:0/2167996043 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe164101890 0x7fe164101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 --2- 172.21.15.27:0/2167996043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1641006a0 0x7fe164100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 -- 172.21.15.27:0/2167996043 >> 172.21.15.27:0/2167996043 conn(0x7fe1640fbc30 msgr2=0x7fe1640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 -- 172.21.15.27:0/2167996043 shutdown_connections 2024-04-25T01:23:20.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.065+0000 7fe1690bd640 1 -- 172.21.15.27:0/2167996043 wait complete. 2024-04-25T01:23:20.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe1690bd640 1 Processor -- start 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe1690bd640 1 -- start start 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe1690bd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1641006a0 0x7fe164104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe1690bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe164101890 0x7fe164102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe1690bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 0x7fe1641031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe1690bd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe16410c320 con 0x7fe164102250 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe1690bd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe16410c1a0 con 0x7fe1641006a0 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe1690bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe16410c4a0 con 0x7fe164101890 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe16359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 0x7fe1641031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe16359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 0x7fe1641031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38464/0 (socket says 172.21.15.27:38464) 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe16359e640 1 -- 172.21.15.27:0/594667371 learned_addr learned my addr 172.21.15.27:0/594667371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:20.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe162d9d640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1641006a0 0x7fe164104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.066+0000 7fe15bfff640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe164101890 0x7fe164102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe16359e640 1 -- 172.21.15.27:0/594667371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe164101890 msgr2=0x7fe164102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe16359e640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe164101890 0x7fe164102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe16359e640 1 -- 172.21.15.27:0/594667371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1641006a0 msgr2=0x7fe164104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe16359e640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1641006a0 0x7fe164104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe16359e640 1 -- 172.21.15.27:0/594667371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe164103a40 con 0x7fe164102250 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe162d9d640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1641006a0 0x7fe164104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe16359e640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 0x7fe1641031b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe15c031ae0 tx=0x7fe15c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe160d99640 1 -- 172.21.15.27:0/594667371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c003b70 con 0x7fe164102250 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe160d99640 1 -- 172.21.15.27:0/594667371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe15c003d10 con 0x7fe164102250 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1641a1b70 con 0x7fe164102250 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe160d99640 1 -- 172.21.15.27:0/594667371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c02fc40 con 0x7fe164102250 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.067+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1641a20c0 con 0x7fe164102250 2024-04-25T01:23:20.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.068+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1641a2290 con 0x7fe164102250 2024-04-25T01:23:20.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.069+0000 7fe160d99640 1 -- 172.21.15.27:0/594667371 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe15c004050 con 0x7fe164102250 2024-04-25T01:23:20.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.070+0000 7fe160d99640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe1500a2c10 0x7fe1500a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.070+0000 7fe160d99640 1 -- 172.21.15.27:0/594667371 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe15c0f04c0 con 0x7fe164102250 2024-04-25T01:23:20.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.070+0000 7fe162d9d640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe1500a2c10 0x7fe1500a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.070+0000 7fe162d9d640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe1500a2c10 0x7fe1500a50d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe1641016f0 tx=0x7fe154005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:20.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.074+0000 7fe160d99640 1 -- 172.21.15.27:0/594667371 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe15c0bd180 con 0x7fe164102250 2024-04-25T01:23:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.259+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1641a2580 con 0x7fe164102250 2024-04-25T01:23:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.261+0000 7fe160d99640 1 -- 172.21.15.27:0/594667371 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7fe15c0bca80 con 0x7fe164102250 2024-04-25T01:23:20.242 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:20.242 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.262+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe1500a2c10 msgr2=0x7fe1500a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.262+0000 7fe1690bd640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe1500a2c10 0x7fe1500a50d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe1641016f0 tx=0x7fe154005f90 comp rx=0 tx=0).stop 2024-04-25T01:23:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 msgr2=0x7fe1641031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 0x7fe1641031b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe15c031ae0 tx=0x7fe15c031ee0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 shutdown_connections 2024-04-25T01:23:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe1500a2c10 0x7fe1500a50d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe164102250 0x7fe1641031b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe164101890 0x7fe164102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 --2- 172.21.15.27:0/594667371 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe1641006a0 0x7fe164104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 >> 172.21.15.27:0/594667371 conn(0x7fe1640fbc30 msgr2=0x7fe1640fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 shutdown_connections 2024-04-25T01:23:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.263+0000 7fe1690bd640 1 -- 172.21.15.27:0/594667371 wait complete. 2024-04-25T01:23:20.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 -- 172.21.15.27:0/1182800450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 msgr2=0x7f428c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 --2- 172.21.15.27:0/1182800450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 0x7f428c1098b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4284009ca0 tx=0x7f428402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 -- 172.21.15.27:0/1182800450 shutdown_connections 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 --2- 172.21.15.27:0/1182800450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 0x7f428c1098b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 --2- 172.21.15.27:0/1182800450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f428c104b30 0x7f428c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 --2- 172.21.15.27:0/1182800450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f428c102200 0x7f428c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 -- 172.21.15.27:0/1182800450 >> 172.21.15.27:0/1182800450 conn(0x7f428c0fbbf0 msgr2=0x7f428c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 -- 172.21.15.27:0/1182800450 shutdown_connections 2024-04-25T01:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.379+0000 7f42918ba640 1 -- 172.21.15.27:0/1182800450 wait complete. 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f42918ba640 1 Processor -- start 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f42918ba640 1 -- start start 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f42918ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f428c102200 0x7f428c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f42918ba640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f428c104b30 0x7f428c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f42918ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 0x7f428c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f42918ba640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f428c10d400 con 0x7f428c107460 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f42918ba640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f428c10d280 con 0x7f428c104b30 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f42918ba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f428c10d580 con 0x7f428c102200 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f428b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 0x7f428c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f428b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 0x7f428c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38486/0 (socket says 172.21.15.27:38486) 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f428b7fe640 1 -- 172.21.15.27:0/1151493154 learned_addr learned my addr 172.21.15.27:0/1151493154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f428affd640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f428c102200 0x7f428c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.380+0000 7f4283fff640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f428c104b30 0x7f428c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f428b7fe640 1 -- 172.21.15.27:0/1151493154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f428c102200 msgr2=0x7f428c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f428b7fe640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f428c102200 0x7f428c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f428b7fe640 1 -- 172.21.15.27:0/1151493154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f428c104b30 msgr2=0x7f428c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f428b7fe640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f428c104b30 0x7f428c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f428b7fe640 1 -- 172.21.15.27:0/1151493154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c19f670 con 0x7f428c107460 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f428b7fe640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 0x7f428c19f000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4284005e30 tx=0x7f4284031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f42908b8640 1 -- 172.21.15.27:0/1151493154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4284003c30 con 0x7f428c107460 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f42908b8640 1 -- 172.21.15.27:0/1151493154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4284003dd0 con 0x7f428c107460 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428c19f900 con 0x7f428c107460 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f42908b8640 1 -- 172.21.15.27:0/1151493154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428402fde0 con 0x7f428c107460 2024-04-25T01:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.381+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428c19fe50 con 0x7f428c107460 2024-04-25T01:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.382+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f428c0fd6e0 con 0x7f428c107460 2024-04-25T01:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.383+0000 7f42908b8640 1 -- 172.21.15.27:0/1151493154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4284004050 con 0x7f428c107460 2024-04-25T01:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.384+0000 7f42908b8640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42780a2b90 0x7f42780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.384+0000 7f428affd640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42780a2b90 0x7f42780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.384+0000 7f42908b8640 1 -- 172.21.15.27:0/1151493154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f42840f0810 con 0x7f428c107460 2024-04-25T01:23:20.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.384+0000 7f428affd640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42780a2b90 0x7f42780a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f427c0058c0 tx=0x7f427c00a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.389+0000 7f42908b8640 1 -- 172.21.15.27:0/1151493154 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f42840bd530 con 0x7f428c107460 2024-04-25T01:23:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.612+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f428c062340 con 0x7f428c107460 2024-04-25T01:23:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.910+0000 7f42908b8640 1 -- 172.21.15.27:0/1151493154 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f42840bce30 con 0x7f428c107460 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42780a2b90 msgr2=0x7f42780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42780a2b90 0x7f42780a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f427c0058c0 tx=0x7f427c00a4c0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 msgr2=0x7f428c19f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 0x7f428c19f000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4284005e30 tx=0x7f4284031ee0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 shutdown_connections 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f42780a2b90 0x7f42780a5050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c107460 0x7f428c19f000 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f428c104b30 0x7f428c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 --2- 172.21.15.27:0/1151493154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f428c102200 0x7f428c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.912+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 >> 172.21.15.27:0/1151493154 conn(0x7f428c0fbbf0 msgr2=0x7f428c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.913+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 shutdown_connections 2024-04-25T01:23:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:20.913+0000 7f42918ba640 1 -- 172.21.15.27:0/1151493154 wait complete. 2024-04-25T01:23:20.906 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T01:23:20.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-25T01:23:21.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.029+0000 7f7e87012640 1 -- 172.21.15.27:0/3828875514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 msgr2=0x7f7e80101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.029+0000 7f7e87012640 1 --2- 172.21.15.27:0/3828875514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 0x7f7e80101d10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009900 tx=0x7f7e7402a410 comp rx=0 tx=0).stop 2024-04-25T01:23:21.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 -- 172.21.15.27:0/3828875514 shutdown_connections 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 --2- 172.21.15.27:0/3828875514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e80102250 0x7f7e80108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 --2- 172.21.15.27:0/3828875514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 0x7f7e80101d10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 --2- 172.21.15.27:0/3828875514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e801006a0 0x7f7e80100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 -- 172.21.15.27:0/3828875514 >> 172.21.15.27:0/3828875514 conn(0x7f7e800fbc30 msgr2=0x7f7e800fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 -- 172.21.15.27:0/3828875514 shutdown_connections 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 -- 172.21.15.27:0/3828875514 wait complete. 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 Processor -- start 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 -- start start 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e801006a0 0x7f7e8006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 0x7f7e8019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e80102250 0x7f7e8019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8010c320 con 0x7f7e80101890 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8010c1a0 con 0x7f7e80102250 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e87012640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8010c4a0 con 0x7f7e801006a0 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 0x7f7e8019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 0x7f7e8019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38504/0 (socket says 172.21.15.27:38504) 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.030+0000 7f7e7ffff640 1 -- 172.21.15.27:0/1560665611 learned_addr learned my addr 172.21.15.27:0/1560665611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e84dae640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e801006a0 0x7f7e8006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7ffff640 1 -- 172.21.15.27:0/1560665611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e801006a0 msgr2=0x7f7e8006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7ffff640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e801006a0 0x7f7e8006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7ffff640 1 -- 172.21.15.27:0/1560665611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e80102250 msgr2=0x7f7e8019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7ffff640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e80102250 0x7f7e8019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7ffff640 1 -- 172.21.15.27:0/1560665611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8019f690 con 0x7f7e80101890 2024-04-25T01:23:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e84dae640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e801006a0 0x7f7e8006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7ffff640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 0x7f7e8019c790 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009230 tx=0x7f7e74004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7dffb640 1 -- 172.21.15.27:0/1560665611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7403f030 con 0x7f7e80101890 2024-04-25T01:23:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7dffb640 1 -- 172.21.15.27:0/1560665611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e74003ce0 con 0x7f7e80101890 2024-04-25T01:23:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8019f920 con 0x7f7e80101890 2024-04-25T01:23:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e7dffb640 1 -- 172.21.15.27:0/1560665611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74035070 con 0x7f7e80101890 2024-04-25T01:23:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.031+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8019fe10 con 0x7f7e80101890 2024-04-25T01:23:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.032+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e801079c0 con 0x7f7e80101890 2024-04-25T01:23:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.033+0000 7f7e7dffb640 1 -- 172.21.15.27:0/1560665611 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7e74042030 con 0x7f7e80101890 2024-04-25T01:23:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.034+0000 7f7e7dffb640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7e680a2b90 0x7f7e680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.034+0000 7f7e84dae640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7e680a2b90 0x7f7e680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.034+0000 7f7e7dffb640 1 -- 172.21.15.27:0/1560665611 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7e740efd70 con 0x7f7e80101890 2024-04-25T01:23:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.034+0000 7f7e84dae640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7e680a2b90 0x7f7e680a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e801016f0 tx=0x7f7e78009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.039+0000 7f7e7dffb640 1 -- 172.21.15.27:0/1560665611 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7e740bca90 con 0x7f7e80101890 2024-04-25T01:23:21.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:20 smithi027 ceph-mon[29197]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:23:21.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:20 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/946883906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:21.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:20 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/594667371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:21.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:20 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1151493154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:23:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.263+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f7e80062340 con 0x7f7e80101890 2024-04-25T01:23:21.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:20 smithi153 ceph-mon[32621]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:23:21.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:20 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/946883906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:21.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:20 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/594667371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:21.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:20 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1151493154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:23:21.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:20 smithi139 ceph-mon[32760]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:23:21.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:20 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/946883906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:21.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:20 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/594667371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:21.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:20 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1151493154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:23:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.920+0000 7f7e7dffb640 1 -- 172.21.15.27:0/1560665611 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f7e740bc390 con 0x7f7e80101890 2024-04-25T01:23:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7e680a2b90 msgr2=0x7f7e680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7e680a2b90 0x7f7e680a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e801016f0 tx=0x7f7e78009520 comp rx=0 tx=0).stop 2024-04-25T01:23:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 msgr2=0x7f7e8019c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 0x7f7e8019c790 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009230 tx=0x7f7e74004360 comp rx=0 tx=0).stop 2024-04-25T01:23:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 shutdown_connections 2024-04-25T01:23:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f7e680a2b90 0x7f7e680a5050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e80102250 0x7f7e8019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e80101890 0x7f7e8019c790 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 --2- 172.21.15.27:0/1560665611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e801006a0 0x7f7e8006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 >> 172.21.15.27:0/1560665611 conn(0x7f7e800fbc30 msgr2=0x7f7e80104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 shutdown_connections 2024-04-25T01:23:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:21.923+0000 7f7e87012640 1 -- 172.21.15.27:0/1560665611 wait complete. 2024-04-25T01:23:21.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T01:23:22.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 -- 172.21.15.27:0/1650579720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf101006a0 msgr2=0x7fcf10100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1650579720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf101006a0 0x7fcf10100ac0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009900 tx=0x7fcf0002a410 comp rx=0 tx=0).stop 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 -- 172.21.15.27:0/1650579720 shutdown_connections 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1650579720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf10102250 0x7fcf10108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1650579720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf10101890 0x7fcf10101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1650579720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf101006a0 0x7fcf10100ac0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 -- 172.21.15.27:0/1650579720 >> 172.21.15.27:0/1650579720 conn(0x7fcf100fbc30 msgr2=0x7fcf100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 -- 172.21.15.27:0/1650579720 shutdown_connections 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.046+0000 7fcf158e1640 1 -- 172.21.15.27:0/1650579720 wait complete. 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf158e1640 1 Processor -- start 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf158e1640 1 -- start start 2024-04-25T01:23:22.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf158e1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf101006a0 0x7fcf10104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf158e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf10101890 0x7fcf10102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf158e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf10102250 0x7fcf101031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf158e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1010c3f0 con 0x7fcf10101890 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf158e1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1010c270 con 0x7fcf101006a0 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf158e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1010c570 con 0x7fcf10102250 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf0e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf10101890 0x7fcf10102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf0e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf10101890 0x7fcf10102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38510/0 (socket says 172.21.15.27:38510) 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf0e7fc640 1 -- 172.21.15.27:0/1204122478 learned_addr learned my addr 172.21.15.27:0/1204122478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.047+0000 7fcf0effd640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf101006a0 0x7fcf10104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0f7fe640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf10102250 0x7fcf101031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0e7fc640 1 -- 172.21.15.27:0/1204122478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf10102250 msgr2=0x7fcf101031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0e7fc640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf10102250 0x7fcf101031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0e7fc640 1 -- 172.21.15.27:0/1204122478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf101006a0 msgr2=0x7fcf10104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0e7fc640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf101006a0 0x7fcf10104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0e7fc640 1 -- 172.21.15.27:0/1204122478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf10103a40 con 0x7fcf10101890 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0f7fe640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf10102250 0x7fcf101031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0effd640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf101006a0 0x7fcf10104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf0e7fc640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf10101890 0x7fcf10102c70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400b4a0 tx=0x7fcf0400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf148df640 1 -- 172.21.15.27:0/1204122478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04014070 con 0x7fcf10101890 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf148df640 1 -- 172.21.15.27:0/1204122478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf04010af0 con 0x7fcf10101890 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf101a1dc0 con 0x7fcf10101890 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf148df640 1 -- 172.21.15.27:0/1204122478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04017d10 con 0x7fcf10101890 2024-04-25T01:23:22.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.048+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf101a22b0 con 0x7fcf10101890 2024-04-25T01:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.049+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf100689b0 con 0x7fcf10101890 2024-04-25T01:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.050+0000 7fcf148df640 1 -- 172.21.15.27:0/1204122478 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcf04017070 con 0x7fcf10101890 2024-04-25T01:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.051+0000 7fcf148df640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcef80a2b90 0x7fcef80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.051+0000 7fcf148df640 1 -- 172.21.15.27:0/1204122478 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcf040ce730 con 0x7fcf10101890 2024-04-25T01:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.051+0000 7fcf0effd640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcef80a2b90 0x7fcef80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:22.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.051+0000 7fcf0effd640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcef80a2b90 0x7fcef80a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcf00000c00 tx=0x7fcf00031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:22.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.055+0000 7fcf148df640 1 -- 172.21.15.27:0/1204122478 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcf0409b450 con 0x7fcf10101890 2024-04-25T01:23:22.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:21 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1151493154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:23:22.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:21 smithi027 ceph-mon[29197]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:22.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:21 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1560665611' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:23:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.283+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fcf101a2d10 con 0x7fcf10101890 2024-04-25T01:23:22.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:21 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1151493154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:23:22.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:21 smithi153 ceph-mon[32621]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:22.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:21 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1560665611' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:23:22.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:21 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1151493154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:23:22.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:21 smithi139 ceph-mon[32760]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:22.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:21 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1560665611' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:23:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.926+0000 7fcf148df640 1 -- 172.21.15.27:0/1204122478 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fcf0409ad50 con 0x7fcf10101890 2024-04-25T01:23:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcef80a2b90 msgr2=0x7fcef80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcef80a2b90 0x7fcef80a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcf00000c00 tx=0x7fcf00031400 comp rx=0 tx=0).stop 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf10101890 msgr2=0x7fcf10102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf10101890 0x7fcf10102c70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400b4a0 tx=0x7fcf0400b960 comp rx=0 tx=0).stop 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 shutdown_connections 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcef80a2b90 0x7fcef80a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf10102250 0x7fcf101031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf10101890 0x7fcf10102c70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 --2- 172.21.15.27:0/1204122478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf101006a0 0x7fcf10104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 >> 172.21.15.27:0/1204122478 conn(0x7fcf100fbc30 msgr2=0x7fcf10069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 shutdown_connections 2024-04-25T01:23:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:22.929+0000 7fcf158e1640 1 -- 172.21.15.27:0/1204122478 wait complete. 2024-04-25T01:23:22.926 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T01:23:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 -- 172.21.15.27:0/645379003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 msgr2=0x7f0e04108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 --2- 172.21.15.27:0/645379003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 0x7f0e04108ae0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009ca0 tx=0x7f0dfc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 -- 172.21.15.27:0/645379003 shutdown_connections 2024-04-25T01:23:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 --2- 172.21.15.27:0/645379003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 0x7f0e04108ae0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 --2- 172.21.15.27:0/645379003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e04101890 0x7f0e04101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 --2- 172.21.15.27:0/645379003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e041006a0 0x7f0e04100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 -- 172.21.15.27:0/645379003 >> 172.21.15.27:0/645379003 conn(0x7f0e040fbc30 msgr2=0x7f0e040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:23.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 -- 172.21.15.27:0/645379003 shutdown_connections 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.051+0000 7f0e09eee640 1 -- 172.21.15.27:0/645379003 wait complete. 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e09eee640 1 Processor -- start 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e09eee640 1 -- start start 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e09eee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e041006a0 0x7f0e04196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e09eee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e04101890 0x7f0e04196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e09eee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 0x7f0e0419df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e09eee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0410c320 con 0x7f0e04102250 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e09eee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0410c1a0 con 0x7f0e04101890 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e09eee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0410c4a0 con 0x7f0e041006a0 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e03fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 0x7f0e0419df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e03fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 0x7f0e0419df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38514/0 (socket says 172.21.15.27:38514) 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e02ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e04101890 0x7f0e04196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e03fff640 1 -- 172.21.15.27:0/1501684678 learned_addr learned my addr 172.21.15.27:0/1501684678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.052+0000 7f0e037fe640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e041006a0 0x7f0e04196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e03fff640 1 -- 172.21.15.27:0/1501684678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e041006a0 msgr2=0x7f0e04196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e03fff640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e041006a0 0x7f0e04196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e03fff640 1 -- 172.21.15.27:0/1501684678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e04101890 msgr2=0x7f0e04196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e03fff640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e04101890 0x7f0e04196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e03fff640 1 -- 172.21.15.27:0/1501684678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0419e580 con 0x7f0e04102250 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e02ffd640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e04101890 0x7f0e04196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e03fff640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 0x7f0e0419df10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc031ae0 tx=0x7f0dfc031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e00ff9640 1 -- 172.21.15.27:0/1501684678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc003c30 con 0x7f0e04102250 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e00ff9640 1 -- 172.21.15.27:0/1501684678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dfc003dd0 con 0x7f0e04102250 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0419e810 con 0x7f0e04102250 2024-04-25T01:23:23.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e00ff9640 1 -- 172.21.15.27:0/1501684678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc02fcb0 con 0x7f0e04102250 2024-04-25T01:23:23.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.053+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0419edb0 con 0x7f0e04102250 2024-04-25T01:23:23.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.054+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e041079c0 con 0x7f0e04102250 2024-04-25T01:23:23.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.055+0000 7f0e00ff9640 1 -- 172.21.15.27:0/1501684678 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0dfc004050 con 0x7f0e04102250 2024-04-25T01:23:23.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.056+0000 7f0e00ff9640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0dec0a2c10 0x7f0dec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.056+0000 7f0e00ff9640 1 -- 172.21.15.27:0/1501684678 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0dfc0f0510 con 0x7f0e04102250 2024-04-25T01:23:23.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.056+0000 7f0e037fe640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0dec0a2c10 0x7f0dec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.056+0000 7f0e037fe640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0dec0a2c10 0x7f0dec0a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0e041016f0 tx=0x7f0df4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.060+0000 7f0e00ff9640 1 -- 172.21.15.27:0/1501684678 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0dfc0bd350 con 0x7f0e04102250 2024-04-25T01:23:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:22 smithi027 ceph-mon[29197]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:23:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:22 smithi027 ceph-mon[29197]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:23:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:22 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1560665611' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:23:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:22 smithi027 ceph-mon[29197]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:23:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:22 smithi027 ceph-mon[29197]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:22 smithi027 ceph-mon[29197]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:23:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:22 smithi027 ceph-mon[29197]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:23:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:22 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1204122478' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:23:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.284+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f0e04062340 con 0x7f0e04102250 2024-04-25T01:23:23.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:22 smithi153 ceph-mon[32621]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:23:23.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:22 smithi153 ceph-mon[32621]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:23:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:22 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1560665611' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:23:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:22 smithi153 ceph-mon[32621]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:23:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:22 smithi153 ceph-mon[32621]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:22 smithi153 ceph-mon[32621]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:23:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:22 smithi153 ceph-mon[32621]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:23:23.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:22 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1204122478' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:23:23.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:22 smithi139 ceph-mon[32760]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:23:23.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:22 smithi139 ceph-mon[32760]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:23:23.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:22 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1560665611' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:23:23.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:22 smithi139 ceph-mon[32760]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:23:23.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:22 smithi139 ceph-mon[32760]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:23:23.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:22 smithi139 ceph-mon[32760]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:23:23.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:22 smithi139 ceph-mon[32760]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:23:23.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:22 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1204122478' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:23:23.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.936+0000 7f0e00ff9640 1 -- 172.21.15.27:0/1501684678 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f0dfc0bcc50 con 0x7f0e04102250 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0dec0a2c10 msgr2=0x7f0dec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0dec0a2c10 0x7f0dec0a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0e041016f0 tx=0x7f0df4005f90 comp rx=0 tx=0).stop 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 msgr2=0x7f0e0419df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 0x7f0e0419df10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc031ae0 tx=0x7f0dfc031ee0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 shutdown_connections 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0dec0a2c10 0x7f0dec0a50d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e04102250 0x7f0e0419df10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e04101890 0x7f0e04196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 --2- 172.21.15.27:0/1501684678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e041006a0 0x7f0e04196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.938+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 >> 172.21.15.27:0/1501684678 conn(0x7f0e040fbc30 msgr2=0x7f0e04104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:23.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.939+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 shutdown_connections 2024-04-25T01:23:23.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:23.939+0000 7f0e09eee640 1 -- 172.21.15.27:0/1501684678 wait complete. 2024-04-25T01:23:23.939 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T01:23:23.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 -- 172.21.15.27:0/1949203414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac104b50 msgr2=0x7f0cac106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 --2- 172.21.15.27:0/1949203414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac104b50 0x7f0cac106f40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009900 tx=0x7f0ca002a410 comp rx=0 tx=0).stop 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 -- 172.21.15.27:0/1949203414 shutdown_connections 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 --2- 172.21.15.27:0/1949203414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac107480 0x7f0cac1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 --2- 172.21.15.27:0/1949203414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac104b50 0x7f0cac106f40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 --2- 172.21.15.27:0/1949203414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cac102220 0x7f0cac104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 -- 172.21.15.27:0/1949203414 >> 172.21.15.27:0/1949203414 conn(0x7f0cac0fbc30 msgr2=0x7f0cac0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 -- 172.21.15.27:0/1949203414 shutdown_connections 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.068+0000 7f0cb3161640 1 -- 172.21.15.27:0/1949203414 wait complete. 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb3161640 1 Processor -- start 2024-04-25T01:23:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb3161640 1 -- start start 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb3161640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac102220 0x7f0cac101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb3161640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac104b50 0x7f0cac1001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb3161640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cac107480 0x7f0cac100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb3161640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac10d410 con 0x7f0cac102220 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb3161640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac10d290 con 0x7f0cac107480 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb3161640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac10d590 con 0x7f0cac104b50 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb0efd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac102220 0x7f0cac101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb0efd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac102220 0x7f0cac101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38534/0 (socket says 172.21.15.27:38534) 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb0efd640 1 -- 172.21.15.27:0/948128820 learned_addr learned my addr 172.21.15.27:0/948128820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.069+0000 7f0cb16fe640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cac107480 0x7f0cac100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cabfff640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac104b50 0x7f0cac1001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cb0efd640 1 -- 172.21.15.27:0/948128820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac104b50 msgr2=0x7f0cac1001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cb0efd640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac104b50 0x7f0cac1001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cb0efd640 1 -- 172.21.15.27:0/948128820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cac107480 msgr2=0x7f0cac100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cb0efd640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cac107480 0x7f0cac100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cb0efd640 1 -- 172.21.15.27:0/948128820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac100fa0 con 0x7f0cac102220 2024-04-25T01:23:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cabfff640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac104b50 0x7f0cac1001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cb0efd640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac102220 0x7f0cac101b90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c00e9f0 tx=0x7f0c9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0ca9ffb640 1 -- 172.21.15.27:0/948128820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c00cd80 con 0x7f0cac102220 2024-04-25T01:23:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0ca9ffb640 1 -- 172.21.15.27:0/948128820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c9c010910 con 0x7f0cac102220 2024-04-25T01:23:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cac1a1bb0 con 0x7f0cac102220 2024-04-25T01:23:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0ca9ffb640 1 -- 172.21.15.27:0/948128820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c018990 con 0x7f0cac102220 2024-04-25T01:23:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.070+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cac1a2100 con 0x7f0cac102220 2024-04-25T01:23:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.071+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cac0689b0 con 0x7f0cac102220 2024-04-25T01:23:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.072+0000 7f0ca9ffb640 1 -- 172.21.15.27:0/948128820 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0c9c010ac0 con 0x7f0cac102220 2024-04-25T01:23:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.073+0000 7f0ca9ffb640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0c940a2b90 0x7f0c940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.073+0000 7f0cabfff640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0c940a2b90 0x7f0c940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.073+0000 7f0ca9ffb640 1 -- 172.21.15.27:0/948128820 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0c9c0cf630 con 0x7f0cac102220 2024-04-25T01:23:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.073+0000 7f0cabfff640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0c940a2b90 0x7f0c940a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0000c00 tx=0x7f0ca0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.077+0000 7f0ca9ffb640 1 -- 172.21.15.27:0/948128820 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0c9c09c350 con 0x7f0cac102220 2024-04-25T01:23:24.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1204122478' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:23:24.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] up:active 2024-04-25T01:23:24.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:23:24.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:23:24.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: Cluster is now healthy 2024-04-25T01:23:24.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:23:24.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T01:23:24.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T01:23:24.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:23 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1501684678' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:23:24.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.288+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0cac061b80 con 0x7f0c940a2b90 2024-04-25T01:23:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1204122478' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:23:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] up:active 2024-04-25T01:23:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:23:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:23:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: Cluster is now healthy 2024-04-25T01:23:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:23:24.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T01:23:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T01:23:24.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:23 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1501684678' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:23:24.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1204122478' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:23:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] up:active 2024-04-25T01:23:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:23:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:23:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: Cluster is now healthy 2024-04-25T01:23:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:23:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T01:23:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T01:23:24.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:23 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1501684678' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:23:24.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.313+0000 7f0ca9ffb640 1 -- 172.21.15.27:0/948128820 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0cac061b80 con 0x7f0c940a2b90 2024-04-25T01:23:24.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0c940a2b90 msgr2=0x7f0c940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0c940a2b90 0x7f0c940a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0000c00 tx=0x7f0ca0031400 comp rx=0 tx=0).stop 2024-04-25T01:23:24.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac102220 msgr2=0x7f0cac101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac102220 0x7f0cac101b90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c00e9f0 tx=0x7f0c9c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 shutdown_connections 2024-04-25T01:23:24.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0c940a2b90 0x7f0c940a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cac107480 0x7f0cac100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac104b50 0x7f0cac1001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 --2- 172.21.15.27:0/948128820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cac102220 0x7f0cac101b90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 >> 172.21.15.27:0/948128820 conn(0x7f0cac0fbc30 msgr2=0x7f0cac102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 shutdown_connections 2024-04-25T01:23:24.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.316+0000 7f0cb3161640 1 -- 172.21.15.27:0/948128820 wait complete. 2024-04-25T01:23:24.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1770930210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d441017a0 msgr2=0x7f0d44101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1770930210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d441017a0 0x7f0d44101bc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009900 tx=0x7f0d3402a410 comp rx=0 tx=0).stop 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1770930210 shutdown_connections 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1770930210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d44103350 0x7f0d44109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1770930210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d44102990 0x7f0d44102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1770930210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d441017a0 0x7f0d44101bc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1770930210 >> 172.21.15.27:0/1770930210 conn(0x7f0d440fcd10 msgr2=0x7f0d440ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1770930210 shutdown_connections 2024-04-25T01:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.434+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1770930210 wait complete. 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d4a0ee640 1 Processor -- start 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d4a0ee640 1 -- start start 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d4a0ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d441017a0 0x7f0d44197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d4a0ee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d44102990 0x7f0d44198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d4a0ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d44103350 0x7f0d4419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d4a0ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4410d440 con 0x7f0d44103350 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d4a0ee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4410d2c0 con 0x7f0d44102990 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d4a0ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4410d5c0 con 0x7f0d441017a0 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d44103350 0x7f0d4419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.435+0000 7f0d43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d44103350 0x7f0d4419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38582/0 (socket says 172.21.15.27:38582) 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d43fff640 1 -- 172.21.15.27:0/1374269571 learned_addr learned my addr 172.21.15.27:0/1374269571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d42ffd640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d44102990 0x7f0d44198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d437fe640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d441017a0 0x7f0d44197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d42ffd640 1 -- 172.21.15.27:0/1374269571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d441017a0 msgr2=0x7f0d44197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d42ffd640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d441017a0 0x7f0d44197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d42ffd640 1 -- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d44103350 msgr2=0x7f0d4419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d42ffd640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d44103350 0x7f0d4419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d42ffd640 1 -- 172.21.15.27:0/1374269571 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4419f700 con 0x7f0d44102990 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d43fff640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d44103350 0x7f0d4419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d42ffd640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d44102990 0x7f0d44198020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00e9f0 tx=0x7f0d2c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d40ff9640 1 -- 172.21.15.27:0/1374269571 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c00ccc0 con 0x7f0d44102990 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4419f9f0 con 0x7f0d44102990 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d40ff9640 1 -- 172.21.15.27:0/1374269571 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c010910 con 0x7f0d44102990 2024-04-25T01:23:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d40ff9640 1 -- 172.21.15.27:0/1374269571 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c018960 con 0x7f0d44102990 2024-04-25T01:23:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.436+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d4419ff90 con 0x7f0d44102990 2024-04-25T01:23:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.437+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d44108ac0 con 0x7f0d44102990 2024-04-25T01:23:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.438+0000 7f0d40ff9640 1 -- 172.21.15.27:0/1374269571 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0d2c010ac0 con 0x7f0d44102990 2024-04-25T01:23:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.439+0000 7f0d40ff9640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0d300a2b90 0x7f0d300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.439+0000 7f0d437fe640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0d300a2b90 0x7f0d300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.439+0000 7f0d40ff9640 1 -- 172.21.15.27:0/1374269571 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0d2c0cfa40 con 0x7f0d44102990 2024-04-25T01:23:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.439+0000 7f0d437fe640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0d300a2b90 0x7f0d300a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0d3402aa00 tx=0x7f0d34031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.442+0000 7f0d40ff9640 1 -- 172.21.15.27:0/1374269571 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0d2c09c760 con 0x7f0d44102990 2024-04-25T01:23:24.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.659+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0d44061c10 con 0x7f0d300a2b90 2024-04-25T01:23:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.685+0000 7f0d40ff9640 1 -- 172.21.15.27:0/1374269571 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0d44061c10 con 0x7f0d300a2b90 2024-04-25T01:23:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0d300a2b90 msgr2=0x7f0d300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0d300a2b90 0x7f0d300a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0d3402aa00 tx=0x7f0d34031400 comp rx=0 tx=0).stop 2024-04-25T01:23:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d44102990 msgr2=0x7f0d44198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d44102990 0x7f0d44198020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00e9f0 tx=0x7f0d2c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 shutdown_connections 2024-04-25T01:23:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0d300a2b90 0x7f0d300a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d44103350 0x7f0d4419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d44102990 0x7f0d44198020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 --2- 172.21.15.27:0/1374269571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d441017a0 0x7f0d44197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 >> 172.21.15.27:0/1374269571 conn(0x7f0d440fcd10 msgr2=0x7f0d44105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 shutdown_connections 2024-04-25T01:23:24.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.688+0000 7f0d4a0ee640 1 -- 172.21.15.27:0/1374269571 wait complete. 2024-04-25T01:23:24.679 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:23:24.293+0000 7f3cb4eda700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:23:24.679 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:23:24.293+0000 7f3cb4eda700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:23:24.679 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:23:24.293+0000 7f3cb4eda700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:23:24.679 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:23:24.293+0000 7f3cb4eda700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:23:24.679 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:23:24.293+0000 7f3cb4eda700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:23:24.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T01:23:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2962273987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44102230 msgr2=0x7f8f44108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2962273987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44102230 0x7f8f44108ac0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009ca0 tx=0x7f8f3402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2962273987 shutdown_connections 2024-04-25T01:23:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2962273987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44102230 0x7f8f44108ac0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2962273987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f44101870 0x7f8f44101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2962273987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44100680 0x7f8f44100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2962273987 >> 172.21.15.27:0/2962273987 conn(0x7f8f440fbbf0 msgr2=0x7f8f440fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2962273987 shutdown_connections 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2962273987 wait complete. 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 Processor -- start 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.806+0000 7f8f4b7a5640 1 -- start start 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f4b7a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44100680 0x7f8f44104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f4b7a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44101870 0x7f8f44102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f4b7a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f44102230 0x7f8f44103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f4b7a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4410c300 con 0x7f8f44101870 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f4b7a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4410c180 con 0x7f8f44102230 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f4b7a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4410c480 con 0x7f8f44100680 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44101870 0x7f8f44102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44101870 0x7f8f44102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38594/0 (socket says 172.21.15.27:38594) 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 -- 172.21.15.27:0/2683891092 learned_addr learned my addr 172.21.15.27:0/2683891092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f49d42640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f44102230 0x7f8f44103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f49541640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44100680 0x7f8f44104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 -- 172.21.15.27:0/2683891092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44100680 msgr2=0x7f8f44104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44100680 0x7f8f44104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 -- 172.21.15.27:0/2683891092 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f44102230 msgr2=0x7f8f44103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f44102230 0x7f8f44103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 -- 172.21.15.27:0/2683891092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f44103a20 con 0x7f8f44101870 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f48d40640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44101870 0x7f8f44102c50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800ea60 tx=0x7f8f3800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f427fc640 1 -- 172.21.15.27:0/2683891092 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3800cde0 con 0x7f8f44101870 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f427fc640 1 -- 172.21.15.27:0/2683891092 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f380035f0 con 0x7f8f44101870 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f441a1b90 con 0x7f8f44101870 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f427fc640 1 -- 172.21.15.27:0/2683891092 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f38010670 con 0x7f8f44101870 2024-04-25T01:23:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.807+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f441a20e0 con 0x7f8f44101870 2024-04-25T01:23:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.808+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f441a2310 con 0x7f8f44101870 2024-04-25T01:23:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.809+0000 7f8f427fc640 1 -- 172.21.15.27:0/2683891092 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8f380037a0 con 0x7f8f44101870 2024-04-25T01:23:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.810+0000 7f8f427fc640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8f2c0a2b90 0x7f8f2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.810+0000 7f8f427fc640 1 -- 172.21.15.27:0/2683891092 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8f380cf680 con 0x7f8f44101870 2024-04-25T01:23:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.810+0000 7f8f49541640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8f2c0a2b90 0x7f8f2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.811+0000 7f8f49541640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8f2c0a2b90 0x7f8f2c0a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c005fd0 tx=0x7f8f3c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:24.813+0000 7f8f427fc640 1 -- 172.21.15.27:0/2683891092 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8f3809c430 con 0x7f8f44101870 2024-04-25T01:23:24.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-mon[29197]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-04-25T01:23:24.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1501684678' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:23:24.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] up:active 2024-04-25T01:23:24.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-mon[29197]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T01:23:24.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:24 smithi027 ceph-mon[29197]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:23:25.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.027+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f8f440fddc0 con 0x7f8f2c0a2b90 2024-04-25T01:23:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.033+0000 7f8f427fc640 1 -- 172.21.15.27:0/2683891092 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8f440fddc0 con 0x7f8f2c0a2b90 2024-04-25T01:23:25.014 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/qa/sv_0/d6cb6d3d-bc48-4a66-8394-b896698a38c9 2024-04-25T01:23:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8f2c0a2b90 msgr2=0x7f8f2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8f2c0a2b90 0x7f8f2c0a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c005fd0 tx=0x7f8f3c00b390 comp rx=0 tx=0).stop 2024-04-25T01:23:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44101870 msgr2=0x7f8f44102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44101870 0x7f8f44102c50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800ea60 tx=0x7f8f3800ef20 comp rx=0 tx=0).stop 2024-04-25T01:23:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 shutdown_connections 2024-04-25T01:23:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8f2c0a2b90 0x7f8f2c0a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f44102230 0x7f8f44103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f44101870 0x7f8f44102c50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 --2- 172.21.15.27:0/2683891092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f44100680 0x7f8f44104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 >> 172.21.15.27:0/2683891092 conn(0x7f8f440fbbf0 msgr2=0x7f8f440fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 shutdown_connections 2024-04-25T01:23:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.035+0000 7f8f4b7a5640 1 -- 172.21.15.27:0/2683891092 wait complete. 2024-04-25T01:23:25.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T01:23:25.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 -- 172.21.15.27:0/538516213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c1017a0 msgr2=0x7faf9c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 --2- 172.21.15.27:0/538516213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c1017a0 0x7faf9c101bc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009900 tx=0x7faf8c02a410 comp rx=0 tx=0).stop 2024-04-25T01:23:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 -- 172.21.15.27:0/538516213 shutdown_connections 2024-04-25T01:23:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 --2- 172.21.15.27:0/538516213 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf9c103350 0x7faf9c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 --2- 172.21.15.27:0/538516213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c102990 0x7faf9c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 --2- 172.21.15.27:0/538516213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c1017a0 0x7faf9c101bc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 -- 172.21.15.27:0/538516213 >> 172.21.15.27:0/538516213 conn(0x7faf9c0fcd10 msgr2=0x7faf9c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 -- 172.21.15.27:0/538516213 shutdown_connections 2024-04-25T01:23:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.153+0000 7fafa0963640 1 -- 172.21.15.27:0/538516213 wait complete. 2024-04-25T01:23:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7fafa0963640 1 Processor -- start 2024-04-25T01:23:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7fafa0963640 1 -- start start 2024-04-25T01:23:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7fafa0963640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf9c1017a0 0x7faf9c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7fafa0963640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c102990 0x7faf9c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7fafa0963640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c103350 0x7faf9c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7fafa0963640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c10d3f0 con 0x7faf9c102990 2024-04-25T01:23:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7fafa0963640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c10d270 con 0x7faf9c1017a0 2024-04-25T01:23:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7fafa0963640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c10d570 con 0x7faf9c103350 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7faf99d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c102990 0x7faf9c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7faf99d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c102990 0x7faf9c073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38612/0 (socket says 172.21.15.27:38612) 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7faf99d9b640 1 -- 172.21.15.27:0/1783842171 learned_addr learned my addr 172.21.15.27:0/1783842171 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7faf9ad9d640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c103350 0x7faf9c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.154+0000 7faf9a59c640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf9c1017a0 0x7faf9c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf99d9b640 1 -- 172.21.15.27:0/1783842171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c103350 msgr2=0x7faf9c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf99d9b640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c103350 0x7faf9c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf99d9b640 1 -- 172.21.15.27:0/1783842171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf9c1017a0 msgr2=0x7faf9c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf99d9b640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf9c1017a0 0x7faf9c0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf99d9b640 1 -- 172.21.15.27:0/1783842171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c074900 con 0x7faf9c102990 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf9a59c640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf9c1017a0 0x7faf9c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf9ad9d640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c103350 0x7faf9c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf99d9b640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c102990 0x7faf9c073b30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faf9000d950 tx=0x7faf9000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf8b7fe640 1 -- 172.21.15.27:0/1783842171 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90017070 con 0x7faf9c102990 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9c1a1bf0 con 0x7faf9c102990 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf8b7fe640 1 -- 172.21.15.27:0/1783842171 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf90009c30 con 0x7faf9c102990 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7faf8b7fe640 1 -- 172.21.15.27:0/1783842171 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9000f6b0 con 0x7faf9c102990 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.155+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9c1a2190 con 0x7faf9c102990 2024-04-25T01:23:25.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.156+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9c108b30 con 0x7faf9c102990 2024-04-25T01:23:25.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.157+0000 7faf8b7fe640 1 -- 172.21.15.27:0/1783842171 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7faf900109d0 con 0x7faf9c102990 2024-04-25T01:23:25.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.158+0000 7faf8b7fe640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faf840a2c10 0x7faf840a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.158+0000 7faf9a59c640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faf840a2c10 0x7faf840a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.158+0000 7faf8b7fe640 1 -- 172.21.15.27:0/1783842171 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7faf900ceec0 con 0x7faf9c102990 2024-04-25T01:23:25.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.158+0000 7faf9a59c640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faf840a2c10 0x7faf840a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faf8c006010 tx=0x7faf8c032cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.162+0000 7faf8b7fe640 1 -- 172.21.15.27:0/1783842171 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faf90009de0 con 0x7faf9c102990 2024-04-25T01:23:25.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:24 smithi153 ceph-mon[32621]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-04-25T01:23:25.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:24 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1501684678' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:23:25.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:24 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] up:active 2024-04-25T01:23:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:24 smithi153 ceph-mon[32621]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T01:23:25.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:24 smithi153 ceph-mon[32621]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:23:25.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:24 smithi139 ceph-mon[32760]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-04-25T01:23:25.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:24 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1501684678' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:23:25.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:24 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] up:active 2024-04-25T01:23:25.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:24 smithi139 ceph-mon[32760]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T01:23:25.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:24 smithi139 ceph-mon[32760]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:23:25.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.375+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7faf9c061c10 con 0x7faf840a2c10 2024-04-25T01:23:25.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.402+0000 7faf8b7fe640 1 -- 172.21.15.27:0/1783842171 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faf9c061c10 con 0x7faf840a2c10 2024-04-25T01:23:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faf840a2c10 msgr2=0x7faf840a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faf840a2c10 0x7faf840a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faf8c006010 tx=0x7faf8c032cf0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c102990 msgr2=0x7faf9c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c102990 0x7faf9c073b30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faf9000d950 tx=0x7faf9000de10 comp rx=0 tx=0).stop 2024-04-25T01:23:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 shutdown_connections 2024-04-25T01:23:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7faf840a2c10 0x7faf840a50d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c103350 0x7faf9c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf9c102990 0x7faf9c073b30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 --2- 172.21.15.27:0/1783842171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf9c1017a0 0x7faf9c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 >> 172.21.15.27:0/1783842171 conn(0x7faf9c0fcd10 msgr2=0x7faf9c105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 shutdown_connections 2024-04-25T01:23:25.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.404+0000 7fafa0963640 1 -- 172.21.15.27:0/1783842171 wait complete. 2024-04-25T01:23:25.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T01:23:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 -- 172.21.15.27:0/3413483484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0102230 msgr2=0x7f8db0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3413483484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0102230 0x7f8db0108ac0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009ca0 tx=0x7f8da002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 -- 172.21.15.27:0/3413483484 shutdown_connections 2024-04-25T01:23:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3413483484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0102230 0x7f8db0108ac0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3413483484 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db0101870 0x7f8db0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3413483484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db0100680 0x7f8db0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 -- 172.21.15.27:0/3413483484 >> 172.21.15.27:0/3413483484 conn(0x7f8db00fbbf0 msgr2=0x7f8db00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 -- 172.21.15.27:0/3413483484 shutdown_connections 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.521+0000 7f8db56cd640 1 -- 172.21.15.27:0/3413483484 wait complete. 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8db56cd640 1 Processor -- start 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8db56cd640 1 -- start start 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8db56cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0100680 0x7f8db0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8db56cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db0101870 0x7f8db0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8db56cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db0102230 0x7f8db019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8db56cd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8db010c2e0 con 0x7f8db0100680 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8db56cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8db010c160 con 0x7f8db0101870 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8db56cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8db010c460 con 0x7f8db0102230 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8daeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0100680 0x7f8db0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8daeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0100680 0x7f8db0197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38630/0 (socket says 172.21.15.27:38630) 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.522+0000 7f8daeffd640 1 -- 172.21.15.27:0/3678259953 learned_addr learned my addr 172.21.15.27:0/3678259953 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8dae7fc640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db0101870 0x7f8db0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8daf7fe640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db0102230 0x7f8db019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8daeffd640 1 -- 172.21.15.27:0/3678259953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db0102230 msgr2=0x7f8db019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8daeffd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db0102230 0x7f8db019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8daeffd640 1 -- 172.21.15.27:0/3678259953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db0101870 msgr2=0x7f8db0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8daeffd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db0101870 0x7f8db0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8daeffd640 1 -- 172.21.15.27:0/3678259953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db019f6c0 con 0x7f8db0100680 2024-04-25T01:23:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8daeffd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0100680 0x7f8db0197aa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8da800d770 tx=0x7f8da800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8d97fff640 1 -- 172.21.15.27:0/3678259953 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8017070 con 0x7f8db0100680 2024-04-25T01:23:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db019f9b0 con 0x7f8db0100680 2024-04-25T01:23:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8d97fff640 1 -- 172.21.15.27:0/3678259953 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da8009c30 con 0x7f8db0100680 2024-04-25T01:23:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8d97fff640 1 -- 172.21.15.27:0/3678259953 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8003dd0 con 0x7f8db0100680 2024-04-25T01:23:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.523+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db019ff50 con 0x7f8db0100680 2024-04-25T01:23:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.524+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db0068560 con 0x7f8db0100680 2024-04-25T01:23:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.525+0000 7f8d97fff640 1 -- 172.21.15.27:0/3678259953 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8da8004000 con 0x7f8db0100680 2024-04-25T01:23:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.526+0000 7f8d97fff640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d980a2b90 0x7f8d980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.526+0000 7f8dae7fc640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d980a2b90 0x7f8d980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.526+0000 7f8d97fff640 1 -- 172.21.15.27:0/3678259953 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8da80cecc0 con 0x7f8db0100680 2024-04-25T01:23:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.526+0000 7f8dae7fc640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d980a2b90 0x7f8d980a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8db0198fb0 tx=0x7f8da4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.530+0000 7f8d97fff640 1 -- 172.21.15.27:0/3678259953 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8da809b9e0 con 0x7f8db0100680 2024-04-25T01:23:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.747+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f8db00fddb0 con 0x7f8d980a2b90 2024-04-25T01:23:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.753+0000 7f8d97fff640 1 -- 172.21.15.27:0/3678259953 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8db00fddb0 con 0x7f8d980a2b90 2024-04-25T01:23:25.734 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/qa/sv_1/2c96bf5e-0240-4a00-9816-53b5e1baa41e 2024-04-25T01:23:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d980a2b90 msgr2=0x7f8d980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d980a2b90 0x7f8d980a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8db0198fb0 tx=0x7f8da4005f90 comp rx=0 tx=0).stop 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0100680 msgr2=0x7f8db0197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0100680 0x7f8db0197aa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8da800d770 tx=0x7f8da800dc30 comp rx=0 tx=0).stop 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 shutdown_connections 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d980a2b90 0x7f8d980a5050 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8db0102230 0x7f8db019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db0101870 0x7f8db0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 --2- 172.21.15.27:0/3678259953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8db0100680 0x7f8db0197aa0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 >> 172.21.15.27:0/3678259953 conn(0x7f8db00fbbf0 msgr2=0x7f8db00fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 shutdown_connections 2024-04-25T01:23:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.755+0000 7f8db56cd640 1 -- 172.21.15.27:0/3678259953 wait complete. 2024-04-25T01:23:25.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:23:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.868+0000 7efebd528640 1 -- 172.21.15.27:0/719651612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb80ffff0 msgr2=0x7efeb8100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.868+0000 7efebd528640 1 --2- 172.21.15.27:0/719651612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb80ffff0 0x7efeb8100410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efea8009900 tx=0x7efea802a410 comp rx=0 tx=0).stop 2024-04-25T01:23:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.868+0000 7efebd528640 1 -- 172.21.15.27:0/719651612 shutdown_connections 2024-04-25T01:23:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.868+0000 7efebd528640 1 --2- 172.21.15.27:0/719651612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb81012f0 0x7efeb8101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.868+0000 7efebd528640 1 --2- 172.21.15.27:0/719651612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efeb8100950 0x7efeb8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.868+0000 7efebd528640 1 --2- 172.21.15.27:0/719651612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb80ffff0 0x7efeb8100410 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.869+0000 7efebd528640 1 -- 172.21.15.27:0/719651612 >> 172.21.15.27:0/719651612 conn(0x7efeb80fbbf0 msgr2=0x7efeb80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:25.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.869+0000 7efebd528640 1 -- 172.21.15.27:0/719651612 shutdown_connections 2024-04-25T01:23:25.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.869+0000 7efebd528640 1 -- 172.21.15.27:0/719651612 wait complete. 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.869+0000 7efebd528640 1 Processor -- start 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efebd528640 1 -- start start 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efebd528640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb80ffff0 0x7efeb8199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efebd528640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efeb8100950 0x7efeb819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efebd528640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb81012f0 0x7efeb81a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efebd528640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efeb810f630 con 0x7efeb81012f0 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efebd528640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efeb810f4b0 con 0x7efeb8100950 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efebd528640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efeb810f7b0 con 0x7efeb80ffff0 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb81012f0 0x7efeb81a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb81012f0 0x7efeb81a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38648/0 (socket says 172.21.15.27:38648) 2024-04-25T01:23:25.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 -- 172.21.15.27:0/1282867695 learned_addr learned my addr 172.21.15.27:0/1282867695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb67fc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efeb8100950 0x7efeb819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb6ffd640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb80ffff0 0x7efeb8199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 -- 172.21.15.27:0/1282867695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb80ffff0 msgr2=0x7efeb8199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb80ffff0 0x7efeb8199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 -- 172.21.15.27:0/1282867695 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efeb8100950 msgr2=0x7efeb819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efeb8100950 0x7efeb819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 -- 172.21.15.27:0/1282867695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb81a18b0 con 0x7efeb81012f0 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.870+0000 7efeb77fe640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb81012f0 0x7efeb81a1210 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efea000e970 tx=0x7efea000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.871+0000 7efeaffff640 1 -- 172.21.15.27:0/1282867695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea000ccc0 con 0x7efeb81012f0 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.871+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb81a1ba0 con 0x7efeb81012f0 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.871+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb81a2110 con 0x7efeb81012f0 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.871+0000 7efeaffff640 1 -- 172.21.15.27:0/1282867695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea000ce60 con 0x7efeb81012f0 2024-04-25T01:23:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.871+0000 7efeaffff640 1 -- 172.21.15.27:0/1282867695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0010450 con 0x7efeb81012f0 2024-04-25T01:23:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.872+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe980052c0 con 0x7efeb81012f0 2024-04-25T01:23:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.873+0000 7efeaffff640 1 -- 172.21.15.27:0/1282867695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7efea00046c0 con 0x7efeb81012f0 2024-04-25T01:23:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.874+0000 7efeaffff640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efea40a2c10 0x7efea40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.874+0000 7efeb6ffd640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efea40a2c10 0x7efea40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.874+0000 7efeaffff640 1 -- 172.21.15.27:0/1282867695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efea00cf560 con 0x7efeb81012f0 2024-04-25T01:23:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.874+0000 7efeb6ffd640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efea40a2c10 0x7efea40a50d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efea8031440 tx=0x7efea80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:25.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.879+0000 7efeaffff640 1 -- 172.21.15.27:0/1282867695 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efea009c280 con 0x7efeb81012f0 2024-04-25T01:23:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:25.953+0000 7efeaffff640 1 -- 172.21.15.27:0/1282867695 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7efea00981f0 con 0x7efeb81012f0 2024-04-25T01:23:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.111+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe980055b0 con 0x7efeb81012f0 2024-04-25T01:23:26.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.112+0000 7efeaffff640 1 -- 172.21.15.27:0/1282867695 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7efea00c6b80 con 0x7efeb81012f0 2024-04-25T01:23:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:26.093 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:23:23.940979+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:23:26.094 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-25T01:23:26.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.113+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efea40a2c10 msgr2=0x7efea40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.113+0000 7efebd528640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efea40a2c10 0x7efea40a50d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efea8031440 tx=0x7efea80315b0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb81012f0 msgr2=0x7efeb81a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb81012f0 0x7efeb81a1210 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efea000e970 tx=0x7efea000ed70 comp rx=0 tx=0).stop 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 shutdown_connections 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7efea40a2c10 0x7efea40a50d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeb81012f0 0x7efeb81a1210 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efeb8100950 0x7efeb819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 --2- 172.21.15.27:0/1282867695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb80ffff0 0x7efeb8199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 >> 172.21.15.27:0/1282867695 conn(0x7efeb80fbbf0 msgr2=0x7efeb80fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 shutdown_connections 2024-04-25T01:23:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.114+0000 7efebd528640 1 -- 172.21.15.27:0/1282867695 wait complete. 2024-04-25T01:23:26.105 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:25 smithi027 ceph-mon[29197]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.105 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:25 smithi027 ceph-mon[29197]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.105 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:25 smithi027 ceph-mon[29197]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 -- 172.21.15.27:0/1904066183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52941006a0 msgr2=0x7f5294100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 --2- 172.21.15.27:0/1904066183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52941006a0 0x7f5294100ac0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5290009900 tx=0x7f529002a410 comp rx=0 tx=0).stop 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 -- 172.21.15.27:0/1904066183 shutdown_connections 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 --2- 172.21.15.27:0/1904066183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5294102250 0x7f5294108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 --2- 172.21.15.27:0/1904066183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294101890 0x7f5294101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 --2- 172.21.15.27:0/1904066183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52941006a0 0x7f5294100ac0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 -- 172.21.15.27:0/1904066183 >> 172.21.15.27:0/1904066183 conn(0x7f52940fbc30 msgr2=0x7f52940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 -- 172.21.15.27:0/1904066183 shutdown_connections 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.227+0000 7f529bd1b640 1 -- 172.21.15.27:0/1904066183 wait complete. 2024-04-25T01:23:26.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529bd1b640 1 Processor -- start 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529bd1b640 1 -- start start 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529bd1b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52941006a0 0x7f5294196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529bd1b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294101890 0x7f5294196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529bd1b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5294102250 0x7f529419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529bd1b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f529410c390 con 0x7f5294102250 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529bd1b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f529410c210 con 0x7f52941006a0 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529bd1b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f529410c510 con 0x7f5294101890 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529a2b8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5294102250 0x7f529419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529a2b8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5294102250 0x7f529419e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38660/0 (socket says 172.21.15.27:38660) 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f529a2b8640 1 -- 172.21.15.27:0/2379213433 learned_addr learned my addr 172.21.15.27:0/2379213433 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f52992b6640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294101890 0x7f5294196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.228+0000 7f5299ab7640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52941006a0 0x7f5294196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f529a2b8640 1 -- 172.21.15.27:0/2379213433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294101890 msgr2=0x7f5294196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f529a2b8640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294101890 0x7f5294196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f529a2b8640 1 -- 172.21.15.27:0/2379213433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52941006a0 msgr2=0x7f5294196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f529a2b8640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52941006a0 0x7f5294196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f529a2b8640 1 -- 172.21.15.27:0/2379213433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529419e680 con 0x7f5294102250 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f529a2b8640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5294102250 0x7f529419e010 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f528c00b970 tx=0x7f528c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f528affd640 1 -- 172.21.15.27:0/2379213433 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c019070 con 0x7f5294102250 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f528affd640 1 -- 172.21.15.27:0/2379213433 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528c00cb40 con 0x7f5294102250 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529419e970 con 0x7f5294102250 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f528affd640 1 -- 172.21.15.27:0/2379213433 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c007650 con 0x7f5294102250 2024-04-25T01:23:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.229+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529419eef0 con 0x7f5294102250 2024-04-25T01:23:26.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.230+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5294107a40 con 0x7f5294102250 2024-04-25T01:23:26.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.231+0000 7f528affd640 1 -- 172.21.15.27:0/2379213433 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f528c01c070 con 0x7f5294102250 2024-04-25T01:23:26.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.232+0000 7f528affd640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f52800a2c20 0x7f52800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.232+0000 7f5299ab7640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f52800a2c20 0x7f52800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.232+0000 7f528affd640 1 -- 172.21.15.27:0/2379213433 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f528c0ce8f0 con 0x7f5294102250 2024-04-25T01:23:26.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.232+0000 7f5299ab7640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f52800a2c20 0x7f52800a50e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5290005ab0 tx=0x7f5290000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.237+0000 7f528affd640 1 -- 172.21.15.27:0/2379213433 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f528c09af70 con 0x7f5294102250 2024-04-25T01:23:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:25 smithi153 ceph-mon[32621]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:25 smithi153 ceph-mon[32621]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:25 smithi153 ceph-mon[32621]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:25 smithi139 ceph-mon[32760]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:25 smithi139 ceph-mon[32760]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:25 smithi139 ceph-mon[32760]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:26.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.422+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f529402d380 con 0x7f5294102250 2024-04-25T01:23:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.424+0000 7f528affd640 1 -- 172.21.15.27:0/2379213433 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7f528c09f270 con 0x7f5294102250 2024-04-25T01:23:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:26.406 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f52800a2c20 msgr2=0x7f52800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f52800a2c20 0x7f52800a50e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5290005ab0 tx=0x7f5290000bc0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5294102250 msgr2=0x7f529419e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5294102250 0x7f529419e010 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f528c00b970 tx=0x7f528c00be30 comp rx=0 tx=0).stop 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 shutdown_connections 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f52800a2c20 0x7f52800a50e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5294102250 0x7f529419e010 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294101890 0x7f5294196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 --2- 172.21.15.27:0/2379213433 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52941006a0 0x7f5294196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 >> 172.21.15.27:0/2379213433 conn(0x7f52940fbc30 msgr2=0x7f5294104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 shutdown_connections 2024-04-25T01:23:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.426+0000 7f529bd1b640 1 -- 172.21.15.27:0/2379213433 wait complete. 2024-04-25T01:23:26.419 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:23:26.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- 172.21.15.27:0/1909111190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7541006a0 msgr2=0x7fa754100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/1909111190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7541006a0 0x7fa754100ac0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa748009900 tx=0x7fa74802a410 comp rx=0 tx=0).stop 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- 172.21.15.27:0/1909111190 shutdown_connections 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/1909111190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa754102250 0x7fa754108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/1909111190 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa754101890 0x7fa754101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/1909111190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7541006a0 0x7fa754100ac0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- 172.21.15.27:0/1909111190 >> 172.21.15.27:0/1909111190 conn(0x7fa7540fbc30 msgr2=0x7fa7540fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- 172.21.15.27:0/1909111190 shutdown_connections 2024-04-25T01:23:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- 172.21.15.27:0/1909111190 wait complete. 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 Processor -- start 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- start start 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7541006a0 0x7fa75406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa754101890 0x7fa75419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa754102250 0x7fa7541a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa75410c3f0 con 0x7fa754102250 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa75410c270 con 0x7fa754101890 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.540+0000 7fa75c2d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa75410c570 con 0x7fa7541006a0 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa754102250 0x7fa7541a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa754102250 0x7fa7541a1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38678/0 (socket says 172.21.15.27:38678) 2024-04-25T01:23:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 -- 172.21.15.27:0/3348522582 learned_addr learned my addr 172.21.15.27:0/3348522582 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75986f640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa754101890 0x7fa75419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a070640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7541006a0 0x7fa75406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 -- 172.21.15.27:0/3348522582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7541006a0 msgr2=0x7fa75406a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7541006a0 0x7fa75406a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 -- 172.21.15.27:0/3348522582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa754101890 msgr2=0x7fa75419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa754101890 0x7fa75419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 -- 172.21.15.27:0/3348522582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7541a1a90 con 0x7fa754102250 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a070640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7541006a0 0x7fa75406a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75a871640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa754102250 0x7fa7541a1340 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009fd0 tx=0x7fa74c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa7477fe640 1 -- 172.21.15.27:0/3348522582 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c011830 con 0x7fa754102250 2024-04-25T01:23:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa7477fe640 1 -- 172.21.15.27:0/3348522582 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa74c016070 con 0x7fa754102250 2024-04-25T01:23:26.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7541a1d80 con 0x7fa754102250 2024-04-25T01:23:26.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa7477fe640 1 -- 172.21.15.27:0/3348522582 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c00f2e0 con 0x7fa754102250 2024-04-25T01:23:26.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.541+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7541a22a0 con 0x7fa754102250 2024-04-25T01:23:26.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.542+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa754107ab0 con 0x7fa754102250 2024-04-25T01:23:26.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.544+0000 7fa7477fe640 1 -- 172.21.15.27:0/3348522582 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa74c0119d0 con 0x7fa754102250 2024-04-25T01:23:26.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.544+0000 7fa7477fe640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa7400a2c50 0x7fa7400a5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.544+0000 7fa75a070640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa7400a2c50 0x7fa7400a5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.544+0000 7fa7477fe640 1 -- 172.21.15.27:0/3348522582 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa74c0ce950 con 0x7fa754102250 2024-04-25T01:23:26.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.545+0000 7fa75a070640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa7400a2c50 0x7fa7400a5110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa748006010 tx=0x7fa748000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.549+0000 7fa7477fe640 1 -- 172.21.15.27:0/3348522582 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa74c09b620 con 0x7fa754102250 2024-04-25T01:23:26.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.730+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa754061b80 con 0x7fa7400a2c50 2024-04-25T01:23:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.740+0000 7fa7477fe640 1 -- 172.21.15.27:0/3348522582 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340547 (secure 0 0 0) 0x7fa754061b80 con 0x7fa7400a2c50 2024-04-25T01:23:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:26.728 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa7400a2c50 msgr2=0x7fa7400a5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa7400a2c50 0x7fa7400a5110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa748006010 tx=0x7fa748000bc0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa754102250 msgr2=0x7fa7541a1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa754102250 0x7fa7541a1340 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009fd0 tx=0x7fa74c00b8e0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 shutdown_connections 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa7400a2c50 0x7fa7400a5110 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa754102250 0x7fa7541a1340 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa754101890 0x7fa75419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 --2- 172.21.15.27:0/3348522582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7541006a0 0x7fa75406a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 >> 172.21.15.27:0/3348522582 conn(0x7fa7540fbc30 msgr2=0x7fa754104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 shutdown_connections 2024-04-25T01:23:26.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.744+0000 7fa75c2d4640 1 -- 172.21.15.27:0/3348522582 wait complete. 2024-04-25T01:23:26.736 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-25T01:23:25.379265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595412,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":141,"num_read_kb":124,"num_write":280,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356576,"kb_used_data":11576,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585248,"statfs":{"total":1151940427776,"available":1151575293952,"internally_reserved":0,"allocated":11853824,"data_stored":7515076,"data_compressed":283599,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2634,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":54,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003354"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423666+0000","last_change":"2024-04-25T01:22:57.847700+0000","last_active":"2024-04-25T01:23:03.423666+0000","last_peered":"2024-04-25T01:23:03.423666+0000","last_clean":"2024-04-25T01:23:03.423666+0000","last_became_active":"2024-04-25T01:22:57.846877+0000","last_became_peered":"2024-04-25T01:22:57.846877+0000","last_unstale":"2024-04-25T01:23:03.423666+0000","last_undegraded":"2024-04-25T01:23:03.423666+0000","last_fullsized":"2024-04-25T01:23:03.423666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:05.918261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980817+0000","last_change":"2024-04-25T01:22:57.848396+0000","last_active":"2024-04-25T01:23:02.980817+0000","last_peered":"2024-04-25T01:23:02.980817+0000","last_clean":"2024-04-25T01:23:02.980817+0000","last_became_active":"2024-04-25T01:22:57.847971+0000","last_became_peered":"2024-04-25T01:22:57.847971+0000","last_unstale":"2024-04-25T01:23:02.980817+0000","last_undegraded":"2024-04-25T01:23:02.980817+0000","last_fullsized":"2024-04-25T01:23:02.980817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:13.143828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980666+0000","last_change":"2024-04-25T01:22:57.856908+0000","last_active":"2024-04-25T01:23:02.980666+0000","last_peered":"2024-04-25T01:23:02.980666+0000","last_clean":"2024-04-25T01:23:02.980666+0000","last_became_active":"2024-04-25T01:22:57.856218+0000","last_became_peered":"2024-04-25T01:22:57.856218+0000","last_unstale":"2024-04-25T01:23:02.980666+0000","last_undegraded":"2024-04-25T01:23:02.980666+0000","last_fullsized":"2024-04-25T01:23:02.980666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:51.971222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424836+0000","last_change":"2024-04-25T01:22:57.849022+0000","last_active":"2024-04-25T01:23:03.424836+0000","last_peered":"2024-04-25T01:23:03.424836+0000","last_clean":"2024-04-25T01:23:03.424836+0000","last_became_active":"2024-04-25T01:22:57.848303+0000","last_became_peered":"2024-04-25T01:22:57.848303+0000","last_unstale":"2024-04-25T01:23:03.424836+0000","last_undegraded":"2024-04-25T01:23:03.424836+0000","last_fullsized":"2024-04-25T01:23:03.424836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:52.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423540+0000","last_change":"2024-04-25T01:22:57.849039+0000","last_active":"2024-04-25T01:23:03.423540+0000","last_peered":"2024-04-25T01:23:03.423540+0000","last_clean":"2024-04-25T01:23:03.423540+0000","last_became_active":"2024-04-25T01:22:57.848702+0000","last_became_peered":"2024-04-25T01:22:57.848702+0000","last_unstale":"2024-04-25T01:23:03.423540+0000","last_undegraded":"2024-04-25T01:23:03.423540+0000","last_fullsized":"2024-04-25T01:23:03.423540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:38.724814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423384+0000","last_change":"2024-04-25T01:22:57.855733+0000","last_active":"2024-04-25T01:23:03.423384+0000","last_peered":"2024-04-25T01:23:03.423384+0000","last_clean":"2024-04-25T01:23:03.423384+0000","last_became_active":"2024-04-25T01:22:57.855075+0000","last_became_peered":"2024-04-25T01:22:57.855075+0000","last_unstale":"2024-04-25T01:23:03.423384+0000","last_undegraded":"2024-04-25T01:23:03.423384+0000","last_fullsized":"2024-04-25T01:23:03.423384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:24.563461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879021+0000","last_change":"2024-04-25T01:22:57.849693+0000","last_active":"2024-04-25T01:23:02.879021+0000","last_peered":"2024-04-25T01:23:02.879021+0000","last_clean":"2024-04-25T01:23:02.879021+0000","last_became_active":"2024-04-25T01:22:57.849324+0000","last_became_peered":"2024-04-25T01:22:57.849324+0000","last_unstale":"2024-04-25T01:23:02.879021+0000","last_undegraded":"2024-04-25T01:23:02.879021+0000","last_fullsized":"2024-04-25T01:23:02.879021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:09.936496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422210+0000","last_change":"2024-04-25T01:22:57.847962+0000","last_active":"2024-04-25T01:23:03.422210+0000","last_peered":"2024-04-25T01:23:03.422210+0000","last_clean":"2024-04-25T01:23:03.422210+0000","last_became_active":"2024-04-25T01:22:57.847592+0000","last_became_peered":"2024-04-25T01:22:57.847592+0000","last_unstale":"2024-04-25T01:23:03.422210+0000","last_undegraded":"2024-04-25T01:23:03.422210+0000","last_fullsized":"2024-04-25T01:23:03.422210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:45.594118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422873+0000","last_change":"2024-04-25T01:22:57.860711+0000","last_active":"2024-04-25T01:23:03.422873+0000","last_peered":"2024-04-25T01:23:03.422873+0000","last_clean":"2024-04-25T01:23:03.422873+0000","last_became_active":"2024-04-25T01:22:57.860393+0000","last_became_peered":"2024-04-25T01:22:57.860393+0000","last_unstale":"2024-04-25T01:23:03.422873+0000","last_undegraded":"2024-04-25T01:23:03.422873+0000","last_fullsized":"2024-04-25T01:23:03.422873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:31.068095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424274+0000","last_change":"2024-04-25T01:22:57.853741+0000","last_active":"2024-04-25T01:23:03.424274+0000","last_peered":"2024-04-25T01:23:03.424274+0000","last_clean":"2024-04-25T01:23:03.424274+0000","last_became_active":"2024-04-25T01:22:57.851015+0000","last_became_peered":"2024-04-25T01:22:57.851015+0000","last_unstale":"2024-04-25T01:23:03.424274+0000","last_undegraded":"2024-04-25T01:23:03.424274+0000","last_fullsized":"2024-04-25T01:23:03.424274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:45.831335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879777+0000","last_change":"2024-04-25T01:22:57.853354+0000","last_active":"2024-04-25T01:23:02.879777+0000","last_peered":"2024-04-25T01:23:02.879777+0000","last_clean":"2024-04-25T01:23:02.879777+0000","last_became_active":"2024-04-25T01:22:57.853043+0000","last_became_peered":"2024-04-25T01:22:57.853043+0000","last_unstale":"2024-04-25T01:23:02.879777+0000","last_undegraded":"2024-04-25T01:23:02.879777+0000","last_fullsized":"2024-04-25T01:23:02.879777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:54.378751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424672+0000","last_change":"2024-04-25T01:22:57.855542+0000","last_active":"2024-04-25T01:23:03.424672+0000","last_peered":"2024-04-25T01:23:03.424672+0000","last_clean":"2024-04-25T01:23:03.424672+0000","last_became_active":"2024-04-25T01:22:57.855196+0000","last_became_peered":"2024-04-25T01:22:57.855196+0000","last_unstale":"2024-04-25T01:23:03.424672+0000","last_undegraded":"2024-04-25T01:23:03.424672+0000","last_fullsized":"2024-04-25T01:23:03.424672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:56.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827084+0000","last_change":"2024-04-25T01:22:57.829353+0000","last_active":"2024-04-25T01:23:03.827084+0000","last_peered":"2024-04-25T01:23:03.827084+0000","last_clean":"2024-04-25T01:23:03.827084+0000","last_became_active":"2024-04-25T01:22:57.828937+0000","last_became_peered":"2024-04-25T01:22:57.828937+0000","last_unstale":"2024-04-25T01:23:03.827084+0000","last_undegraded":"2024-04-25T01:23:03.827084+0000","last_fullsized":"2024-04-25T01:23:03.827084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.541556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980409+0000","last_change":"2024-04-25T01:22:57.854635+0000","last_active":"2024-04-25T01:23:02.980409+0000","last_peered":"2024-04-25T01:23:02.980409+0000","last_clean":"2024-04-25T01:23:02.980409+0000","last_became_active":"2024-04-25T01:22:57.854292+0000","last_became_peered":"2024-04-25T01:22:57.854292+0000","last_unstale":"2024-04-25T01:23:02.980409+0000","last_undegraded":"2024-04-25T01:23:02.980409+0000","last_fullsized":"2024-04-25T01:23:02.980409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:48.209059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980313+0000","last_change":"2024-04-25T01:22:57.852210+0000","last_active":"2024-04-25T01:23:02.980313+0000","last_peered":"2024-04-25T01:23:02.980313+0000","last_clean":"2024-04-25T01:23:02.980313+0000","last_became_active":"2024-04-25T01:22:57.851833+0000","last_became_peered":"2024-04-25T01:22:57.851833+0000","last_unstale":"2024-04-25T01:23:02.980313+0000","last_undegraded":"2024-04-25T01:23:02.980313+0000","last_fullsized":"2024-04-25T01:23:02.980313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:14.664785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422420+0000","last_change":"2024-04-25T01:22:57.849357+0000","last_active":"2024-04-25T01:23:03.422420+0000","last_peered":"2024-04-25T01:23:03.422420+0000","last_clean":"2024-04-25T01:23:03.422420+0000","last_became_active":"2024-04-25T01:22:57.849050+0000","last_became_peered":"2024-04-25T01:22:57.849050+0000","last_unstale":"2024-04-25T01:23:03.422420+0000","last_undegraded":"2024-04-25T01:23:03.422420+0000","last_fullsized":"2024-04-25T01:23:03.422420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:37.669424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423551+0000","last_change":"2024-04-25T01:22:57.844790+0000","last_active":"2024-04-25T01:23:03.423551+0000","last_peered":"2024-04-25T01:23:03.423551+0000","last_clean":"2024-04-25T01:23:03.423551+0000","last_became_active":"2024-04-25T01:22:57.844442+0000","last_became_peered":"2024-04-25T01:22:57.844442+0000","last_unstale":"2024-04-25T01:23:03.423551+0000","last_undegraded":"2024-04-25T01:23:03.423551+0000","last_fullsized":"2024-04-25T01:23:03.423551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:33.766034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878818+0000","last_change":"2024-04-25T01:22:57.844430+0000","last_active":"2024-04-25T01:23:02.878818+0000","last_peered":"2024-04-25T01:23:02.878818+0000","last_clean":"2024-04-25T01:23:02.878818+0000","last_became_active":"2024-04-25T01:22:57.844110+0000","last_became_peered":"2024-04-25T01:22:57.844110+0000","last_unstale":"2024-04-25T01:23:02.878818+0000","last_undegraded":"2024-04-25T01:23:02.878818+0000","last_fullsized":"2024-04-25T01:23:02.878818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:50.348249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879371+0000","last_change":"2024-04-25T01:22:57.852705+0000","last_active":"2024-04-25T01:23:02.879371+0000","last_peered":"2024-04-25T01:23:02.879371+0000","last_clean":"2024-04-25T01:23:02.879371+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:23:02.879371+0000","last_undegraded":"2024-04-25T01:23:02.879371+0000","last_fullsized":"2024-04-25T01:23:02.879371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:10.320459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879033+0000","last_change":"2024-04-25T01:23:01.879341+0000","last_active":"2024-04-25T01:23:02.879033+0000","last_peered":"2024-04-25T01:23:02.879033+0000","last_clean":"2024-04-25T01:23:02.879033+0000","last_became_active":"2024-04-25T01:23:01.878881+0000","last_became_peered":"2024-04-25T01:23:01.878881+0000","last_unstale":"2024-04-25T01:23:02.879033+0000","last_undegraded":"2024-04-25T01:23:02.879033+0000","last_fullsized":"2024-04-25T01:23:02.879033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:46.036868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878919+0000","last_change":"2024-04-25T01:22:56.839867+0000","last_active":"2024-04-25T01:23:02.878919+0000","last_peered":"2024-04-25T01:23:02.878919+0000","last_clean":"2024-04-25T01:23:02.878919+0000","last_became_active":"2024-04-25T01:22:56.839521+0000","last_became_peered":"2024-04-25T01:22:56.839521+0000","last_unstale":"2024-04-25T01:23:02.878919+0000","last_undegraded":"2024-04-25T01:23:02.878919+0000","last_fullsized":"2024-04-25T01:23:02.878919+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:00.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422759+0000","last_change":"2024-04-25T01:22:57.855922+0000","last_active":"2024-04-25T01:23:03.422759+0000","last_peered":"2024-04-25T01:23:03.422759+0000","last_clean":"2024-04-25T01:23:03.422759+0000","last_became_active":"2024-04-25T01:22:57.855584+0000","last_became_peered":"2024-04-25T01:22:57.855584+0000","last_unstale":"2024-04-25T01:23:03.422759+0000","last_undegraded":"2024-04-25T01:23:03.422759+0000","last_fullsized":"2024-04-25T01:23:03.422759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:02.791269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423924+0000","last_change":"2024-04-25T01:23:01.872340+0000","last_active":"2024-04-25T01:23:03.423924+0000","last_peered":"2024-04-25T01:23:03.423924+0000","last_clean":"2024-04-25T01:23:03.423924+0000","last_became_active":"2024-04-25T01:23:01.871900+0000","last_became_peered":"2024-04-25T01:23:01.871900+0000","last_unstale":"2024-04-25T01:23:03.423924+0000","last_undegraded":"2024-04-25T01:23:03.423924+0000","last_fullsized":"2024-04-25T01:23:03.423924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:11.918022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422828+0000","last_change":"2024-04-25T01:22:56.843851+0000","last_active":"2024-04-25T01:23:03.422828+0000","last_peered":"2024-04-25T01:23:03.422828+0000","last_clean":"2024-04-25T01:23:03.422828+0000","last_became_active":"2024-04-25T01:22:56.843232+0000","last_became_peered":"2024-04-25T01:22:56.843232+0000","last_unstale":"2024-04-25T01:23:03.422828+0000","last_undegraded":"2024-04-25T01:23:03.422828+0000","last_fullsized":"2024-04-25T01:23:03.422828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:13.758328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980740+0000","last_change":"2024-04-25T01:22:57.857686+0000","last_active":"2024-04-25T01:23:02.980740+0000","last_peered":"2024-04-25T01:23:02.980740+0000","last_clean":"2024-04-25T01:23:02.980740+0000","last_became_active":"2024-04-25T01:22:57.857175+0000","last_became_peered":"2024-04-25T01:22:57.857175+0000","last_unstale":"2024-04-25T01:23:02.980740+0000","last_undegraded":"2024-04-25T01:23:02.980740+0000","last_fullsized":"2024-04-25T01:23:02.980740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:24.777861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424365+0000","last_change":"2024-04-25T01:23:01.867914+0000","last_active":"2024-04-25T01:23:03.424365+0000","last_peered":"2024-04-25T01:23:03.424365+0000","last_clean":"2024-04-25T01:23:03.424365+0000","last_became_active":"2024-04-25T01:23:01.867103+0000","last_became_peered":"2024-04-25T01:23:01.867103+0000","last_unstale":"2024-04-25T01:23:03.424365+0000","last_undegraded":"2024-04-25T01:23:03.424365+0000","last_fullsized":"2024-04-25T01:23:03.424365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:26.425433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.941671+0000","last_change":"2024-04-25T01:22:56.841310+0000","last_active":"2024-04-25T01:23:22.941671+0000","last_peered":"2024-04-25T01:23:22.941671+0000","last_clean":"2024-04-25T01:23:22.941671+0000","last_became_active":"2024-04-25T01:22:56.840882+0000","last_became_peered":"2024-04-25T01:22:56.840882+0000","last_unstale":"2024-04-25T01:23:22.941671+0000","last_undegraded":"2024-04-25T01:23:22.941671+0000","last_fullsized":"2024-04-25T01:23:22.941671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:29.971448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422919+0000","last_change":"2024-04-25T01:22:57.842810+0000","last_active":"2024-04-25T01:23:03.422919+0000","last_peered":"2024-04-25T01:23:03.422919+0000","last_clean":"2024-04-25T01:23:03.422919+0000","last_became_active":"2024-04-25T01:22:57.842457+0000","last_became_peered":"2024-04-25T01:22:57.842457+0000","last_unstale":"2024-04-25T01:23:03.422919+0000","last_undegraded":"2024-04-25T01:23:03.422919+0000","last_fullsized":"2024-04-25T01:23:03.422919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:45.512037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423655+0000","last_change":"2024-04-25T01:23:01.877298+0000","last_active":"2024-04-25T01:23:03.423655+0000","last_peered":"2024-04-25T01:23:03.423655+0000","last_clean":"2024-04-25T01:23:03.423655+0000","last_became_active":"2024-04-25T01:23:01.876874+0000","last_became_peered":"2024-04-25T01:23:01.876874+0000","last_unstale":"2024-04-25T01:23:03.423655+0000","last_undegraded":"2024-04-25T01:23:03.423655+0000","last_fullsized":"2024-04-25T01:23:03.423655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:35:08.221388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424003+0000","last_change":"2024-04-25T01:22:56.838914+0000","last_active":"2024-04-25T01:23:03.424003+0000","last_peered":"2024-04-25T01:23:03.424003+0000","last_clean":"2024-04-25T01:23:03.424003+0000","last_became_active":"2024-04-25T01:22:56.838616+0000","last_became_peered":"2024-04-25T01:22:56.838616+0000","last_unstale":"2024-04-25T01:23:03.424003+0000","last_undegraded":"2024-04-25T01:23:03.424003+0000","last_fullsized":"2024-04-25T01:23:03.424003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:20.761770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424890+0000","last_change":"2024-04-25T01:22:57.851210+0000","last_active":"2024-04-25T01:23:03.424890+0000","last_peered":"2024-04-25T01:23:03.424890+0000","last_clean":"2024-04-25T01:23:03.424890+0000","last_became_active":"2024-04-25T01:22:57.850867+0000","last_became_peered":"2024-04-25T01:22:57.850867+0000","last_unstale":"2024-04-25T01:23:03.424890+0000","last_undegraded":"2024-04-25T01:23:03.424890+0000","last_fullsized":"2024-04-25T01:23:03.424890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:15.998433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879340+0000","last_change":"2024-04-25T01:23:01.864658+0000","last_active":"2024-04-25T01:23:02.879340+0000","last_peered":"2024-04-25T01:23:02.879340+0000","last_clean":"2024-04-25T01:23:02.879340+0000","last_became_active":"2024-04-25T01:23:01.864231+0000","last_became_peered":"2024-04-25T01:23:01.864231+0000","last_unstale":"2024-04-25T01:23:02.879340+0000","last_undegraded":"2024-04-25T01:23:02.879340+0000","last_fullsized":"2024-04-25T01:23:02.879340+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:13.925715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424466+0000","last_change":"2024-04-25T01:22:56.834741+0000","last_active":"2024-04-25T01:23:03.424466+0000","last_peered":"2024-04-25T01:23:03.424466+0000","last_clean":"2024-04-25T01:23:03.424466+0000","last_became_active":"2024-04-25T01:22:56.834296+0000","last_became_peered":"2024-04-25T01:22:56.834296+0000","last_unstale":"2024-04-25T01:23:03.424466+0000","last_undegraded":"2024-04-25T01:23:03.424466+0000","last_fullsized":"2024-04-25T01:23:03.424466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:38.940603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423840+0000","last_change":"2024-04-25T01:22:57.854646+0000","last_active":"2024-04-25T01:23:03.423840+0000","last_peered":"2024-04-25T01:23:03.423840+0000","last_clean":"2024-04-25T01:23:03.423840+0000","last_became_active":"2024-04-25T01:22:57.853839+0000","last_became_peered":"2024-04-25T01:22:57.853839+0000","last_unstale":"2024-04-25T01:23:03.423840+0000","last_undegraded":"2024-04-25T01:23:03.423840+0000","last_fullsized":"2024-04-25T01:23:03.423840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:36.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879147+0000","last_change":"2024-04-25T01:23:01.875478+0000","last_active":"2024-04-25T01:23:02.879147+0000","last_peered":"2024-04-25T01:23:02.879147+0000","last_clean":"2024-04-25T01:23:02.879147+0000","last_became_active":"2024-04-25T01:23:01.874643+0000","last_became_peered":"2024-04-25T01:23:01.874643+0000","last_unstale":"2024-04-25T01:23:02.879147+0000","last_undegraded":"2024-04-25T01:23:02.879147+0000","last_fullsized":"2024-04-25T01:23:02.879147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.953507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.942174+0000","last_change":"2024-04-25T01:22:56.843118+0000","last_active":"2024-04-25T01:23:22.942174+0000","last_peered":"2024-04-25T01:23:22.942174+0000","last_clean":"2024-04-25T01:23:22.942174+0000","last_became_active":"2024-04-25T01:22:56.842798+0000","last_became_peered":"2024-04-25T01:22:56.842798+0000","last_unstale":"2024-04-25T01:23:22.942174+0000","last_undegraded":"2024-04-25T01:23:22.942174+0000","last_fullsized":"2024-04-25T01:23:22.942174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:00.054913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424596+0000","last_change":"2024-04-25T01:22:57.850265+0000","last_active":"2024-04-25T01:23:03.424596+0000","last_peered":"2024-04-25T01:23:03.424596+0000","last_clean":"2024-04-25T01:23:03.424596+0000","last_became_active":"2024-04-25T01:22:57.849480+0000","last_became_peered":"2024-04-25T01:22:57.849480+0000","last_unstale":"2024-04-25T01:23:03.424596+0000","last_undegraded":"2024-04-25T01:23:03.424596+0000","last_fullsized":"2024-04-25T01:23:03.424596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:48.577252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423231+0000","last_change":"2024-04-25T01:23:01.865369+0000","last_active":"2024-04-25T01:23:03.423231+0000","last_peered":"2024-04-25T01:23:03.423231+0000","last_clean":"2024-04-25T01:23:03.423231+0000","last_became_active":"2024-04-25T01:23:01.864872+0000","last_became_peered":"2024-04-25T01:23:01.864872+0000","last_unstale":"2024-04-25T01:23:03.423231+0000","last_undegraded":"2024-04-25T01:23:03.423231+0000","last_fullsized":"2024-04-25T01:23:03.423231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:56.266514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.944164+0000","last_change":"2024-04-25T01:22:56.842576+0000","last_active":"2024-04-25T01:23:22.944164+0000","last_peered":"2024-04-25T01:23:22.944164+0000","last_clean":"2024-04-25T01:23:22.944164+0000","last_became_active":"2024-04-25T01:22:56.841081+0000","last_became_peered":"2024-04-25T01:22:56.841081+0000","last_unstale":"2024-04-25T01:23:22.944164+0000","last_undegraded":"2024-04-25T01:23:22.944164+0000","last_fullsized":"2024-04-25T01:23:22.944164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:08.174426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422101+0000","last_change":"2024-04-25T01:22:57.842812+0000","last_active":"2024-04-25T01:23:03.422101+0000","last_peered":"2024-04-25T01:23:03.422101+0000","last_clean":"2024-04-25T01:23:03.422101+0000","last_became_active":"2024-04-25T01:22:57.842462+0000","last_became_peered":"2024-04-25T01:22:57.842462+0000","last_unstale":"2024-04-25T01:23:03.422101+0000","last_undegraded":"2024-04-25T01:23:03.422101+0000","last_fullsized":"2024-04-25T01:23:03.422101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:14:13.444850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980843+0000","last_change":"2024-04-25T01:23:01.870401+0000","last_active":"2024-04-25T01:23:02.980843+0000","last_peered":"2024-04-25T01:23:02.980843+0000","last_clean":"2024-04-25T01:23:02.980843+0000","last_became_active":"2024-04-25T01:23:01.869932+0000","last_became_peered":"2024-04-25T01:23:01.869932+0000","last_unstale":"2024-04-25T01:23:02.980843+0000","last_undegraded":"2024-04-25T01:23:02.980843+0000","last_fullsized":"2024-04-25T01:23:02.980843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:13.323209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424141+0000","last_change":"2024-04-25T01:22:56.835946+0000","last_active":"2024-04-25T01:23:03.424141+0000","last_peered":"2024-04-25T01:23:03.424141+0000","last_clean":"2024-04-25T01:23:03.424141+0000","last_became_active":"2024-04-25T01:22:56.835630+0000","last_became_peered":"2024-04-25T01:22:56.835630+0000","last_unstale":"2024-04-25T01:23:03.424141+0000","last_undegraded":"2024-04-25T01:23:03.424141+0000","last_fullsized":"2024-04-25T01:23:03.424141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:02.128275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878327+0000","last_change":"2024-04-25T01:22:57.844462+0000","last_active":"2024-04-25T01:23:02.878327+0000","last_peered":"2024-04-25T01:23:02.878327+0000","last_clean":"2024-04-25T01:23:02.878327+0000","last_became_active":"2024-04-25T01:22:57.843426+0000","last_became_peered":"2024-04-25T01:22:57.843426+0000","last_unstale":"2024-04-25T01:23:02.878327+0000","last_undegraded":"2024-04-25T01:23:02.878327+0000","last_fullsized":"2024-04-25T01:23:02.878327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:42.896622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422461+0000","last_change":"2024-04-25T01:23:01.871387+0000","last_active":"2024-04-25T01:23:03.422461+0000","last_peered":"2024-04-25T01:23:03.422461+0000","last_clean":"2024-04-25T01:23:03.422461+0000","last_became_active":"2024-04-25T01:23:01.870901+0000","last_became_peered":"2024-04-25T01:23:01.870901+0000","last_unstale":"2024-04-25T01:23:03.422461+0000","last_undegraded":"2024-04-25T01:23:03.422461+0000","last_fullsized":"2024-04-25T01:23:03.422461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:49.582515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.936721+0000","last_change":"2024-04-25T01:22:56.834440+0000","last_active":"2024-04-25T01:23:21.936721+0000","last_peered":"2024-04-25T01:23:21.936721+0000","last_clean":"2024-04-25T01:23:21.936721+0000","last_became_active":"2024-04-25T01:22:56.834116+0000","last_became_peered":"2024-04-25T01:22:56.834116+0000","last_unstale":"2024-04-25T01:23:21.936721+0000","last_undegraded":"2024-04-25T01:23:21.936721+0000","last_fullsized":"2024-04-25T01:23:21.936721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:01.176601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423560+0000","last_change":"2024-04-25T01:22:57.853560+0000","last_active":"2024-04-25T01:23:03.423560+0000","last_peered":"2024-04-25T01:23:03.423560+0000","last_clean":"2024-04-25T01:23:03.423560+0000","last_became_active":"2024-04-25T01:22:57.853234+0000","last_became_peered":"2024-04-25T01:22:57.853234+0000","last_unstale":"2024-04-25T01:23:03.423560+0000","last_undegraded":"2024-04-25T01:23:03.423560+0000","last_fullsized":"2024-04-25T01:23:03.423560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:51.139371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878291+0000","last_change":"2024-04-25T01:23:01.873856+0000","last_active":"2024-04-25T01:23:02.878291+0000","last_peered":"2024-04-25T01:23:02.878291+0000","last_clean":"2024-04-25T01:23:02.878291+0000","last_became_active":"2024-04-25T01:23:01.873413+0000","last_became_peered":"2024-04-25T01:23:01.873413+0000","last_unstale":"2024-04-25T01:23:02.878291+0000","last_undegraded":"2024-04-25T01:23:02.878291+0000","last_fullsized":"2024-04-25T01:23:02.878291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:11.802479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937963+0000","last_change":"2024-04-25T01:22:56.831080+0000","last_active":"2024-04-25T01:23:21.937963+0000","last_peered":"2024-04-25T01:23:21.937963+0000","last_clean":"2024-04-25T01:23:21.937963+0000","last_became_active":"2024-04-25T01:22:56.830735+0000","last_became_peered":"2024-04-25T01:22:56.830735+0000","last_unstale":"2024-04-25T01:23:21.937963+0000","last_undegraded":"2024-04-25T01:23:21.937963+0000","last_fullsized":"2024-04-25T01:23:21.937963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:38.418439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980670+0000","last_change":"2024-04-25T01:22:57.853838+0000","last_active":"2024-04-25T01:23:02.980670+0000","last_peered":"2024-04-25T01:23:02.980670+0000","last_clean":"2024-04-25T01:23:02.980670+0000","last_became_active":"2024-04-25T01:22:57.853023+0000","last_became_peered":"2024-04-25T01:22:57.853023+0000","last_unstale":"2024-04-25T01:23:02.980670+0000","last_undegraded":"2024-04-25T01:23:02.980670+0000","last_fullsized":"2024-04-25T01:23:02.980670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:18.515831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424513+0000","last_change":"2024-04-25T01:23:01.865446+0000","last_active":"2024-04-25T01:23:03.424513+0000","last_peered":"2024-04-25T01:23:03.424513+0000","last_clean":"2024-04-25T01:23:03.424513+0000","last_became_active":"2024-04-25T01:23:01.864937+0000","last_became_peered":"2024-04-25T01:23:01.864937+0000","last_unstale":"2024-04-25T01:23:03.424513+0000","last_undegraded":"2024-04-25T01:23:03.424513+0000","last_fullsized":"2024-04-25T01:23:03.424513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:18.041430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424731+0000","last_change":"2024-04-25T01:22:56.836055+0000","last_active":"2024-04-25T01:23:03.424731+0000","last_peered":"2024-04-25T01:23:03.424731+0000","last_clean":"2024-04-25T01:23:03.424731+0000","last_became_active":"2024-04-25T01:22:56.835760+0000","last_became_peered":"2024-04-25T01:22:56.835760+0000","last_unstale":"2024-04-25T01:23:03.424731+0000","last_undegraded":"2024-04-25T01:23:03.424731+0000","last_fullsized":"2024-04-25T01:23:03.424731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:59.249996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422960+0000","last_change":"2024-04-25T01:22:57.852444+0000","last_active":"2024-04-25T01:23:03.422960+0000","last_peered":"2024-04-25T01:23:03.422960+0000","last_clean":"2024-04-25T01:23:03.422960+0000","last_became_active":"2024-04-25T01:22:57.852127+0000","last_became_peered":"2024-04-25T01:22:57.852127+0000","last_unstale":"2024-04-25T01:23:03.422960+0000","last_undegraded":"2024-04-25T01:23:03.422960+0000","last_fullsized":"2024-04-25T01:23:03.422960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:11.718512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878328+0000","last_change":"2024-04-25T01:23:01.868011+0000","last_active":"2024-04-25T01:23:02.878328+0000","last_peered":"2024-04-25T01:23:02.878328+0000","last_clean":"2024-04-25T01:23:02.878328+0000","last_became_active":"2024-04-25T01:23:01.867198+0000","last_became_peered":"2024-04-25T01:23:01.867198+0000","last_unstale":"2024-04-25T01:23:02.878328+0000","last_undegraded":"2024-04-25T01:23:02.878328+0000","last_fullsized":"2024-04-25T01:23:02.878328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:24.847131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.935595+0000","last_change":"2024-04-25T01:22:56.834125+0000","last_active":"2024-04-25T01:23:21.935595+0000","last_peered":"2024-04-25T01:23:21.935595+0000","last_clean":"2024-04-25T01:23:21.935595+0000","last_became_active":"2024-04-25T01:22:56.833817+0000","last_became_peered":"2024-04-25T01:22:56.833817+0000","last_unstale":"2024-04-25T01:23:21.935595+0000","last_undegraded":"2024-04-25T01:23:21.935595+0000","last_fullsized":"2024-04-25T01:23:21.935595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:58.451250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424493+0000","last_change":"2024-04-25T01:22:57.854554+0000","last_active":"2024-04-25T01:23:03.424493+0000","last_peered":"2024-04-25T01:23:03.424493+0000","last_clean":"2024-04-25T01:23:03.424493+0000","last_became_active":"2024-04-25T01:22:57.853328+0000","last_became_peered":"2024-04-25T01:22:57.853328+0000","last_unstale":"2024-04-25T01:23:03.424493+0000","last_undegraded":"2024-04-25T01:23:03.424493+0000","last_fullsized":"2024-04-25T01:23:03.424493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:21.774148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.425123+0000","last_change":"2024-04-25T01:23:01.870571+0000","last_active":"2024-04-25T01:23:03.425123+0000","last_peered":"2024-04-25T01:23:03.425123+0000","last_clean":"2024-04-25T01:23:03.425123+0000","last_became_active":"2024-04-25T01:23:01.869982+0000","last_became_peered":"2024-04-25T01:23:01.869982+0000","last_unstale":"2024-04-25T01:23:03.425123+0000","last_undegraded":"2024-04-25T01:23:03.425123+0000","last_fullsized":"2024-04-25T01:23:03.425123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:17.294254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878967+0000","last_change":"2024-04-25T01:22:56.835134+0000","last_active":"2024-04-25T01:23:02.878967+0000","last_peered":"2024-04-25T01:23:02.878967+0000","last_clean":"2024-04-25T01:23:02.878967+0000","last_became_active":"2024-04-25T01:22:56.834207+0000","last_became_peered":"2024-04-25T01:22:56.834207+0000","last_unstale":"2024-04-25T01:23:02.878967+0000","last_undegraded":"2024-04-25T01:23:02.878967+0000","last_fullsized":"2024-04-25T01:23:02.878967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:07.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827025+0000","last_change":"2024-04-25T01:22:57.847693+0000","last_active":"2024-04-25T01:23:03.827025+0000","last_peered":"2024-04-25T01:23:03.827025+0000","last_clean":"2024-04-25T01:23:03.827025+0000","last_became_active":"2024-04-25T01:22:57.847211+0000","last_became_peered":"2024-04-25T01:22:57.847211+0000","last_unstale":"2024-04-25T01:23:03.827025+0000","last_undegraded":"2024-04-25T01:23:03.827025+0000","last_fullsized":"2024-04-25T01:23:03.827025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:48.295608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423376+0000","last_change":"2024-04-25T01:23:01.870340+0000","last_active":"2024-04-25T01:23:03.423376+0000","last_peered":"2024-04-25T01:23:03.423376+0000","last_clean":"2024-04-25T01:23:03.423376+0000","last_became_active":"2024-04-25T01:23:01.869662+0000","last_became_peered":"2024-04-25T01:23:01.869662+0000","last_unstale":"2024-04-25T01:23:03.423376+0000","last_undegraded":"2024-04-25T01:23:03.423376+0000","last_fullsized":"2024-04-25T01:23:03.423376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:05:57.313991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424441+0000","last_change":"2024-04-25T01:22:56.838146+0000","last_active":"2024-04-25T01:23:03.424441+0000","last_peered":"2024-04-25T01:23:03.424441+0000","last_clean":"2024-04-25T01:23:03.424441+0000","last_became_active":"2024-04-25T01:22:56.837815+0000","last_became_peered":"2024-04-25T01:22:56.837815+0000","last_unstale":"2024-04-25T01:23:03.424441+0000","last_undegraded":"2024-04-25T01:23:03.424441+0000","last_fullsized":"2024-04-25T01:23:03.424441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:14.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879400+0000","last_change":"2024-04-25T01:22:57.850840+0000","last_active":"2024-04-25T01:23:02.879400+0000","last_peered":"2024-04-25T01:23:02.879400+0000","last_clean":"2024-04-25T01:23:02.879400+0000","last_became_active":"2024-04-25T01:22:57.850255+0000","last_became_peered":"2024-04-25T01:22:57.850255+0000","last_unstale":"2024-04-25T01:23:02.879400+0000","last_undegraded":"2024-04-25T01:23:02.879400+0000","last_fullsized":"2024-04-25T01:23:02.879400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:25.887235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878785+0000","last_change":"2024-04-25T01:23:01.869396+0000","last_active":"2024-04-25T01:23:02.878785+0000","last_peered":"2024-04-25T01:23:02.878785+0000","last_clean":"2024-04-25T01:23:02.878785+0000","last_became_active":"2024-04-25T01:23:01.868777+0000","last_became_peered":"2024-04-25T01:23:01.868777+0000","last_unstale":"2024-04-25T01:23:02.878785+0000","last_undegraded":"2024-04-25T01:23:02.878785+0000","last_fullsized":"2024-04-25T01:23:02.878785+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:37.276122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424109+0000","last_change":"2024-04-25T01:22:56.833289+0000","last_active":"2024-04-25T01:23:03.424109+0000","last_peered":"2024-04-25T01:23:03.424109+0000","last_clean":"2024-04-25T01:23:03.424109+0000","last_became_active":"2024-04-25T01:22:56.832353+0000","last_became_peered":"2024-04-25T01:22:56.832353+0000","last_unstale":"2024-04-25T01:23:03.424109+0000","last_undegraded":"2024-04-25T01:23:03.424109+0000","last_fullsized":"2024-04-25T01:23:03.424109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:18.251018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879197+0000","last_change":"2024-04-25T01:22:57.851974+0000","last_active":"2024-04-25T01:23:02.879197+0000","last_peered":"2024-04-25T01:23:02.879197+0000","last_clean":"2024-04-25T01:23:02.879197+0000","last_became_active":"2024-04-25T01:22:57.851573+0000","last_became_peered":"2024-04-25T01:22:57.851573+0000","last_unstale":"2024-04-25T01:23:02.879197+0000","last_undegraded":"2024-04-25T01:23:02.879197+0000","last_fullsized":"2024-04-25T01:23:02.879197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:27.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827355+0000","last_change":"2024-04-25T01:23:01.876980+0000","last_active":"2024-04-25T01:23:03.827355+0000","last_peered":"2024-04-25T01:23:03.827355+0000","last_clean":"2024-04-25T01:23:03.827355+0000","last_became_active":"2024-04-25T01:23:01.876548+0000","last_became_peered":"2024-04-25T01:23:01.876548+0000","last_unstale":"2024-04-25T01:23:03.827355+0000","last_undegraded":"2024-04-25T01:23:03.827355+0000","last_fullsized":"2024-04-25T01:23:03.827355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:13.470736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.944445+0000","last_change":"2024-04-25T01:22:56.839694+0000","last_active":"2024-04-25T01:23:22.944445+0000","last_peered":"2024-04-25T01:23:22.944445+0000","last_clean":"2024-04-25T01:23:22.944445+0000","last_became_active":"2024-04-25T01:22:56.839367+0000","last_became_peered":"2024-04-25T01:22:56.839367+0000","last_unstale":"2024-04-25T01:23:22.944445+0000","last_undegraded":"2024-04-25T01:23:22.944445+0000","last_fullsized":"2024-04-25T01:23:22.944445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:49.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423814+0000","last_change":"2024-04-25T01:22:57.841526+0000","last_active":"2024-04-25T01:23:03.423814+0000","last_peered":"2024-04-25T01:23:03.423814+0000","last_clean":"2024-04-25T01:23:03.423814+0000","last_became_active":"2024-04-25T01:22:57.841192+0000","last_became_peered":"2024-04-25T01:22:57.841192+0000","last_unstale":"2024-04-25T01:23:03.423814+0000","last_undegraded":"2024-04-25T01:23:03.423814+0000","last_fullsized":"2024-04-25T01:23:03.423814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:38.958901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423696+0000","last_change":"2024-04-25T01:23:01.865486+0000","last_active":"2024-04-25T01:23:03.423696+0000","last_peered":"2024-04-25T01:23:03.423696+0000","last_clean":"2024-04-25T01:23:03.423696+0000","last_became_active":"2024-04-25T01:23:01.865068+0000","last_became_peered":"2024-04-25T01:23:01.865068+0000","last_unstale":"2024-04-25T01:23:03.423696+0000","last_undegraded":"2024-04-25T01:23:03.423696+0000","last_fullsized":"2024-04-25T01:23:03.423696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:50.432724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423697+0000","last_change":"2024-04-25T01:22:56.841507+0000","last_active":"2024-04-25T01:23:03.423697+0000","last_peered":"2024-04-25T01:23:03.423697+0000","last_clean":"2024-04-25T01:23:03.423697+0000","last_became_active":"2024-04-25T01:22:56.839970+0000","last_became_peered":"2024-04-25T01:22:56.839970+0000","last_unstale":"2024-04-25T01:23:03.423697+0000","last_undegraded":"2024-04-25T01:23:03.423697+0000","last_fullsized":"2024-04-25T01:23:03.423697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:37.670052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423144+0000","last_change":"2024-04-25T01:22:57.850853+0000","last_active":"2024-04-25T01:23:03.423144+0000","last_peered":"2024-04-25T01:23:03.423144+0000","last_clean":"2024-04-25T01:23:03.423144+0000","last_became_active":"2024-04-25T01:22:57.850498+0000","last_became_peered":"2024-04-25T01:22:57.850498+0000","last_unstale":"2024-04-25T01:23:03.423144+0000","last_undegraded":"2024-04-25T01:23:03.423144+0000","last_fullsized":"2024-04-25T01:23:03.423144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:13.271048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423729+0000","last_change":"2024-04-25T01:23:01.874254+0000","last_active":"2024-04-25T01:23:03.423729+0000","last_peered":"2024-04-25T01:23:03.423729+0000","last_clean":"2024-04-25T01:23:03.423729+0000","last_became_active":"2024-04-25T01:23:01.873711+0000","last_became_peered":"2024-04-25T01:23:01.873711+0000","last_unstale":"2024-04-25T01:23:03.423729+0000","last_undegraded":"2024-04-25T01:23:03.423729+0000","last_fullsized":"2024-04-25T01:23:03.423729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:13.510061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878559+0000","last_change":"2024-04-25T01:22:56.827137+0000","last_active":"2024-04-25T01:23:02.878559+0000","last_peered":"2024-04-25T01:23:02.878559+0000","last_clean":"2024-04-25T01:23:02.878559+0000","last_became_active":"2024-04-25T01:22:56.826769+0000","last_became_peered":"2024-04-25T01:22:56.826769+0000","last_unstale":"2024-04-25T01:23:02.878559+0000","last_undegraded":"2024-04-25T01:23:02.878559+0000","last_fullsized":"2024-04-25T01:23:02.878559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:21.072643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424004+0000","last_change":"2024-04-25T01:22:57.843713+0000","last_active":"2024-04-25T01:23:03.424004+0000","last_peered":"2024-04-25T01:23:03.424004+0000","last_clean":"2024-04-25T01:23:03.424004+0000","last_became_active":"2024-04-25T01:22:57.843384+0000","last_became_peered":"2024-04-25T01:22:57.843384+0000","last_unstale":"2024-04-25T01:23:03.424004+0000","last_undegraded":"2024-04-25T01:23:03.424004+0000","last_fullsized":"2024-04-25T01:23:03.424004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:18.772379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827307+0000","last_change":"2024-04-25T01:23:01.874150+0000","last_active":"2024-04-25T01:23:03.827307+0000","last_peered":"2024-04-25T01:23:03.827307+0000","last_clean":"2024-04-25T01:23:03.827307+0000","last_became_active":"2024-04-25T01:23:01.873718+0000","last_became_peered":"2024-04-25T01:23:01.873718+0000","last_unstale":"2024-04-25T01:23:03.827307+0000","last_undegraded":"2024-04-25T01:23:03.827307+0000","last_fullsized":"2024-04-25T01:23:03.827307+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:26.317703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879243+0000","last_change":"2024-04-25T01:22:56.843605+0000","last_active":"2024-04-25T01:23:02.879243+0000","last_peered":"2024-04-25T01:23:02.879243+0000","last_clean":"2024-04-25T01:23:02.879243+0000","last_became_active":"2024-04-25T01:22:56.843313+0000","last_became_peered":"2024-04-25T01:22:56.843313+0000","last_unstale":"2024-04-25T01:23:02.879243+0000","last_undegraded":"2024-04-25T01:23:02.879243+0000","last_fullsized":"2024-04-25T01:23:02.879243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:41.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423964+0000","last_change":"2024-04-25T01:22:57.854896+0000","last_active":"2024-04-25T01:23:03.423964+0000","last_peered":"2024-04-25T01:23:03.423964+0000","last_clean":"2024-04-25T01:23:03.423964+0000","last_became_active":"2024-04-25T01:22:57.854544+0000","last_became_peered":"2024-04-25T01:22:57.854544+0000","last_unstale":"2024-04-25T01:23:03.423964+0000","last_undegraded":"2024-04-25T01:23:03.423964+0000","last_fullsized":"2024-04-25T01:23:03.423964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:25.317859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827554+0000","last_change":"2024-04-25T01:23:01.872792+0000","last_active":"2024-04-25T01:23:03.827554+0000","last_peered":"2024-04-25T01:23:03.827554+0000","last_clean":"2024-04-25T01:23:03.827554+0000","last_became_active":"2024-04-25T01:23:01.872391+0000","last_became_peered":"2024-04-25T01:23:01.872391+0000","last_unstale":"2024-04-25T01:23:03.827554+0000","last_undegraded":"2024-04-25T01:23:03.827554+0000","last_fullsized":"2024-04-25T01:23:03.827554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:13.088996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424293+0000","last_change":"2024-04-25T01:22:57.843131+0000","last_active":"2024-04-25T01:23:03.424293+0000","last_peered":"2024-04-25T01:23:03.424293+0000","last_clean":"2024-04-25T01:23:03.424293+0000","last_became_active":"2024-04-25T01:22:57.842540+0000","last_became_peered":"2024-04-25T01:22:57.842540+0000","last_unstale":"2024-04-25T01:23:03.424293+0000","last_undegraded":"2024-04-25T01:23:03.424293+0000","last_fullsized":"2024-04-25T01:23:03.424293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:51.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.944540+0000","last_change":"2024-04-25T01:22:56.844610+0000","last_active":"2024-04-25T01:23:22.944540+0000","last_peered":"2024-04-25T01:23:22.944540+0000","last_clean":"2024-04-25T01:23:22.944540+0000","last_became_active":"2024-04-25T01:22:56.844237+0000","last_became_peered":"2024-04-25T01:22:56.844237+0000","last_unstale":"2024-04-25T01:23:22.944540+0000","last_undegraded":"2024-04-25T01:23:22.944540+0000","last_fullsized":"2024-04-25T01:23:22.944540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:09.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879728+0000","last_change":"2024-04-25T01:23:01.865711+0000","last_active":"2024-04-25T01:23:02.879728+0000","last_peered":"2024-04-25T01:23:02.879728+0000","last_clean":"2024-04-25T01:23:02.879728+0000","last_became_active":"2024-04-25T01:23:01.865317+0000","last_became_peered":"2024-04-25T01:23:01.865317+0000","last_unstale":"2024-04-25T01:23:02.879728+0000","last_undegraded":"2024-04-25T01:23:02.879728+0000","last_fullsized":"2024-04-25T01:23:02.879728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:08.995490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422230+0000","last_change":"2024-04-25T01:22:57.851623+0000","last_active":"2024-04-25T01:23:03.422230+0000","last_peered":"2024-04-25T01:23:03.422230+0000","last_clean":"2024-04-25T01:23:03.422230+0000","last_became_active":"2024-04-25T01:22:57.850616+0000","last_became_peered":"2024-04-25T01:22:57.850616+0000","last_unstale":"2024-04-25T01:23:03.422230+0000","last_undegraded":"2024-04-25T01:23:03.422230+0000","last_fullsized":"2024-04-25T01:23:03.422230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:27.447996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937263+0000","last_change":"2024-04-25T01:22:56.832094+0000","last_active":"2024-04-25T01:23:21.937263+0000","last_peered":"2024-04-25T01:23:21.937263+0000","last_clean":"2024-04-25T01:23:21.937263+0000","last_became_active":"2024-04-25T01:22:56.831743+0000","last_became_peered":"2024-04-25T01:22:56.831743+0000","last_unstale":"2024-04-25T01:23:21.937263+0000","last_undegraded":"2024-04-25T01:23:21.937263+0000","last_fullsized":"2024-04-25T01:23:21.937263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:06.116081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878940+0000","last_change":"2024-04-25T01:21:40.899517+0000","last_active":"2024-04-25T01:23:02.878940+0000","last_peered":"2024-04-25T01:23:02.878940+0000","last_clean":"2024-04-25T01:23:02.878940+0000","last_became_active":"2024-04-25T01:21:40.899043+0000","last_became_peered":"2024-04-25T01:21:40.899043+0000","last_unstale":"2024-04-25T01:23:02.878940+0000","last_undegraded":"2024-04-25T01:23:02.878940+0000","last_fullsized":"2024-04-25T01:23:02.878940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_clean_scrub_stamp":"2024-04-25T01:18:12.053452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:41.234852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424672+0000","last_change":"2024-04-25T01:23:01.878334+0000","last_active":"2024-04-25T01:23:03.424672+0000","last_peered":"2024-04-25T01:23:03.424672+0000","last_clean":"2024-04-25T01:23:03.424672+0000","last_became_active":"2024-04-25T01:23:01.877955+0000","last_became_peered":"2024-04-25T01:23:01.877955+0000","last_unstale":"2024-04-25T01:23:03.424672+0000","last_undegraded":"2024-04-25T01:23:03.424672+0000","last_fullsized":"2024-04-25T01:23:03.424672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:33.339430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.826840+0000","last_change":"2024-04-25T01:22:57.843020+0000","last_active":"2024-04-25T01:23:03.826840+0000","last_peered":"2024-04-25T01:23:03.826840+0000","last_clean":"2024-04-25T01:23:03.826840+0000","last_became_active":"2024-04-25T01:22:57.842671+0000","last_became_peered":"2024-04-25T01:22:57.842671+0000","last_unstale":"2024-04-25T01:23:03.826840+0000","last_undegraded":"2024-04-25T01:23:03.826840+0000","last_fullsized":"2024-04-25T01:23:03.826840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:53.831432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.943269+0000","last_change":"2024-04-25T01:22:56.836106+0000","last_active":"2024-04-25T01:23:22.943269+0000","last_peered":"2024-04-25T01:23:22.943269+0000","last_clean":"2024-04-25T01:23:22.943269+0000","last_became_active":"2024-04-25T01:22:56.835814+0000","last_became_peered":"2024-04-25T01:22:56.835814+0000","last_unstale":"2024-04-25T01:23:22.943269+0000","last_undegraded":"2024-04-25T01:23:22.943269+0000","last_fullsized":"2024-04-25T01:23:22.943269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:18.004312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423383+0000","last_change":"2024-04-25T01:23:01.870723+0000","last_active":"2024-04-25T01:23:03.423383+0000","last_peered":"2024-04-25T01:23:03.423383+0000","last_clean":"2024-04-25T01:23:03.423383+0000","last_became_active":"2024-04-25T01:23:01.870205+0000","last_became_peered":"2024-04-25T01:23:01.870205+0000","last_unstale":"2024-04-25T01:23:03.423383+0000","last_undegraded":"2024-04-25T01:23:03.423383+0000","last_fullsized":"2024-04-25T01:23:03.423383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:47.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937371+0000","last_change":"2024-04-25T01:22:56.839180+0000","last_active":"2024-04-25T01:23:21.937371+0000","last_peered":"2024-04-25T01:23:21.937371+0000","last_clean":"2024-04-25T01:23:21.937371+0000","last_became_active":"2024-04-25T01:22:56.838897+0000","last_became_peered":"2024-04-25T01:22:56.838897+0000","last_unstale":"2024-04-25T01:23:21.937371+0000","last_undegraded":"2024-04-25T01:23:21.937371+0000","last_fullsized":"2024-04-25T01:23:21.937371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:27.176333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827461+0000","last_change":"2024-04-25T01:22:57.841727+0000","last_active":"2024-04-25T01:23:03.827461+0000","last_peered":"2024-04-25T01:23:03.827461+0000","last_clean":"2024-04-25T01:23:03.827461+0000","last_became_active":"2024-04-25T01:22:57.841366+0000","last_became_peered":"2024-04-25T01:22:57.841366+0000","last_unstale":"2024-04-25T01:23:03.827461+0000","last_undegraded":"2024-04-25T01:23:03.827461+0000","last_fullsized":"2024-04-25T01:23:03.827461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:20.585916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422169+0000","last_change":"2024-04-25T01:23:01.869814+0000","last_active":"2024-04-25T01:23:03.422169+0000","last_peered":"2024-04-25T01:23:03.422169+0000","last_clean":"2024-04-25T01:23:03.422169+0000","last_became_active":"2024-04-25T01:23:01.869387+0000","last_became_peered":"2024-04-25T01:23:01.869387+0000","last_unstale":"2024-04-25T01:23:03.422169+0000","last_undegraded":"2024-04-25T01:23:03.422169+0000","last_fullsized":"2024-04-25T01:23:03.422169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:18:45.098295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878818+0000","last_change":"2024-04-25T01:22:56.838506+0000","last_active":"2024-04-25T01:23:02.878818+0000","last_peered":"2024-04-25T01:23:02.878818+0000","last_clean":"2024-04-25T01:23:02.878818+0000","last_became_active":"2024-04-25T01:22:56.838185+0000","last_became_peered":"2024-04-25T01:22:56.838185+0000","last_unstale":"2024-04-25T01:23:02.878818+0000","last_undegraded":"2024-04-25T01:23:02.878818+0000","last_fullsized":"2024-04-25T01:23:02.878818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:35.306231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422766+0000","last_change":"2024-04-25T01:22:57.846544+0000","last_active":"2024-04-25T01:23:03.422766+0000","last_peered":"2024-04-25T01:23:03.422766+0000","last_clean":"2024-04-25T01:23:03.422766+0000","last_became_active":"2024-04-25T01:22:57.843507+0000","last_became_peered":"2024-04-25T01:22:57.843507+0000","last_unstale":"2024-04-25T01:23:03.422766+0000","last_undegraded":"2024-04-25T01:23:03.422766+0000","last_fullsized":"2024-04-25T01:23:03.422766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:30.600317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878808+0000","last_change":"2024-04-25T01:23:01.863986+0000","last_active":"2024-04-25T01:23:02.878808+0000","last_peered":"2024-04-25T01:23:02.878808+0000","last_clean":"2024-04-25T01:23:02.878808+0000","last_became_active":"2024-04-25T01:23:01.863397+0000","last_became_peered":"2024-04-25T01:23:01.863397+0000","last_unstale":"2024-04-25T01:23:02.878808+0000","last_undegraded":"2024-04-25T01:23:02.878808+0000","last_fullsized":"2024-04-25T01:23:02.878808+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:45:56.229349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937389+0000","last_change":"2024-04-25T01:22:56.838201+0000","last_active":"2024-04-25T01:23:21.937389+0000","last_peered":"2024-04-25T01:23:21.937389+0000","last_clean":"2024-04-25T01:23:21.937389+0000","last_became_active":"2024-04-25T01:22:56.837890+0000","last_became_peered":"2024-04-25T01:22:56.837890+0000","last_unstale":"2024-04-25T01:23:21.937389+0000","last_undegraded":"2024-04-25T01:23:21.937389+0000","last_fullsized":"2024-04-25T01:23:21.937389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:17.418112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980716+0000","last_change":"2024-04-25T01:22:57.853923+0000","last_active":"2024-04-25T01:23:02.980716+0000","last_peered":"2024-04-25T01:23:02.980716+0000","last_clean":"2024-04-25T01:23:02.980716+0000","last_became_active":"2024-04-25T01:22:57.853566+0000","last_became_peered":"2024-04-25T01:22:57.853566+0000","last_unstale":"2024-04-25T01:23:02.980716+0000","last_undegraded":"2024-04-25T01:23:02.980716+0000","last_fullsized":"2024-04-25T01:23:02.980716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:36.509650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424749+0000","last_change":"2024-04-25T01:23:01.878257+0000","last_active":"2024-04-25T01:23:03.424749+0000","last_peered":"2024-04-25T01:23:03.424749+0000","last_clean":"2024-04-25T01:23:03.424749+0000","last_became_active":"2024-04-25T01:23:01.877849+0000","last_became_peered":"2024-04-25T01:23:01.877849+0000","last_unstale":"2024-04-25T01:23:03.424749+0000","last_undegraded":"2024-04-25T01:23:03.424749+0000","last_fullsized":"2024-04-25T01:23:03.424749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:32.374927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.935753+0000","last_change":"2024-04-25T01:22:56.828651+0000","last_active":"2024-04-25T01:23:21.935753+0000","last_peered":"2024-04-25T01:23:21.935753+0000","last_clean":"2024-04-25T01:23:21.935753+0000","last_became_active":"2024-04-25T01:22:56.828186+0000","last_became_peered":"2024-04-25T01:22:56.828186+0000","last_unstale":"2024-04-25T01:23:21.935753+0000","last_undegraded":"2024-04-25T01:23:21.935753+0000","last_fullsized":"2024-04-25T01:23:21.935753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:56.020822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423147+0000","last_change":"2024-04-25T01:22:57.849768+0000","last_active":"2024-04-25T01:23:03.423147+0000","last_peered":"2024-04-25T01:23:03.423147+0000","last_clean":"2024-04-25T01:23:03.423147+0000","last_became_active":"2024-04-25T01:22:57.849418+0000","last_became_peered":"2024-04-25T01:22:57.849418+0000","last_unstale":"2024-04-25T01:23:03.423147+0000","last_undegraded":"2024-04-25T01:23:03.423147+0000","last_fullsized":"2024-04-25T01:23:03.423147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:41.292915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878951+0000","last_change":"2024-04-25T01:23:01.872827+0000","last_active":"2024-04-25T01:23:02.878951+0000","last_peered":"2024-04-25T01:23:02.878951+0000","last_clean":"2024-04-25T01:23:02.878951+0000","last_became_active":"2024-04-25T01:23:01.872436+0000","last_became_peered":"2024-04-25T01:23:01.872436+0000","last_unstale":"2024-04-25T01:23:02.878951+0000","last_undegraded":"2024-04-25T01:23:02.878951+0000","last_fullsized":"2024-04-25T01:23:02.878951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:19.258894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.935261+0000","last_change":"2024-04-25T01:22:56.844672+0000","last_active":"2024-04-25T01:23:21.935261+0000","last_peered":"2024-04-25T01:23:21.935261+0000","last_clean":"2024-04-25T01:23:21.935261+0000","last_became_active":"2024-04-25T01:22:56.844141+0000","last_became_peered":"2024-04-25T01:22:56.844141+0000","last_unstale":"2024-04-25T01:23:21.935261+0000","last_undegraded":"2024-04-25T01:23:21.935261+0000","last_fullsized":"2024-04-25T01:23:21.935261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:14.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422637+0000","last_change":"2024-04-25T01:22:57.845006+0000","last_active":"2024-04-25T01:23:03.422637+0000","last_peered":"2024-04-25T01:23:03.422637+0000","last_clean":"2024-04-25T01:23:03.422637+0000","last_became_active":"2024-04-25T01:22:57.844672+0000","last_became_peered":"2024-04-25T01:22:57.844672+0000","last_unstale":"2024-04-25T01:23:03.422637+0000","last_undegraded":"2024-04-25T01:23:03.422637+0000","last_fullsized":"2024-04-25T01:23:03.422637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:18.238512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424499+0000","last_change":"2024-04-25T01:23:01.869438+0000","last_active":"2024-04-25T01:23:03.424499+0000","last_peered":"2024-04-25T01:23:03.424499+0000","last_clean":"2024-04-25T01:23:03.424499+0000","last_became_active":"2024-04-25T01:23:01.868985+0000","last_became_peered":"2024-04-25T01:23:01.868985+0000","last_unstale":"2024-04-25T01:23:03.424499+0000","last_undegraded":"2024-04-25T01:23:03.424499+0000","last_fullsized":"2024-04-25T01:23:03.424499+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:35.509098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878763+0000","last_change":"2024-04-25T01:22:56.834475+0000","last_active":"2024-04-25T01:23:02.878763+0000","last_peered":"2024-04-25T01:23:02.878763+0000","last_clean":"2024-04-25T01:23:02.878763+0000","last_became_active":"2024-04-25T01:22:56.834076+0000","last_became_peered":"2024-04-25T01:22:56.834076+0000","last_unstale":"2024-04-25T01:23:02.878763+0000","last_undegraded":"2024-04-25T01:23:02.878763+0000","last_fullsized":"2024-04-25T01:23:02.878763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:55:47.464009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422570+0000","last_change":"2024-04-25T01:22:57.860549+0000","last_active":"2024-04-25T01:23:03.422570+0000","last_peered":"2024-04-25T01:23:03.422570+0000","last_clean":"2024-04-25T01:23:03.422570+0000","last_became_active":"2024-04-25T01:22:57.860226+0000","last_became_peered":"2024-04-25T01:22:57.860226+0000","last_unstale":"2024-04-25T01:23:03.422570+0000","last_undegraded":"2024-04-25T01:23:03.422570+0000","last_fullsized":"2024-04-25T01:23:03.422570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:10.663508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424563+0000","last_change":"2024-04-25T01:23:01.867602+0000","last_active":"2024-04-25T01:23:03.424563+0000","last_peered":"2024-04-25T01:23:03.424563+0000","last_clean":"2024-04-25T01:23:03.424563+0000","last_became_active":"2024-04-25T01:23:01.867150+0000","last_became_peered":"2024-04-25T01:23:01.867150+0000","last_unstale":"2024-04-25T01:23:03.424563+0000","last_undegraded":"2024-04-25T01:23:03.424563+0000","last_fullsized":"2024-04-25T01:23:03.424563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:34.571031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937996+0000","last_change":"2024-04-25T01:22:56.841082+0000","last_active":"2024-04-25T01:23:21.937996+0000","last_peered":"2024-04-25T01:23:21.937996+0000","last_clean":"2024-04-25T01:23:21.937996+0000","last_became_active":"2024-04-25T01:22:56.839856+0000","last_became_peered":"2024-04-25T01:22:56.839856+0000","last_unstale":"2024-04-25T01:23:21.937996+0000","last_undegraded":"2024-04-25T01:23:21.937996+0000","last_fullsized":"2024-04-25T01:23:21.937996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:14.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879299+0000","last_change":"2024-04-25T01:22:57.852242+0000","last_active":"2024-04-25T01:23:02.879299+0000","last_peered":"2024-04-25T01:23:02.879299+0000","last_clean":"2024-04-25T01:23:02.879299+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:23:02.879299+0000","last_undegraded":"2024-04-25T01:23:02.879299+0000","last_fullsized":"2024-04-25T01:23:02.879299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:38:02.810025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422554+0000","last_change":"2024-04-25T01:23:01.871074+0000","last_active":"2024-04-25T01:23:03.422554+0000","last_peered":"2024-04-25T01:23:03.422554+0000","last_clean":"2024-04-25T01:23:03.422554+0000","last_became_active":"2024-04-25T01:23:01.870570+0000","last_became_peered":"2024-04-25T01:23:01.870570+0000","last_unstale":"2024-04-25T01:23:03.422554+0000","last_undegraded":"2024-04-25T01:23:03.422554+0000","last_fullsized":"2024-04-25T01:23:03.422554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:14.476400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879167+0000","last_change":"2024-04-25T01:22:56.828863+0000","last_active":"2024-04-25T01:23:02.879167+0000","last_peered":"2024-04-25T01:23:02.879167+0000","last_clean":"2024-04-25T01:23:02.879167+0000","last_became_active":"2024-04-25T01:22:56.828520+0000","last_became_peered":"2024-04-25T01:22:56.828520+0000","last_unstale":"2024-04-25T01:23:02.879167+0000","last_undegraded":"2024-04-25T01:23:02.879167+0000","last_fullsized":"2024-04-25T01:23:02.879167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:10.357618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879805+0000","last_change":"2024-04-25T01:22:57.843173+0000","last_active":"2024-04-25T01:23:02.879805+0000","last_peered":"2024-04-25T01:23:02.879805+0000","last_clean":"2024-04-25T01:23:02.879805+0000","last_became_active":"2024-04-25T01:22:57.842734+0000","last_became_peered":"2024-04-25T01:22:57.842734+0000","last_unstale":"2024-04-25T01:23:02.879805+0000","last_undegraded":"2024-04-25T01:23:02.879805+0000","last_fullsized":"2024-04-25T01:23:02.879805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:33.197493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980554+0000","last_change":"2024-04-25T01:23:01.864068+0000","last_active":"2024-04-25T01:23:02.980554+0000","last_peered":"2024-04-25T01:23:02.980554+0000","last_clean":"2024-04-25T01:23:02.980554+0000","last_became_active":"2024-04-25T01:23:01.863640+0000","last_became_peered":"2024-04-25T01:23:01.863640+0000","last_unstale":"2024-04-25T01:23:02.980554+0000","last_undegraded":"2024-04-25T01:23:02.980554+0000","last_fullsized":"2024-04-25T01:23:02.980554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:26.925012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.425002+0000","last_change":"2024-04-25T01:22:56.837969+0000","last_active":"2024-04-25T01:23:03.425002+0000","last_peered":"2024-04-25T01:23:03.425002+0000","last_clean":"2024-04-25T01:23:03.425002+0000","last_became_active":"2024-04-25T01:22:56.837641+0000","last_became_peered":"2024-04-25T01:22:56.837641+0000","last_unstale":"2024-04-25T01:23:03.425002+0000","last_undegraded":"2024-04-25T01:23:03.425002+0000","last_fullsized":"2024-04-25T01:23:03.425002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:04.376038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879389+0000","last_change":"2024-04-25T01:22:57.851057+0000","last_active":"2024-04-25T01:23:02.879389+0000","last_peered":"2024-04-25T01:23:02.879389+0000","last_clean":"2024-04-25T01:23:02.879389+0000","last_became_active":"2024-04-25T01:22:57.850278+0000","last_became_peered":"2024-04-25T01:22:57.850278+0000","last_unstale":"2024-04-25T01:23:02.879389+0000","last_undegraded":"2024-04-25T01:23:02.879389+0000","last_fullsized":"2024-04-25T01:23:02.879389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:15.332193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423367+0000","last_change":"2024-04-25T01:23:01.866673+0000","last_active":"2024-04-25T01:23:03.423367+0000","last_peered":"2024-04-25T01:23:03.423367+0000","last_clean":"2024-04-25T01:23:03.423367+0000","last_became_active":"2024-04-25T01:23:01.866210+0000","last_became_peered":"2024-04-25T01:23:01.866210+0000","last_unstale":"2024-04-25T01:23:03.423367+0000","last_undegraded":"2024-04-25T01:23:03.423367+0000","last_fullsized":"2024-04-25T01:23:03.423367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:20.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827349+0000","last_change":"2024-04-25T01:22:56.844590+0000","last_active":"2024-04-25T01:23:03.827349+0000","last_peered":"2024-04-25T01:23:03.827349+0000","last_clean":"2024-04-25T01:23:03.827349+0000","last_became_active":"2024-04-25T01:22:56.844279+0000","last_became_peered":"2024-04-25T01:22:56.844279+0000","last_unstale":"2024-04-25T01:23:03.827349+0000","last_undegraded":"2024-04-25T01:23:03.827349+0000","last_fullsized":"2024-04-25T01:23:03.827349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:47.155832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879713+0000","last_change":"2024-04-25T01:22:57.858850+0000","last_active":"2024-04-25T01:23:02.879713+0000","last_peered":"2024-04-25T01:23:02.879713+0000","last_clean":"2024-04-25T01:23:02.879713+0000","last_became_active":"2024-04-25T01:22:57.858498+0000","last_became_peered":"2024-04-25T01:22:57.858498+0000","last_unstale":"2024-04-25T01:23:02.879713+0000","last_undegraded":"2024-04-25T01:23:02.879713+0000","last_fullsized":"2024-04-25T01:23:02.879713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:04.836062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422711+0000","last_change":"2024-04-25T01:22:57.843795+0000","last_active":"2024-04-25T01:23:03.422711+0000","last_peered":"2024-04-25T01:23:03.422711+0000","last_clean":"2024-04-25T01:23:03.422711+0000","last_became_active":"2024-04-25T01:22:57.843423+0000","last_became_peered":"2024-04-25T01:22:57.843423+0000","last_unstale":"2024-04-25T01:23:03.422711+0000","last_undegraded":"2024-04-25T01:23:03.422711+0000","last_fullsized":"2024-04-25T01:23:03.422711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:35.013817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879190+0000","last_change":"2024-04-25T01:22:57.844608+0000","last_active":"2024-04-25T01:23:02.879190+0000","last_peered":"2024-04-25T01:23:02.879190+0000","last_clean":"2024-04-25T01:23:02.879190+0000","last_became_active":"2024-04-25T01:22:57.844285+0000","last_became_peered":"2024-04-25T01:22:57.844285+0000","last_unstale":"2024-04-25T01:23:02.879190+0000","last_undegraded":"2024-04-25T01:23:02.879190+0000","last_fullsized":"2024-04-25T01:23:02.879190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:36.348133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878716+0000","last_change":"2024-04-25T01:22:57.851138+0000","last_active":"2024-04-25T01:23:02.878716+0000","last_peered":"2024-04-25T01:23:02.878716+0000","last_clean":"2024-04-25T01:23:02.878716+0000","last_became_active":"2024-04-25T01:22:57.849107+0000","last_became_peered":"2024-04-25T01:22:57.849107+0000","last_unstale":"2024-04-25T01:23:02.878716+0000","last_undegraded":"2024-04-25T01:23:02.878716+0000","last_fullsized":"2024-04-25T01:23:02.878716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:47.665356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980588+0000","last_change":"2024-04-25T01:22:57.853390+0000","last_active":"2024-04-25T01:23:02.980588+0000","last_peered":"2024-04-25T01:23:02.980588+0000","last_clean":"2024-04-25T01:23:02.980588+0000","last_became_active":"2024-04-25T01:22:57.853000+0000","last_became_peered":"2024-04-25T01:22:57.853000+0000","last_unstale":"2024-04-25T01:23:02.980588+0000","last_undegraded":"2024-04-25T01:23:02.980588+0000","last_fullsized":"2024-04-25T01:23:02.980588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:40.593133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827432+0000","last_change":"2024-04-25T01:22:57.851715+0000","last_active":"2024-04-25T01:23:03.827432+0000","last_peered":"2024-04-25T01:23:03.827432+0000","last_clean":"2024-04-25T01:23:03.827432+0000","last_became_active":"2024-04-25T01:22:57.847287+0000","last_became_peered":"2024-04-25T01:22:57.847287+0000","last_unstale":"2024-04-25T01:23:03.827432+0000","last_undegraded":"2024-04-25T01:23:03.827432+0000","last_fullsized":"2024-04-25T01:23:03.827432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.749931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423042+0000","last_change":"2024-04-25T01:22:57.846075+0000","last_active":"2024-04-25T01:23:03.423042+0000","last_peered":"2024-04-25T01:23:03.423042+0000","last_clean":"2024-04-25T01:23:03.423042+0000","last_became_active":"2024-04-25T01:22:57.845388+0000","last_became_peered":"2024-04-25T01:22:57.845388+0000","last_unstale":"2024-04-25T01:23:03.423042+0000","last_undegraded":"2024-04-25T01:23:03.423042+0000","last_fullsized":"2024-04-25T01:23:03.423042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:19.859806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879517+0000","last_change":"2024-04-25T01:22:57.851446+0000","last_active":"2024-04-25T01:23:02.879517+0000","last_peered":"2024-04-25T01:23:02.879517+0000","last_clean":"2024-04-25T01:23:02.879517+0000","last_became_active":"2024-04-25T01:22:57.848729+0000","last_became_peered":"2024-04-25T01:22:57.848729+0000","last_unstale":"2024-04-25T01:23:02.879517+0000","last_undegraded":"2024-04-25T01:23:02.879517+0000","last_fullsized":"2024-04-25T01:23:02.879517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:49.721024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422849+0000","last_change":"2024-04-25T01:22:57.847598+0000","last_active":"2024-04-25T01:23:03.422849+0000","last_peered":"2024-04-25T01:23:03.422849+0000","last_clean":"2024-04-25T01:23:03.422849+0000","last_became_active":"2024-04-25T01:22:57.847232+0000","last_became_peered":"2024-04-25T01:22:57.847232+0000","last_unstale":"2024-04-25T01:23:03.422849+0000","last_undegraded":"2024-04-25T01:23:03.422849+0000","last_fullsized":"2024-04-25T01:23:03.422849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.227700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424715+0000","last_change":"2024-04-25T01:22:57.842881+0000","last_active":"2024-04-25T01:23:03.424715+0000","last_peered":"2024-04-25T01:23:03.424715+0000","last_clean":"2024-04-25T01:23:03.424715+0000","last_became_active":"2024-04-25T01:22:57.842108+0000","last_became_peered":"2024-04-25T01:22:57.842108+0000","last_unstale":"2024-04-25T01:23:03.424715+0000","last_undegraded":"2024-04-25T01:23:03.424715+0000","last_fullsized":"2024-04-25T01:23:03.424715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:53.208188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422167+0000","last_change":"2024-04-25T01:22:57.860047+0000","last_active":"2024-04-25T01:23:03.422167+0000","last_peered":"2024-04-25T01:23:03.422167+0000","last_clean":"2024-04-25T01:23:03.422167+0000","last_became_active":"2024-04-25T01:22:57.859702+0000","last_became_peered":"2024-04-25T01:22:57.859702+0000","last_unstale":"2024-04-25T01:23:03.422167+0000","last_undegraded":"2024-04-25T01:23:03.422167+0000","last_fullsized":"2024-04-25T01:23:03.422167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:00.684055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423064+0000","last_change":"2024-04-25T01:22:57.847996+0000","last_active":"2024-04-25T01:23:03.423064+0000","last_peered":"2024-04-25T01:23:03.423064+0000","last_clean":"2024-04-25T01:23:03.423064+0000","last_became_active":"2024-04-25T01:22:57.847561+0000","last_became_peered":"2024-04-25T01:22:57.847561+0000","last_unstale":"2024-04-25T01:23:03.423064+0000","last_undegraded":"2024-04-25T01:23:03.423064+0000","last_fullsized":"2024-04-25T01:23:03.423064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:16.327558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827150+0000","last_change":"2024-04-25T01:22:57.845216+0000","last_active":"2024-04-25T01:23:03.827150+0000","last_peered":"2024-04-25T01:23:03.827150+0000","last_clean":"2024-04-25T01:23:03.827150+0000","last_became_active":"2024-04-25T01:22:57.844883+0000","last_became_peered":"2024-04-25T01:22:57.844883+0000","last_unstale":"2024-04-25T01:23:03.827150+0000","last_undegraded":"2024-04-25T01:23:03.827150+0000","last_fullsized":"2024-04-25T01:23:03.827150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:33.782048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423232+0000","last_change":"2024-04-25T01:22:57.853251+0000","last_active":"2024-04-25T01:23:03.423232+0000","last_peered":"2024-04-25T01:23:03.423232+0000","last_clean":"2024-04-25T01:23:03.423232+0000","last_became_active":"2024-04-25T01:22:57.852878+0000","last_became_peered":"2024-04-25T01:22:57.852878+0000","last_unstale":"2024-04-25T01:23:03.423232+0000","last_undegraded":"2024-04-25T01:23:03.423232+0000","last_fullsized":"2024-04-25T01:23:03.423232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:42.487837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281206,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5044,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":99,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":19000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50457,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50457,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477909,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477409,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50457,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:26.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 -- 172.21.15.27:0/4046874906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5464102250 msgr2=0x7f5464108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 --2- 172.21.15.27:0/4046874906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5464102250 0x7f5464108ae0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f545c009ca0 tx=0x7f545c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 -- 172.21.15.27:0/4046874906 shutdown_connections 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 --2- 172.21.15.27:0/4046874906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5464102250 0x7f5464108ae0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 --2- 172.21.15.27:0/4046874906 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5464101890 0x7f5464101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 --2- 172.21.15.27:0/4046874906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54641006a0 0x7f5464100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 -- 172.21.15.27:0/4046874906 >> 172.21.15.27:0/4046874906 conn(0x7f54640fbc30 msgr2=0x7f54640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 -- 172.21.15.27:0/4046874906 shutdown_connections 2024-04-25T01:23:26.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.897+0000 7f5469b61640 1 -- 172.21.15.27:0/4046874906 wait complete. 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5469b61640 1 Processor -- start 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5469b61640 1 -- start start 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5469b61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54641006a0 0x7f5464197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5469b61640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464101890 0x7f5464197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5469b61640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5464102250 0x7f546419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5469b61640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f546410d400 con 0x7f54641006a0 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5469b61640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f546410d280 con 0x7f5464102250 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5469b61640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f546410d580 con 0x7f5464101890 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f54637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54641006a0 0x7f5464197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f54637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54641006a0 0x7f5464197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38694/0 (socket says 172.21.15.27:38694) 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f54637fe640 1 -- 172.21.15.27:0/303739338 learned_addr learned my addr 172.21.15.27:0/303739338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5462ffd640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464101890 0x7f5464197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.898+0000 7f5463fff640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5464102250 0x7f546419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f54637fe640 1 -- 172.21.15.27:0/303739338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464101890 msgr2=0x7f5464197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f54637fe640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464101890 0x7f5464197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f54637fe640 1 -- 172.21.15.27:0/303739338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5464102250 msgr2=0x7f546419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f54637fe640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5464102250 0x7f546419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f54637fe640 1 -- 172.21.15.27:0/303739338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546419f570 con 0x7f54641006a0 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f5462ffd640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464101890 0x7f5464197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f5463fff640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5464102250 0x7f546419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f54637fe640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54641006a0 0x7f5464197950 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f545400c970 tx=0x7f545400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f5460ff9640 1 -- 172.21.15.27:0/303739338 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5454007c10 con 0x7f54641006a0 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546419f860 con 0x7f54641006a0 2024-04-25T01:23:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f5460ff9640 1 -- 172.21.15.27:0/303739338 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5454007db0 con 0x7f54641006a0 2024-04-25T01:23:26.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f546419fe00 con 0x7f54641006a0 2024-04-25T01:23:26.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.899+0000 7f5460ff9640 1 -- 172.21.15.27:0/303739338 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545400f8c0 con 0x7f54641006a0 2024-04-25T01:23:26.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.900+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54640689b0 con 0x7f54641006a0 2024-04-25T01:23:26.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.901+0000 7f5460ff9640 1 -- 172.21.15.27:0/303739338 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5454018480 con 0x7f54641006a0 2024-04-25T01:23:26.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.902+0000 7f5460ff9640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f544c0a2c20 0x7f544c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.902+0000 7f5462ffd640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f544c0a2c20 0x7f544c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.902+0000 7f5460ff9640 1 -- 172.21.15.27:0/303739338 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f54540cf770 con 0x7f54641006a0 2024-04-25T01:23:26.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.902+0000 7f5462ffd640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f544c0a2c20 0x7f544c0a50e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5464198e60 tx=0x7f5458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:26.906+0000 7f5460ff9640 1 -- 172.21.15.27:0/303739338 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f545409c3c0 con 0x7f54641006a0 2024-04-25T01:23:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.090+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5464061b80 con 0x7f544c0a2c20 2024-04-25T01:23:27.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.100+0000 7f5460ff9640 1 -- 172.21.15.27:0/303739338 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340547 (secure 0 0 0) 0x7f5464061b80 con 0x7f544c0a2c20 2024-04-25T01:23:27.085 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:27.087 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f544c0a2c20 msgr2=0x7f544c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f544c0a2c20 0x7f544c0a50e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5464198e60 tx=0x7f5458005f90 comp rx=0 tx=0).stop 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54641006a0 msgr2=0x7f5464197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54641006a0 0x7f5464197950 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f545400c970 tx=0x7f545400ce30 comp rx=0 tx=0).stop 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 shutdown_connections 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f544c0a2c20 0x7f544c0a50e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5464102250 0x7f546419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464101890 0x7f5464197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 --2- 172.21.15.27:0/303739338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54641006a0 0x7f5464197950 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.103+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 >> 172.21.15.27:0/303739338 conn(0x7f54640fbc30 msgr2=0x7f5464104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:27.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.104+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 shutdown_connections 2024-04-25T01:23:27.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:27.104+0000 7f5469b61640 1 -- 172.21.15.27:0/303739338 wait complete. 2024-04-25T01:23:27.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:26 smithi027 ceph-mon[29197]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:27.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:26 smithi027 ceph-mon[29197]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-04-25T01:23:27.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:26 smithi027 ceph-mon[29197]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:27.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:26 smithi027 ceph-mon[29197]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:23:27.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:26 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1282867695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:27.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:26 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2379213433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:27.096 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-25T01:23:25.379265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595412,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":141,"num_read_kb":124,"num_write":280,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356576,"kb_used_data":11576,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585248,"statfs":{"total":1151940427776,"available":1151575293952,"internally_reserved":0,"allocated":11853824,"data_stored":7515076,"data_compressed":283599,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2634,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":54,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003354"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423666+0000","last_change":"2024-04-25T01:22:57.847700+0000","last_active":"2024-04-25T01:23:03.423666+0000","last_peered":"2024-04-25T01:23:03.423666+0000","last_clean":"2024-04-25T01:23:03.423666+0000","last_became_active":"2024-04-25T01:22:57.846877+0000","last_became_peered":"2024-04-25T01:22:57.846877+0000","last_unstale":"2024-04-25T01:23:03.423666+0000","last_undegraded":"2024-04-25T01:23:03.423666+0000","last_fullsized":"2024-04-25T01:23:03.423666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:05.918261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980817+0000","last_change":"2024-04-25T01:22:57.848396+0000","last_active":"2024-04-25T01:23:02.980817+0000","last_peered":"2024-04-25T01:23:02.980817+0000","last_clean":"2024-04-25T01:23:02.980817+0000","last_became_active":"2024-04-25T01:22:57.847971+0000","last_became_peered":"2024-04-25T01:22:57.847971+0000","last_unstale":"2024-04-25T01:23:02.980817+0000","last_undegraded":"2024-04-25T01:23:02.980817+0000","last_fullsized":"2024-04-25T01:23:02.980817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:13.143828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980666+0000","last_change":"2024-04-25T01:22:57.856908+0000","last_active":"2024-04-25T01:23:02.980666+0000","last_peered":"2024-04-25T01:23:02.980666+0000","last_clean":"2024-04-25T01:23:02.980666+0000","last_became_active":"2024-04-25T01:22:57.856218+0000","last_became_peered":"2024-04-25T01:22:57.856218+0000","last_unstale":"2024-04-25T01:23:02.980666+0000","last_undegraded":"2024-04-25T01:23:02.980666+0000","last_fullsized":"2024-04-25T01:23:02.980666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:51.971222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424836+0000","last_change":"2024-04-25T01:22:57.849022+0000","last_active":"2024-04-25T01:23:03.424836+0000","last_peered":"2024-04-25T01:23:03.424836+0000","last_clean":"2024-04-25T01:23:03.424836+0000","last_became_active":"2024-04-25T01:22:57.848303+0000","last_became_peered":"2024-04-25T01:22:57.848303+0000","last_unstale":"2024-04-25T01:23:03.424836+0000","last_undegraded":"2024-04-25T01:23:03.424836+0000","last_fullsized":"2024-04-25T01:23:03.424836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:52.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423540+0000","last_change":"2024-04-25T01:22:57.849039+0000","last_active":"2024-04-25T01:23:03.423540+0000","last_peered":"2024-04-25T01:23:03.423540+0000","last_clean":"2024-04-25T01:23:03.423540+0000","last_became_active":"2024-04-25T01:22:57.848702+0000","last_became_peered":"2024-04-25T01:22:57.848702+0000","last_unstale":"2024-04-25T01:23:03.423540+0000","last_undegraded":"2024-04-25T01:23:03.423540+0000","last_fullsized":"2024-04-25T01:23:03.423540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:38.724814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423384+0000","last_change":"2024-04-25T01:22:57.855733+0000","last_active":"2024-04-25T01:23:03.423384+0000","last_peered":"2024-04-25T01:23:03.423384+0000","last_clean":"2024-04-25T01:23:03.423384+0000","last_became_active":"2024-04-25T01:22:57.855075+0000","last_became_peered":"2024-04-25T01:22:57.855075+0000","last_unstale":"2024-04-25T01:23:03.423384+0000","last_undegraded":"2024-04-25T01:23:03.423384+0000","last_fullsized":"2024-04-25T01:23:03.423384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:24.563461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879021+0000","last_change":"2024-04-25T01:22:57.849693+0000","last_active":"2024-04-25T01:23:02.879021+0000","last_peered":"2024-04-25T01:23:02.879021+0000","last_clean":"2024-04-25T01:23:02.879021+0000","last_became_active":"2024-04-25T01:22:57.849324+0000","last_became_peered":"2024-04-25T01:22:57.849324+0000","last_unstale":"2024-04-25T01:23:02.879021+0000","last_undegraded":"2024-04-25T01:23:02.879021+0000","last_fullsized":"2024-04-25T01:23:02.879021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:09.936496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422210+0000","last_change":"2024-04-25T01:22:57.847962+0000","last_active":"2024-04-25T01:23:03.422210+0000","last_peered":"2024-04-25T01:23:03.422210+0000","last_clean":"2024-04-25T01:23:03.422210+0000","last_became_active":"2024-04-25T01:22:57.847592+0000","last_became_peered":"2024-04-25T01:22:57.847592+0000","last_unstale":"2024-04-25T01:23:03.422210+0000","last_undegraded":"2024-04-25T01:23:03.422210+0000","last_fullsized":"2024-04-25T01:23:03.422210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:45.594118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422873+0000","last_change":"2024-04-25T01:22:57.860711+0000","last_active":"2024-04-25T01:23:03.422873+0000","last_peered":"2024-04-25T01:23:03.422873+0000","last_clean":"2024-04-25T01:23:03.422873+0000","last_became_active":"2024-04-25T01:22:57.860393+0000","last_became_peered":"2024-04-25T01:22:57.860393+0000","last_unstale":"2024-04-25T01:23:03.422873+0000","last_undegraded":"2024-04-25T01:23:03.422873+0000","last_fullsized":"2024-04-25T01:23:03.422873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:31.068095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424274+0000","last_change":"2024-04-25T01:22:57.853741+0000","last_active":"2024-04-25T01:23:03.424274+0000","last_peered":"2024-04-25T01:23:03.424274+0000","last_clean":"2024-04-25T01:23:03.424274+0000","last_became_active":"2024-04-25T01:22:57.851015+0000","last_became_peered":"2024-04-25T01:22:57.851015+0000","last_unstale":"2024-04-25T01:23:03.424274+0000","last_undegraded":"2024-04-25T01:23:03.424274+0000","last_fullsized":"2024-04-25T01:23:03.424274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:45.831335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879777+0000","last_change":"2024-04-25T01:22:57.853354+0000","last_active":"2024-04-25T01:23:02.879777+0000","last_peered":"2024-04-25T01:23:02.879777+0000","last_clean":"2024-04-25T01:23:02.879777+0000","last_became_active":"2024-04-25T01:22:57.853043+0000","last_became_peered":"2024-04-25T01:22:57.853043+0000","last_unstale":"2024-04-25T01:23:02.879777+0000","last_undegraded":"2024-04-25T01:23:02.879777+0000","last_fullsized":"2024-04-25T01:23:02.879777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:54.378751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424672+0000","last_change":"2024-04-25T01:22:57.855542+0000","last_active":"2024-04-25T01:23:03.424672+0000","last_peered":"2024-04-25T01:23:03.424672+0000","last_clean":"2024-04-25T01:23:03.424672+0000","last_became_active":"2024-04-25T01:22:57.855196+0000","last_became_peered":"2024-04-25T01:22:57.855196+0000","last_unstale":"2024-04-25T01:23:03.424672+0000","last_undegraded":"2024-04-25T01:23:03.424672+0000","last_fullsized":"2024-04-25T01:23:03.424672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:56.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827084+0000","last_change":"2024-04-25T01:22:57.829353+0000","last_active":"2024-04-25T01:23:03.827084+0000","last_peered":"2024-04-25T01:23:03.827084+0000","last_clean":"2024-04-25T01:23:03.827084+0000","last_became_active":"2024-04-25T01:22:57.828937+0000","last_became_peered":"2024-04-25T01:22:57.828937+0000","last_unstale":"2024-04-25T01:23:03.827084+0000","last_undegraded":"2024-04-25T01:23:03.827084+0000","last_fullsized":"2024-04-25T01:23:03.827084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.541556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980409+0000","last_change":"2024-04-25T01:22:57.854635+0000","last_active":"2024-04-25T01:23:02.980409+0000","last_peered":"2024-04-25T01:23:02.980409+0000","last_clean":"2024-04-25T01:23:02.980409+0000","last_became_active":"2024-04-25T01:22:57.854292+0000","last_became_peered":"2024-04-25T01:22:57.854292+0000","last_unstale":"2024-04-25T01:23:02.980409+0000","last_undegraded":"2024-04-25T01:23:02.980409+0000","last_fullsized":"2024-04-25T01:23:02.980409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:48.209059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980313+0000","last_change":"2024-04-25T01:22:57.852210+0000","last_active":"2024-04-25T01:23:02.980313+0000","last_peered":"2024-04-25T01:23:02.980313+0000","last_clean":"2024-04-25T01:23:02.980313+0000","last_became_active":"2024-04-25T01:22:57.851833+0000","last_became_peered":"2024-04-25T01:22:57.851833+0000","last_unstale":"2024-04-25T01:23:02.980313+0000","last_undegraded":"2024-04-25T01:23:02.980313+0000","last_fullsized":"2024-04-25T01:23:02.980313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:14.664785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422420+0000","last_change":"2024-04-25T01:22:57.849357+0000","last_active":"2024-04-25T01:23:03.422420+0000","last_peered":"2024-04-25T01:23:03.422420+0000","last_clean":"2024-04-25T01:23:03.422420+0000","last_became_active":"2024-04-25T01:22:57.849050+0000","last_became_peered":"2024-04-25T01:22:57.849050+0000","last_unstale":"2024-04-25T01:23:03.422420+0000","last_undegraded":"2024-04-25T01:23:03.422420+0000","last_fullsized":"2024-04-25T01:23:03.422420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:37.669424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423551+0000","last_change":"2024-04-25T01:22:57.844790+0000","last_active":"2024-04-25T01:23:03.423551+0000","last_peered":"2024-04-25T01:23:03.423551+0000","last_clean":"2024-04-25T01:23:03.423551+0000","last_became_active":"2024-04-25T01:22:57.844442+0000","last_became_peered":"2024-04-25T01:22:57.844442+0000","last_unstale":"2024-04-25T01:23:03.423551+0000","last_undegraded":"2024-04-25T01:23:03.423551+0000","last_fullsized":"2024-04-25T01:23:03.423551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:33.766034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878818+0000","last_change":"2024-04-25T01:22:57.844430+0000","last_active":"2024-04-25T01:23:02.878818+0000","last_peered":"2024-04-25T01:23:02.878818+0000","last_clean":"2024-04-25T01:23:02.878818+0000","last_became_active":"2024-04-25T01:22:57.844110+0000","last_became_peered":"2024-04-25T01:22:57.844110+0000","last_unstale":"2024-04-25T01:23:02.878818+0000","last_undegraded":"2024-04-25T01:23:02.878818+0000","last_fullsized":"2024-04-25T01:23:02.878818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:50.348249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879371+0000","last_change":"2024-04-25T01:22:57.852705+0000","last_active":"2024-04-25T01:23:02.879371+0000","last_peered":"2024-04-25T01:23:02.879371+0000","last_clean":"2024-04-25T01:23:02.879371+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:23:02.879371+0000","last_undegraded":"2024-04-25T01:23:02.879371+0000","last_fullsized":"2024-04-25T01:23:02.879371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:10.320459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879033+0000","last_change":"2024-04-25T01:23:01.879341+0000","last_active":"2024-04-25T01:23:02.879033+0000","last_peered":"2024-04-25T01:23:02.879033+0000","last_clean":"2024-04-25T01:23:02.879033+0000","last_became_active":"2024-04-25T01:23:01.878881+0000","last_became_peered":"2024-04-25T01:23:01.878881+0000","last_unstale":"2024-04-25T01:23:02.879033+0000","last_undegraded":"2024-04-25T01:23:02.879033+0000","last_fullsized":"2024-04-25T01:23:02.879033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:46.036868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878919+0000","last_change":"2024-04-25T01:22:56.839867+0000","last_active":"2024-04-25T01:23:02.878919+0000","last_peered":"2024-04-25T01:23:02.878919+0000","last_clean":"2024-04-25T01:23:02.878919+0000","last_became_active":"2024-04-25T01:22:56.839521+0000","last_became_peered":"2024-04-25T01:22:56.839521+0000","last_unstale":"2024-04-25T01:23:02.878919+0000","last_undegraded":"2024-04-25T01:23:02.878919+0000","last_fullsized":"2024-04-25T01:23:02.878919+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:00.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422759+0000","last_change":"2024-04-25T01:22:57.855922+0000","last_active":"2024-04-25T01:23:03.422759+0000","last_peered":"2024-04-25T01:23:03.422759+0000","last_clean":"2024-04-25T01:23:03.422759+0000","last_became_active":"2024-04-25T01:22:57.855584+0000","last_became_peered":"2024-04-25T01:22:57.855584+0000","last_unstale":"2024-04-25T01:23:03.422759+0000","last_undegraded":"2024-04-25T01:23:03.422759+0000","last_fullsized":"2024-04-25T01:23:03.422759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:02.791269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423924+0000","last_change":"2024-04-25T01:23:01.872340+0000","last_active":"2024-04-25T01:23:03.423924+0000","last_peered":"2024-04-25T01:23:03.423924+0000","last_clean":"2024-04-25T01:23:03.423924+0000","last_became_active":"2024-04-25T01:23:01.871900+0000","last_became_peered":"2024-04-25T01:23:01.871900+0000","last_unstale":"2024-04-25T01:23:03.423924+0000","last_undegraded":"2024-04-25T01:23:03.423924+0000","last_fullsized":"2024-04-25T01:23:03.423924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:11.918022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422828+0000","last_change":"2024-04-25T01:22:56.843851+0000","last_active":"2024-04-25T01:23:03.422828+0000","last_peered":"2024-04-25T01:23:03.422828+0000","last_clean":"2024-04-25T01:23:03.422828+0000","last_became_active":"2024-04-25T01:22:56.843232+0000","last_became_peered":"2024-04-25T01:22:56.843232+0000","last_unstale":"2024-04-25T01:23:03.422828+0000","last_undegraded":"2024-04-25T01:23:03.422828+0000","last_fullsized":"2024-04-25T01:23:03.422828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:13.758328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980740+0000","last_change":"2024-04-25T01:22:57.857686+0000","last_active":"2024-04-25T01:23:02.980740+0000","last_peered":"2024-04-25T01:23:02.980740+0000","last_clean":"2024-04-25T01:23:02.980740+0000","last_became_active":"2024-04-25T01:22:57.857175+0000","last_became_peered":"2024-04-25T01:22:57.857175+0000","last_unstale":"2024-04-25T01:23:02.980740+0000","last_undegraded":"2024-04-25T01:23:02.980740+0000","last_fullsized":"2024-04-25T01:23:02.980740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:24.777861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424365+0000","last_change":"2024-04-25T01:23:01.867914+0000","last_active":"2024-04-25T01:23:03.424365+0000","last_peered":"2024-04-25T01:23:03.424365+0000","last_clean":"2024-04-25T01:23:03.424365+0000","last_became_active":"2024-04-25T01:23:01.867103+0000","last_became_peered":"2024-04-25T01:23:01.867103+0000","last_unstale":"2024-04-25T01:23:03.424365+0000","last_undegraded":"2024-04-25T01:23:03.424365+0000","last_fullsized":"2024-04-25T01:23:03.424365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:26.425433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.941671+0000","last_change":"2024-04-25T01:22:56.841310+0000","last_active":"2024-04-25T01:23:22.941671+0000","last_peered":"2024-04-25T01:23:22.941671+0000","last_clean":"2024-04-25T01:23:22.941671+0000","last_became_active":"2024-04-25T01:22:56.840882+0000","last_became_peered":"2024-04-25T01:22:56.840882+0000","last_unstale":"2024-04-25T01:23:22.941671+0000","last_undegraded":"2024-04-25T01:23:22.941671+0000","last_fullsized":"2024-04-25T01:23:22.941671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:29.971448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422919+0000","last_change":"2024-04-25T01:22:57.842810+0000","last_active":"2024-04-25T01:23:03.422919+0000","last_peered":"2024-04-25T01:23:03.422919+0000","last_clean":"2024-04-25T01:23:03.422919+0000","last_became_active":"2024-04-25T01:22:57.842457+0000","last_became_peered":"2024-04-25T01:22:57.842457+0000","last_unstale":"2024-04-25T01:23:03.422919+0000","last_undegraded":"2024-04-25T01:23:03.422919+0000","last_fullsized":"2024-04-25T01:23:03.422919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:45.512037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423655+0000","last_change":"2024-04-25T01:23:01.877298+0000","last_active":"2024-04-25T01:23:03.423655+0000","last_peered":"2024-04-25T01:23:03.423655+0000","last_clean":"2024-04-25T01:23:03.423655+0000","last_became_active":"2024-04-25T01:23:01.876874+0000","last_became_peered":"2024-04-25T01:23:01.876874+0000","last_unstale":"2024-04-25T01:23:03.423655+0000","last_undegraded":"2024-04-25T01:23:03.423655+0000","last_fullsized":"2024-04-25T01:23:03.423655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:35:08.221388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424003+0000","last_change":"2024-04-25T01:22:56.838914+0000","last_active":"2024-04-25T01:23:03.424003+0000","last_peered":"2024-04-25T01:23:03.424003+0000","last_clean":"2024-04-25T01:23:03.424003+0000","last_became_active":"2024-04-25T01:22:56.838616+0000","last_became_peered":"2024-04-25T01:22:56.838616+0000","last_unstale":"2024-04-25T01:23:03.424003+0000","last_undegraded":"2024-04-25T01:23:03.424003+0000","last_fullsized":"2024-04-25T01:23:03.424003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:20.761770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424890+0000","last_change":"2024-04-25T01:22:57.851210+0000","last_active":"2024-04-25T01:23:03.424890+0000","last_peered":"2024-04-25T01:23:03.424890+0000","last_clean":"2024-04-25T01:23:03.424890+0000","last_became_active":"2024-04-25T01:22:57.850867+0000","last_became_peered":"2024-04-25T01:22:57.850867+0000","last_unstale":"2024-04-25T01:23:03.424890+0000","last_undegraded":"2024-04-25T01:23:03.424890+0000","last_fullsized":"2024-04-25T01:23:03.424890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:15.998433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879340+0000","last_change":"2024-04-25T01:23:01.864658+0000","last_active":"2024-04-25T01:23:02.879340+0000","last_peered":"2024-04-25T01:23:02.879340+0000","last_clean":"2024-04-25T01:23:02.879340+0000","last_became_active":"2024-04-25T01:23:01.864231+0000","last_became_peered":"2024-04-25T01:23:01.864231+0000","last_unstale":"2024-04-25T01:23:02.879340+0000","last_undegraded":"2024-04-25T01:23:02.879340+0000","last_fullsized":"2024-04-25T01:23:02.879340+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:13.925715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424466+0000","last_change":"2024-04-25T01:22:56.834741+0000","last_active":"2024-04-25T01:23:03.424466+0000","last_peered":"2024-04-25T01:23:03.424466+0000","last_clean":"2024-04-25T01:23:03.424466+0000","last_became_active":"2024-04-25T01:22:56.834296+0000","last_became_peered":"2024-04-25T01:22:56.834296+0000","last_unstale":"2024-04-25T01:23:03.424466+0000","last_undegraded":"2024-04-25T01:23:03.424466+0000","last_fullsized":"2024-04-25T01:23:03.424466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:38.940603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423840+0000","last_change":"2024-04-25T01:22:57.854646+0000","last_active":"2024-04-25T01:23:03.423840+0000","last_peered":"2024-04-25T01:23:03.423840+0000","last_clean":"2024-04-25T01:23:03.423840+0000","last_became_active":"2024-04-25T01:22:57.853839+0000","last_became_peered":"2024-04-25T01:22:57.853839+0000","last_unstale":"2024-04-25T01:23:03.423840+0000","last_undegraded":"2024-04-25T01:23:03.423840+0000","last_fullsized":"2024-04-25T01:23:03.423840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:36.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879147+0000","last_change":"2024-04-25T01:23:01.875478+0000","last_active":"2024-04-25T01:23:02.879147+0000","last_peered":"2024-04-25T01:23:02.879147+0000","last_clean":"2024-04-25T01:23:02.879147+0000","last_became_active":"2024-04-25T01:23:01.874643+0000","last_became_peered":"2024-04-25T01:23:01.874643+0000","last_unstale":"2024-04-25T01:23:02.879147+0000","last_undegraded":"2024-04-25T01:23:02.879147+0000","last_fullsized":"2024-04-25T01:23:02.879147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.953507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.942174+0000","last_change":"2024-04-25T01:22:56.843118+0000","last_active":"2024-04-25T01:23:22.942174+0000","last_peered":"2024-04-25T01:23:22.942174+0000","last_clean":"2024-04-25T01:23:22.942174+0000","last_became_active":"2024-04-25T01:22:56.842798+0000","last_became_peered":"2024-04-25T01:22:56.842798+0000","last_unstale":"2024-04-25T01:23:22.942174+0000","last_undegraded":"2024-04-25T01:23:22.942174+0000","last_fullsized":"2024-04-25T01:23:22.942174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:00.054913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424596+0000","last_change":"2024-04-25T01:22:57.850265+0000","last_active":"2024-04-25T01:23:03.424596+0000","last_peered":"2024-04-25T01:23:03.424596+0000","last_clean":"2024-04-25T01:23:03.424596+0000","last_became_active":"2024-04-25T01:22:57.849480+0000","last_became_peered":"2024-04-25T01:22:57.849480+0000","last_unstale":"2024-04-25T01:23:03.424596+0000","last_undegraded":"2024-04-25T01:23:03.424596+0000","last_fullsized":"2024-04-25T01:23:03.424596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:48.577252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423231+0000","last_change":"2024-04-25T01:23:01.865369+0000","last_active":"2024-04-25T01:23:03.423231+0000","last_peered":"2024-04-25T01:23:03.423231+0000","last_clean":"2024-04-25T01:23:03.423231+0000","last_became_active":"2024-04-25T01:23:01.864872+0000","last_became_peered":"2024-04-25T01:23:01.864872+0000","last_unstale":"2024-04-25T01:23:03.423231+0000","last_undegraded":"2024-04-25T01:23:03.423231+0000","last_fullsized":"2024-04-25T01:23:03.423231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:56.266514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.944164+0000","last_change":"2024-04-25T01:22:56.842576+0000","last_active":"2024-04-25T01:23:22.944164+0000","last_peered":"2024-04-25T01:23:22.944164+0000","last_clean":"2024-04-25T01:23:22.944164+0000","last_became_active":"2024-04-25T01:22:56.841081+0000","last_became_peered":"2024-04-25T01:22:56.841081+0000","last_unstale":"2024-04-25T01:23:22.944164+0000","last_undegraded":"2024-04-25T01:23:22.944164+0000","last_fullsized":"2024-04-25T01:23:22.944164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:08.174426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422101+0000","last_change":"2024-04-25T01:22:57.842812+0000","last_active":"2024-04-25T01:23:03.422101+0000","last_peered":"2024-04-25T01:23:03.422101+0000","last_clean":"2024-04-25T01:23:03.422101+0000","last_became_active":"2024-04-25T01:22:57.842462+0000","last_became_peered":"2024-04-25T01:22:57.842462+0000","last_unstale":"2024-04-25T01:23:03.422101+0000","last_undegraded":"2024-04-25T01:23:03.422101+0000","last_fullsized":"2024-04-25T01:23:03.422101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:14:13.444850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980843+0000","last_change":"2024-04-25T01:23:01.870401+0000","last_active":"2024-04-25T01:23:02.980843+0000","last_peered":"2024-04-25T01:23:02.980843+0000","last_clean":"2024-04-25T01:23:02.980843+0000","last_became_active":"2024-04-25T01:23:01.869932+0000","last_became_peered":"2024-04-25T01:23:01.869932+0000","last_unstale":"2024-04-25T01:23:02.980843+0000","last_undegraded":"2024-04-25T01:23:02.980843+0000","last_fullsized":"2024-04-25T01:23:02.980843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:13.323209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424141+0000","last_change":"2024-04-25T01:22:56.835946+0000","last_active":"2024-04-25T01:23:03.424141+0000","last_peered":"2024-04-25T01:23:03.424141+0000","last_clean":"2024-04-25T01:23:03.424141+0000","last_became_active":"2024-04-25T01:22:56.835630+0000","last_became_peered":"2024-04-25T01:22:56.835630+0000","last_unstale":"2024-04-25T01:23:03.424141+0000","last_undegraded":"2024-04-25T01:23:03.424141+0000","last_fullsized":"2024-04-25T01:23:03.424141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:02.128275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878327+0000","last_change":"2024-04-25T01:22:57.844462+0000","last_active":"2024-04-25T01:23:02.878327+0000","last_peered":"2024-04-25T01:23:02.878327+0000","last_clean":"2024-04-25T01:23:02.878327+0000","last_became_active":"2024-04-25T01:22:57.843426+0000","last_became_peered":"2024-04-25T01:22:57.843426+0000","last_unstale":"2024-04-25T01:23:02.878327+0000","last_undegraded":"2024-04-25T01:23:02.878327+0000","last_fullsized":"2024-04-25T01:23:02.878327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:42.896622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422461+0000","last_change":"2024-04-25T01:23:01.871387+0000","last_active":"2024-04-25T01:23:03.422461+0000","last_peered":"2024-04-25T01:23:03.422461+0000","last_clean":"2024-04-25T01:23:03.422461+0000","last_became_active":"2024-04-25T01:23:01.870901+0000","last_became_peered":"2024-04-25T01:23:01.870901+0000","last_unstale":"2024-04-25T01:23:03.422461+0000","last_undegraded":"2024-04-25T01:23:03.422461+0000","last_fullsized":"2024-04-25T01:23:03.422461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:49.582515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.936721+0000","last_change":"2024-04-25T01:22:56.834440+0000","last_active":"2024-04-25T01:23:21.936721+0000","last_peered":"2024-04-25T01:23:21.936721+0000","last_clean":"2024-04-25T01:23:21.936721+0000","last_became_active":"2024-04-25T01:22:56.834116+0000","last_became_peered":"2024-04-25T01:22:56.834116+0000","last_unstale":"2024-04-25T01:23:21.936721+0000","last_undegraded":"2024-04-25T01:23:21.936721+0000","last_fullsized":"2024-04-25T01:23:21.936721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:01.176601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423560+0000","last_change":"2024-04-25T01:22:57.853560+0000","last_active":"2024-04-25T01:23:03.423560+0000","last_peered":"2024-04-25T01:23:03.423560+0000","last_clean":"2024-04-25T01:23:03.423560+0000","last_became_active":"2024-04-25T01:22:57.853234+0000","last_became_peered":"2024-04-25T01:22:57.853234+0000","last_unstale":"2024-04-25T01:23:03.423560+0000","last_undegraded":"2024-04-25T01:23:03.423560+0000","last_fullsized":"2024-04-25T01:23:03.423560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:51.139371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878291+0000","last_change":"2024-04-25T01:23:01.873856+0000","last_active":"2024-04-25T01:23:02.878291+0000","last_peered":"2024-04-25T01:23:02.878291+0000","last_clean":"2024-04-25T01:23:02.878291+0000","last_became_active":"2024-04-25T01:23:01.873413+0000","last_became_peered":"2024-04-25T01:23:01.873413+0000","last_unstale":"2024-04-25T01:23:02.878291+0000","last_undegraded":"2024-04-25T01:23:02.878291+0000","last_fullsized":"2024-04-25T01:23:02.878291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:11.802479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937963+0000","last_change":"2024-04-25T01:22:56.831080+0000","last_active":"2024-04-25T01:23:21.937963+0000","last_peered":"2024-04-25T01:23:21.937963+0000","last_clean":"2024-04-25T01:23:21.937963+0000","last_became_active":"2024-04-25T01:22:56.830735+0000","last_became_peered":"2024-04-25T01:22:56.830735+0000","last_unstale":"2024-04-25T01:23:21.937963+0000","last_undegraded":"2024-04-25T01:23:21.937963+0000","last_fullsized":"2024-04-25T01:23:21.937963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:38.418439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980670+0000","last_change":"2024-04-25T01:22:57.853838+0000","last_active":"2024-04-25T01:23:02.980670+0000","last_peered":"2024-04-25T01:23:02.980670+0000","last_clean":"2024-04-25T01:23:02.980670+0000","last_became_active":"2024-04-25T01:22:57.853023+0000","last_became_peered":"2024-04-25T01:22:57.853023+0000","last_unstale":"2024-04-25T01:23:02.980670+0000","last_undegraded":"2024-04-25T01:23:02.980670+0000","last_fullsized":"2024-04-25T01:23:02.980670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:18.515831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424513+0000","last_change":"2024-04-25T01:23:01.865446+0000","last_active":"2024-04-25T01:23:03.424513+0000","last_peered":"2024-04-25T01:23:03.424513+0000","last_clean":"2024-04-25T01:23:03.424513+0000","last_became_active":"2024-04-25T01:23:01.864937+0000","last_became_peered":"2024-04-25T01:23:01.864937+0000","last_unstale":"2024-04-25T01:23:03.424513+0000","last_undegraded":"2024-04-25T01:23:03.424513+0000","last_fullsized":"2024-04-25T01:23:03.424513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:18.041430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424731+0000","last_change":"2024-04-25T01:22:56.836055+0000","last_active":"2024-04-25T01:23:03.424731+0000","last_peered":"2024-04-25T01:23:03.424731+0000","last_clean":"2024-04-25T01:23:03.424731+0000","last_became_active":"2024-04-25T01:22:56.835760+0000","last_became_peered":"2024-04-25T01:22:56.835760+0000","last_unstale":"2024-04-25T01:23:03.424731+0000","last_undegraded":"2024-04-25T01:23:03.424731+0000","last_fullsized":"2024-04-25T01:23:03.424731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:59.249996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422960+0000","last_change":"2024-04-25T01:22:57.852444+0000","last_active":"2024-04-25T01:23:03.422960+0000","last_peered":"2024-04-25T01:23:03.422960+0000","last_clean":"2024-04-25T01:23:03.422960+0000","last_became_active":"2024-04-25T01:22:57.852127+0000","last_became_peered":"2024-04-25T01:22:57.852127+0000","last_unstale":"2024-04-25T01:23:03.422960+0000","last_undegraded":"2024-04-25T01:23:03.422960+0000","last_fullsized":"2024-04-25T01:23:03.422960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:11.718512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878328+0000","last_change":"2024-04-25T01:23:01.868011+0000","last_active":"2024-04-25T01:23:02.878328+0000","last_peered":"2024-04-25T01:23:02.878328+0000","last_clean":"2024-04-25T01:23:02.878328+0000","last_became_active":"2024-04-25T01:23:01.867198+0000","last_became_peered":"2024-04-25T01:23:01.867198+0000","last_unstale":"2024-04-25T01:23:02.878328+0000","last_undegraded":"2024-04-25T01:23:02.878328+0000","last_fullsized":"2024-04-25T01:23:02.878328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:24.847131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.935595+0000","last_change":"2024-04-25T01:22:56.834125+0000","last_active":"2024-04-25T01:23:21.935595+0000","last_peered":"2024-04-25T01:23:21.935595+0000","last_clean":"2024-04-25T01:23:21.935595+0000","last_became_active":"2024-04-25T01:22:56.833817+0000","last_became_peered":"2024-04-25T01:22:56.833817+0000","last_unstale":"2024-04-25T01:23:21.935595+0000","last_undegraded":"2024-04-25T01:23:21.935595+0000","last_fullsized":"2024-04-25T01:23:21.935595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:58.451250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424493+0000","last_change":"2024-04-25T01:22:57.854554+0000","last_active":"2024-04-25T01:23:03.424493+0000","last_peered":"2024-04-25T01:23:03.424493+0000","last_clean":"2024-04-25T01:23:03.424493+0000","last_became_active":"2024-04-25T01:22:57.853328+0000","last_became_peered":"2024-04-25T01:22:57.853328+0000","last_unstale":"2024-04-25T01:23:03.424493+0000","last_undegraded":"2024-04-25T01:23:03.424493+0000","last_fullsized":"2024-04-25T01:23:03.424493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:21.774148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.425123+0000","last_change":"2024-04-25T01:23:01.870571+0000","last_active":"2024-04-25T01:23:03.425123+0000","last_peered":"2024-04-25T01:23:03.425123+0000","last_clean":"2024-04-25T01:23:03.425123+0000","last_became_active":"2024-04-25T01:23:01.869982+0000","last_became_peered":"2024-04-25T01:23:01.869982+0000","last_unstale":"2024-04-25T01:23:03.425123+0000","last_undegraded":"2024-04-25T01:23:03.425123+0000","last_fullsized":"2024-04-25T01:23:03.425123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:17.294254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878967+0000","last_change":"2024-04-25T01:22:56.835134+0000","last_active":"2024-04-25T01:23:02.878967+0000","last_peered":"2024-04-25T01:23:02.878967+0000","last_clean":"2024-04-25T01:23:02.878967+0000","last_became_active":"2024-04-25T01:22:56.834207+0000","last_became_peered":"2024-04-25T01:22:56.834207+0000","last_unstale":"2024-04-25T01:23:02.878967+0000","last_undegraded":"2024-04-25T01:23:02.878967+0000","last_fullsized":"2024-04-25T01:23:02.878967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:07.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827025+0000","last_change":"2024-04-25T01:22:57.847693+0000","last_active":"2024-04-25T01:23:03.827025+0000","last_peered":"2024-04-25T01:23:03.827025+0000","last_clean":"2024-04-25T01:23:03.827025+0000","last_became_active":"2024-04-25T01:22:57.847211+0000","last_became_peered":"2024-04-25T01:22:57.847211+0000","last_unstale":"2024-04-25T01:23:03.827025+0000","last_undegraded":"2024-04-25T01:23:03.827025+0000","last_fullsized":"2024-04-25T01:23:03.827025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:48.295608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423376+0000","last_change":"2024-04-25T01:23:01.870340+0000","last_active":"2024-04-25T01:23:03.423376+0000","last_peered":"2024-04-25T01:23:03.423376+0000","last_clean":"2024-04-25T01:23:03.423376+0000","last_became_active":"2024-04-25T01:23:01.869662+0000","last_became_peered":"2024-04-25T01:23:01.869662+0000","last_unstale":"2024-04-25T01:23:03.423376+0000","last_undegraded":"2024-04-25T01:23:03.423376+0000","last_fullsized":"2024-04-25T01:23:03.423376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:05:57.313991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424441+0000","last_change":"2024-04-25T01:22:56.838146+0000","last_active":"2024-04-25T01:23:03.424441+0000","last_peered":"2024-04-25T01:23:03.424441+0000","last_clean":"2024-04-25T01:23:03.424441+0000","last_became_active":"2024-04-25T01:22:56.837815+0000","last_became_peered":"2024-04-25T01:22:56.837815+0000","last_unstale":"2024-04-25T01:23:03.424441+0000","last_undegraded":"2024-04-25T01:23:03.424441+0000","last_fullsized":"2024-04-25T01:23:03.424441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:14.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879400+0000","last_change":"2024-04-25T01:22:57.850840+0000","last_active":"2024-04-25T01:23:02.879400+0000","last_peered":"2024-04-25T01:23:02.879400+0000","last_clean":"2024-04-25T01:23:02.879400+0000","last_became_active":"2024-04-25T01:22:57.850255+0000","last_became_peered":"2024-04-25T01:22:57.850255+0000","last_unstale":"2024-04-25T01:23:02.879400+0000","last_undegraded":"2024-04-25T01:23:02.879400+0000","last_fullsized":"2024-04-25T01:23:02.879400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:25.887235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878785+0000","last_change":"2024-04-25T01:23:01.869396+0000","last_active":"2024-04-25T01:23:02.878785+0000","last_peered":"2024-04-25T01:23:02.878785+0000","last_clean":"2024-04-25T01:23:02.878785+0000","last_became_active":"2024-04-25T01:23:01.868777+0000","last_became_peered":"2024-04-25T01:23:01.868777+0000","last_unstale":"2024-04-25T01:23:02.878785+0000","last_undegraded":"2024-04-25T01:23:02.878785+0000","last_fullsized":"2024-04-25T01:23:02.878785+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:37.276122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424109+0000","last_change":"2024-04-25T01:22:56.833289+0000","last_active":"2024-04-25T01:23:03.424109+0000","last_peered":"2024-04-25T01:23:03.424109+0000","last_clean":"2024-04-25T01:23:03.424109+0000","last_became_active":"2024-04-25T01:22:56.832353+0000","last_became_peered":"2024-04-25T01:22:56.832353+0000","last_unstale":"2024-04-25T01:23:03.424109+0000","last_undegraded":"2024-04-25T01:23:03.424109+0000","last_fullsized":"2024-04-25T01:23:03.424109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:18.251018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879197+0000","last_change":"2024-04-25T01:22:57.851974+0000","last_active":"2024-04-25T01:23:02.879197+0000","last_peered":"2024-04-25T01:23:02.879197+0000","last_clean":"2024-04-25T01:23:02.879197+0000","last_became_active":"2024-04-25T01:22:57.851573+0000","last_became_peered":"2024-04-25T01:22:57.851573+0000","last_unstale":"2024-04-25T01:23:02.879197+0000","last_undegraded":"2024-04-25T01:23:02.879197+0000","last_fullsized":"2024-04-25T01:23:02.879197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:27.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827355+0000","last_change":"2024-04-25T01:23:01.876980+0000","last_active":"2024-04-25T01:23:03.827355+0000","last_peered":"2024-04-25T01:23:03.827355+0000","last_clean":"2024-04-25T01:23:03.827355+0000","last_became_active":"2024-04-25T01:23:01.876548+0000","last_became_peered":"2024-04-25T01:23:01.876548+0000","last_unstale":"2024-04-25T01:23:03.827355+0000","last_undegraded":"2024-04-25T01:23:03.827355+0000","last_fullsized":"2024-04-25T01:23:03.827355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:13.470736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.944445+0000","last_change":"2024-04-25T01:22:56.839694+0000","last_active":"2024-04-25T01:23:22.944445+0000","last_peered":"2024-04-25T01:23:22.944445+0000","last_clean":"2024-04-25T01:23:22.944445+0000","last_became_active":"2024-04-25T01:22:56.839367+0000","last_became_peered":"2024-04-25T01:22:56.839367+0000","last_unstale":"2024-04-25T01:23:22.944445+0000","last_undegraded":"2024-04-25T01:23:22.944445+0000","last_fullsized":"2024-04-25T01:23:22.944445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:49.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423814+0000","last_change":"2024-04-25T01:22:57.841526+0000","last_active":"2024-04-25T01:23:03.423814+0000","last_peered":"2024-04-25T01:23:03.423814+0000","last_clean":"2024-04-25T01:23:03.423814+0000","last_became_active":"2024-04-25T01:22:57.841192+0000","last_became_peered":"2024-04-25T01:22:57.841192+0000","last_unstale":"2024-04-25T01:23:03.423814+0000","last_undegraded":"2024-04-25T01:23:03.423814+0000","last_fullsized":"2024-04-25T01:23:03.423814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:38.958901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423696+0000","last_change":"2024-04-25T01:23:01.865486+0000","last_active":"2024-04-25T01:23:03.423696+0000","last_peered":"2024-04-25T01:23:03.423696+0000","last_clean":"2024-04-25T01:23:03.423696+0000","last_became_active":"2024-04-25T01:23:01.865068+0000","last_became_peered":"2024-04-25T01:23:01.865068+0000","last_unstale":"2024-04-25T01:23:03.423696+0000","last_undegraded":"2024-04-25T01:23:03.423696+0000","last_fullsized":"2024-04-25T01:23:03.423696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:50.432724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423697+0000","last_change":"2024-04-25T01:22:56.841507+0000","last_active":"2024-04-25T01:23:03.423697+0000","last_peered":"2024-04-25T01:23:03.423697+0000","last_clean":"2024-04-25T01:23:03.423697+0000","last_became_active":"2024-04-25T01:22:56.839970+0000","last_became_peered":"2024-04-25T01:22:56.839970+0000","last_unstale":"2024-04-25T01:23:03.423697+0000","last_undegraded":"2024-04-25T01:23:03.423697+0000","last_fullsized":"2024-04-25T01:23:03.423697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:37.670052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423144+0000","last_change":"2024-04-25T01:22:57.850853+0000","last_active":"2024-04-25T01:23:03.423144+0000","last_peered":"2024-04-25T01:23:03.423144+0000","last_clean":"2024-04-25T01:23:03.423144+0000","last_became_active":"2024-04-25T01:22:57.850498+0000","last_became_peered":"2024-04-25T01:22:57.850498+0000","last_unstale":"2024-04-25T01:23:03.423144+0000","last_undegraded":"2024-04-25T01:23:03.423144+0000","last_fullsized":"2024-04-25T01:23:03.423144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:13.271048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423729+0000","last_change":"2024-04-25T01:23:01.874254+0000","last_active":"2024-04-25T01:23:03.423729+0000","last_peered":"2024-04-25T01:23:03.423729+0000","last_clean":"2024-04-25T01:23:03.423729+0000","last_became_active":"2024-04-25T01:23:01.873711+0000","last_became_peered":"2024-04-25T01:23:01.873711+0000","last_unstale":"2024-04-25T01:23:03.423729+0000","last_undegraded":"2024-04-25T01:23:03.423729+0000","last_fullsized":"2024-04-25T01:23:03.423729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:13.510061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878559+0000","last_change":"2024-04-25T01:22:56.827137+0000","last_active":"2024-04-25T01:23:02.878559+0000","last_peered":"2024-04-25T01:23:02.878559+0000","last_clean":"2024-04-25T01:23:02.878559+0000","last_became_active":"2024-04-25T01:22:56.826769+0000","last_became_peered":"2024-04-25T01:22:56.826769+0000","last_unstale":"2024-04-25T01:23:02.878559+0000","last_undegraded":"2024-04-25T01:23:02.878559+0000","last_fullsized":"2024-04-25T01:23:02.878559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:21.072643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424004+0000","last_change":"2024-04-25T01:22:57.843713+0000","last_active":"2024-04-25T01:23:03.424004+0000","last_peered":"2024-04-25T01:23:03.424004+0000","last_clean":"2024-04-25T01:23:03.424004+0000","last_became_active":"2024-04-25T01:22:57.843384+0000","last_became_peered":"2024-04-25T01:22:57.843384+0000","last_unstale":"2024-04-25T01:23:03.424004+0000","last_undegraded":"2024-04-25T01:23:03.424004+0000","last_fullsized":"2024-04-25T01:23:03.424004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:18.772379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827307+0000","last_change":"2024-04-25T01:23:01.874150+0000","last_active":"2024-04-25T01:23:03.827307+0000","last_peered":"2024-04-25T01:23:03.827307+0000","last_clean":"2024-04-25T01:23:03.827307+0000","last_became_active":"2024-04-25T01:23:01.873718+0000","last_became_peered":"2024-04-25T01:23:01.873718+0000","last_unstale":"2024-04-25T01:23:03.827307+0000","last_undegraded":"2024-04-25T01:23:03.827307+0000","last_fullsized":"2024-04-25T01:23:03.827307+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:26.317703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879243+0000","last_change":"2024-04-25T01:22:56.843605+0000","last_active":"2024-04-25T01:23:02.879243+0000","last_peered":"2024-04-25T01:23:02.879243+0000","last_clean":"2024-04-25T01:23:02.879243+0000","last_became_active":"2024-04-25T01:22:56.843313+0000","last_became_peered":"2024-04-25T01:22:56.843313+0000","last_unstale":"2024-04-25T01:23:02.879243+0000","last_undegraded":"2024-04-25T01:23:02.879243+0000","last_fullsized":"2024-04-25T01:23:02.879243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:41.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423964+0000","last_change":"2024-04-25T01:22:57.854896+0000","last_active":"2024-04-25T01:23:03.423964+0000","last_peered":"2024-04-25T01:23:03.423964+0000","last_clean":"2024-04-25T01:23:03.423964+0000","last_became_active":"2024-04-25T01:22:57.854544+0000","last_became_peered":"2024-04-25T01:22:57.854544+0000","last_unstale":"2024-04-25T01:23:03.423964+0000","last_undegraded":"2024-04-25T01:23:03.423964+0000","last_fullsized":"2024-04-25T01:23:03.423964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:25.317859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827554+0000","last_change":"2024-04-25T01:23:01.872792+0000","last_active":"2024-04-25T01:23:03.827554+0000","last_peered":"2024-04-25T01:23:03.827554+0000","last_clean":"2024-04-25T01:23:03.827554+0000","last_became_active":"2024-04-25T01:23:01.872391+0000","last_became_peered":"2024-04-25T01:23:01.872391+0000","last_unstale":"2024-04-25T01:23:03.827554+0000","last_undegraded":"2024-04-25T01:23:03.827554+0000","last_fullsized":"2024-04-25T01:23:03.827554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:13.088996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424293+0000","last_change":"2024-04-25T01:22:57.843131+0000","last_active":"2024-04-25T01:23:03.424293+0000","last_peered":"2024-04-25T01:23:03.424293+0000","last_clean":"2024-04-25T01:23:03.424293+0000","last_became_active":"2024-04-25T01:22:57.842540+0000","last_became_peered":"2024-04-25T01:22:57.842540+0000","last_unstale":"2024-04-25T01:23:03.424293+0000","last_undegraded":"2024-04-25T01:23:03.424293+0000","last_fullsized":"2024-04-25T01:23:03.424293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:51.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.944540+0000","last_change":"2024-04-25T01:22:56.844610+0000","last_active":"2024-04-25T01:23:22.944540+0000","last_peered":"2024-04-25T01:23:22.944540+0000","last_clean":"2024-04-25T01:23:22.944540+0000","last_became_active":"2024-04-25T01:22:56.844237+0000","last_became_peered":"2024-04-25T01:22:56.844237+0000","last_unstale":"2024-04-25T01:23:22.944540+0000","last_undegraded":"2024-04-25T01:23:22.944540+0000","last_fullsized":"2024-04-25T01:23:22.944540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:09.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879728+0000","last_change":"2024-04-25T01:23:01.865711+0000","last_active":"2024-04-25T01:23:02.879728+0000","last_peered":"2024-04-25T01:23:02.879728+0000","last_clean":"2024-04-25T01:23:02.879728+0000","last_became_active":"2024-04-25T01:23:01.865317+0000","last_became_peered":"2024-04-25T01:23:01.865317+0000","last_unstale":"2024-04-25T01:23:02.879728+0000","last_undegraded":"2024-04-25T01:23:02.879728+0000","last_fullsized":"2024-04-25T01:23:02.879728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:08.995490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422230+0000","last_change":"2024-04-25T01:22:57.851623+0000","last_active":"2024-04-25T01:23:03.422230+0000","last_peered":"2024-04-25T01:23:03.422230+0000","last_clean":"2024-04-25T01:23:03.422230+0000","last_became_active":"2024-04-25T01:22:57.850616+0000","last_became_peered":"2024-04-25T01:22:57.850616+0000","last_unstale":"2024-04-25T01:23:03.422230+0000","last_undegraded":"2024-04-25T01:23:03.422230+0000","last_fullsized":"2024-04-25T01:23:03.422230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:27.447996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937263+0000","last_change":"2024-04-25T01:22:56.832094+0000","last_active":"2024-04-25T01:23:21.937263+0000","last_peered":"2024-04-25T01:23:21.937263+0000","last_clean":"2024-04-25T01:23:21.937263+0000","last_became_active":"2024-04-25T01:22:56.831743+0000","last_became_peered":"2024-04-25T01:22:56.831743+0000","last_unstale":"2024-04-25T01:23:21.937263+0000","last_undegraded":"2024-04-25T01:23:21.937263+0000","last_fullsized":"2024-04-25T01:23:21.937263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:06.116081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878940+0000","last_change":"2024-04-25T01:21:40.899517+0000","last_active":"2024-04-25T01:23:02.878940+0000","last_peered":"2024-04-25T01:23:02.878940+0000","last_clean":"2024-04-25T01:23:02.878940+0000","last_became_active":"2024-04-25T01:21:40.899043+0000","last_became_peered":"2024-04-25T01:21:40.899043+0000","last_unstale":"2024-04-25T01:23:02.878940+0000","last_undegraded":"2024-04-25T01:23:02.878940+0000","last_fullsized":"2024-04-25T01:23:02.878940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_clean_scrub_stamp":"2024-04-25T01:18:12.053452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:41.234852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424672+0000","last_change":"2024-04-25T01:23:01.878334+0000","last_active":"2024-04-25T01:23:03.424672+0000","last_peered":"2024-04-25T01:23:03.424672+0000","last_clean":"2024-04-25T01:23:03.424672+0000","last_became_active":"2024-04-25T01:23:01.877955+0000","last_became_peered":"2024-04-25T01:23:01.877955+0000","last_unstale":"2024-04-25T01:23:03.424672+0000","last_undegraded":"2024-04-25T01:23:03.424672+0000","last_fullsized":"2024-04-25T01:23:03.424672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:33.339430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.826840+0000","last_change":"2024-04-25T01:22:57.843020+0000","last_active":"2024-04-25T01:23:03.826840+0000","last_peered":"2024-04-25T01:23:03.826840+0000","last_clean":"2024-04-25T01:23:03.826840+0000","last_became_active":"2024-04-25T01:22:57.842671+0000","last_became_peered":"2024-04-25T01:22:57.842671+0000","last_unstale":"2024-04-25T01:23:03.826840+0000","last_undegraded":"2024-04-25T01:23:03.826840+0000","last_fullsized":"2024-04-25T01:23:03.826840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:53.831432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:22.943269+0000","last_change":"2024-04-25T01:22:56.836106+0000","last_active":"2024-04-25T01:23:22.943269+0000","last_peered":"2024-04-25T01:23:22.943269+0000","last_clean":"2024-04-25T01:23:22.943269+0000","last_became_active":"2024-04-25T01:22:56.835814+0000","last_became_peered":"2024-04-25T01:22:56.835814+0000","last_unstale":"2024-04-25T01:23:22.943269+0000","last_undegraded":"2024-04-25T01:23:22.943269+0000","last_fullsized":"2024-04-25T01:23:22.943269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:18.004312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423383+0000","last_change":"2024-04-25T01:23:01.870723+0000","last_active":"2024-04-25T01:23:03.423383+0000","last_peered":"2024-04-25T01:23:03.423383+0000","last_clean":"2024-04-25T01:23:03.423383+0000","last_became_active":"2024-04-25T01:23:01.870205+0000","last_became_peered":"2024-04-25T01:23:01.870205+0000","last_unstale":"2024-04-25T01:23:03.423383+0000","last_undegraded":"2024-04-25T01:23:03.423383+0000","last_fullsized":"2024-04-25T01:23:03.423383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:47.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937371+0000","last_change":"2024-04-25T01:22:56.839180+0000","last_active":"2024-04-25T01:23:21.937371+0000","last_peered":"2024-04-25T01:23:21.937371+0000","last_clean":"2024-04-25T01:23:21.937371+0000","last_became_active":"2024-04-25T01:22:56.838897+0000","last_became_peered":"2024-04-25T01:22:56.838897+0000","last_unstale":"2024-04-25T01:23:21.937371+0000","last_undegraded":"2024-04-25T01:23:21.937371+0000","last_fullsized":"2024-04-25T01:23:21.937371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:27.176333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827461+0000","last_change":"2024-04-25T01:22:57.841727+0000","last_active":"2024-04-25T01:23:03.827461+0000","last_peered":"2024-04-25T01:23:03.827461+0000","last_clean":"2024-04-25T01:23:03.827461+0000","last_became_active":"2024-04-25T01:22:57.841366+0000","last_became_peered":"2024-04-25T01:22:57.841366+0000","last_unstale":"2024-04-25T01:23:03.827461+0000","last_undegraded":"2024-04-25T01:23:03.827461+0000","last_fullsized":"2024-04-25T01:23:03.827461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:20.585916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422169+0000","last_change":"2024-04-25T01:23:01.869814+0000","last_active":"2024-04-25T01:23:03.422169+0000","last_peered":"2024-04-25T01:23:03.422169+0000","last_clean":"2024-04-25T01:23:03.422169+0000","last_became_active":"2024-04-25T01:23:01.869387+0000","last_became_peered":"2024-04-25T01:23:01.869387+0000","last_unstale":"2024-04-25T01:23:03.422169+0000","last_undegraded":"2024-04-25T01:23:03.422169+0000","last_fullsized":"2024-04-25T01:23:03.422169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:18:45.098295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878818+0000","last_change":"2024-04-25T01:22:56.838506+0000","last_active":"2024-04-25T01:23:02.878818+0000","last_peered":"2024-04-25T01:23:02.878818+0000","last_clean":"2024-04-25T01:23:02.878818+0000","last_became_active":"2024-04-25T01:22:56.838185+0000","last_became_peered":"2024-04-25T01:22:56.838185+0000","last_unstale":"2024-04-25T01:23:02.878818+0000","last_undegraded":"2024-04-25T01:23:02.878818+0000","last_fullsized":"2024-04-25T01:23:02.878818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:35.306231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422766+0000","last_change":"2024-04-25T01:22:57.846544+0000","last_active":"2024-04-25T01:23:03.422766+0000","last_peered":"2024-04-25T01:23:03.422766+0000","last_clean":"2024-04-25T01:23:03.422766+0000","last_became_active":"2024-04-25T01:22:57.843507+0000","last_became_peered":"2024-04-25T01:22:57.843507+0000","last_unstale":"2024-04-25T01:23:03.422766+0000","last_undegraded":"2024-04-25T01:23:03.422766+0000","last_fullsized":"2024-04-25T01:23:03.422766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:30.600317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878808+0000","last_change":"2024-04-25T01:23:01.863986+0000","last_active":"2024-04-25T01:23:02.878808+0000","last_peered":"2024-04-25T01:23:02.878808+0000","last_clean":"2024-04-25T01:23:02.878808+0000","last_became_active":"2024-04-25T01:23:01.863397+0000","last_became_peered":"2024-04-25T01:23:01.863397+0000","last_unstale":"2024-04-25T01:23:02.878808+0000","last_undegraded":"2024-04-25T01:23:02.878808+0000","last_fullsized":"2024-04-25T01:23:02.878808+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:45:56.229349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937389+0000","last_change":"2024-04-25T01:22:56.838201+0000","last_active":"2024-04-25T01:23:21.937389+0000","last_peered":"2024-04-25T01:23:21.937389+0000","last_clean":"2024-04-25T01:23:21.937389+0000","last_became_active":"2024-04-25T01:22:56.837890+0000","last_became_peered":"2024-04-25T01:22:56.837890+0000","last_unstale":"2024-04-25T01:23:21.937389+0000","last_undegraded":"2024-04-25T01:23:21.937389+0000","last_fullsized":"2024-04-25T01:23:21.937389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:17.418112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980716+0000","last_change":"2024-04-25T01:22:57.853923+0000","last_active":"2024-04-25T01:23:02.980716+0000","last_peered":"2024-04-25T01:23:02.980716+0000","last_clean":"2024-04-25T01:23:02.980716+0000","last_became_active":"2024-04-25T01:22:57.853566+0000","last_became_peered":"2024-04-25T01:22:57.853566+0000","last_unstale":"2024-04-25T01:23:02.980716+0000","last_undegraded":"2024-04-25T01:23:02.980716+0000","last_fullsized":"2024-04-25T01:23:02.980716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:36.509650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424749+0000","last_change":"2024-04-25T01:23:01.878257+0000","last_active":"2024-04-25T01:23:03.424749+0000","last_peered":"2024-04-25T01:23:03.424749+0000","last_clean":"2024-04-25T01:23:03.424749+0000","last_became_active":"2024-04-25T01:23:01.877849+0000","last_became_peered":"2024-04-25T01:23:01.877849+0000","last_unstale":"2024-04-25T01:23:03.424749+0000","last_undegraded":"2024-04-25T01:23:03.424749+0000","last_fullsized":"2024-04-25T01:23:03.424749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:32.374927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.935753+0000","last_change":"2024-04-25T01:22:56.828651+0000","last_active":"2024-04-25T01:23:21.935753+0000","last_peered":"2024-04-25T01:23:21.935753+0000","last_clean":"2024-04-25T01:23:21.935753+0000","last_became_active":"2024-04-25T01:22:56.828186+0000","last_became_peered":"2024-04-25T01:22:56.828186+0000","last_unstale":"2024-04-25T01:23:21.935753+0000","last_undegraded":"2024-04-25T01:23:21.935753+0000","last_fullsized":"2024-04-25T01:23:21.935753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:56.020822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423147+0000","last_change":"2024-04-25T01:22:57.849768+0000","last_active":"2024-04-25T01:23:03.423147+0000","last_peered":"2024-04-25T01:23:03.423147+0000","last_clean":"2024-04-25T01:23:03.423147+0000","last_became_active":"2024-04-25T01:22:57.849418+0000","last_became_peered":"2024-04-25T01:22:57.849418+0000","last_unstale":"2024-04-25T01:23:03.423147+0000","last_undegraded":"2024-04-25T01:23:03.423147+0000","last_fullsized":"2024-04-25T01:23:03.423147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:41.292915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878951+0000","last_change":"2024-04-25T01:23:01.872827+0000","last_active":"2024-04-25T01:23:02.878951+0000","last_peered":"2024-04-25T01:23:02.878951+0000","last_clean":"2024-04-25T01:23:02.878951+0000","last_became_active":"2024-04-25T01:23:01.872436+0000","last_became_peered":"2024-04-25T01:23:01.872436+0000","last_unstale":"2024-04-25T01:23:02.878951+0000","last_undegraded":"2024-04-25T01:23:02.878951+0000","last_fullsized":"2024-04-25T01:23:02.878951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:19.258894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.935261+0000","last_change":"2024-04-25T01:22:56.844672+0000","last_active":"2024-04-25T01:23:21.935261+0000","last_peered":"2024-04-25T01:23:21.935261+0000","last_clean":"2024-04-25T01:23:21.935261+0000","last_became_active":"2024-04-25T01:22:56.844141+0000","last_became_peered":"2024-04-25T01:22:56.844141+0000","last_unstale":"2024-04-25T01:23:21.935261+0000","last_undegraded":"2024-04-25T01:23:21.935261+0000","last_fullsized":"2024-04-25T01:23:21.935261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:14.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422637+0000","last_change":"2024-04-25T01:22:57.845006+0000","last_active":"2024-04-25T01:23:03.422637+0000","last_peered":"2024-04-25T01:23:03.422637+0000","last_clean":"2024-04-25T01:23:03.422637+0000","last_became_active":"2024-04-25T01:22:57.844672+0000","last_became_peered":"2024-04-25T01:22:57.844672+0000","last_unstale":"2024-04-25T01:23:03.422637+0000","last_undegraded":"2024-04-25T01:23:03.422637+0000","last_fullsized":"2024-04-25T01:23:03.422637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:18.238512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424499+0000","last_change":"2024-04-25T01:23:01.869438+0000","last_active":"2024-04-25T01:23:03.424499+0000","last_peered":"2024-04-25T01:23:03.424499+0000","last_clean":"2024-04-25T01:23:03.424499+0000","last_became_active":"2024-04-25T01:23:01.868985+0000","last_became_peered":"2024-04-25T01:23:01.868985+0000","last_unstale":"2024-04-25T01:23:03.424499+0000","last_undegraded":"2024-04-25T01:23:03.424499+0000","last_fullsized":"2024-04-25T01:23:03.424499+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:35.509098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878763+0000","last_change":"2024-04-25T01:22:56.834475+0000","last_active":"2024-04-25T01:23:02.878763+0000","last_peered":"2024-04-25T01:23:02.878763+0000","last_clean":"2024-04-25T01:23:02.878763+0000","last_became_active":"2024-04-25T01:22:56.834076+0000","last_became_peered":"2024-04-25T01:22:56.834076+0000","last_unstale":"2024-04-25T01:23:02.878763+0000","last_undegraded":"2024-04-25T01:23:02.878763+0000","last_fullsized":"2024-04-25T01:23:02.878763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:55:47.464009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422570+0000","last_change":"2024-04-25T01:22:57.860549+0000","last_active":"2024-04-25T01:23:03.422570+0000","last_peered":"2024-04-25T01:23:03.422570+0000","last_clean":"2024-04-25T01:23:03.422570+0000","last_became_active":"2024-04-25T01:22:57.860226+0000","last_became_peered":"2024-04-25T01:22:57.860226+0000","last_unstale":"2024-04-25T01:23:03.422570+0000","last_undegraded":"2024-04-25T01:23:03.422570+0000","last_fullsized":"2024-04-25T01:23:03.422570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:10.663508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424563+0000","last_change":"2024-04-25T01:23:01.867602+0000","last_active":"2024-04-25T01:23:03.424563+0000","last_peered":"2024-04-25T01:23:03.424563+0000","last_clean":"2024-04-25T01:23:03.424563+0000","last_became_active":"2024-04-25T01:23:01.867150+0000","last_became_peered":"2024-04-25T01:23:01.867150+0000","last_unstale":"2024-04-25T01:23:03.424563+0000","last_undegraded":"2024-04-25T01:23:03.424563+0000","last_fullsized":"2024-04-25T01:23:03.424563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:34.571031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:21.937996+0000","last_change":"2024-04-25T01:22:56.841082+0000","last_active":"2024-04-25T01:23:21.937996+0000","last_peered":"2024-04-25T01:23:21.937996+0000","last_clean":"2024-04-25T01:23:21.937996+0000","last_became_active":"2024-04-25T01:22:56.839856+0000","last_became_peered":"2024-04-25T01:22:56.839856+0000","last_unstale":"2024-04-25T01:23:21.937996+0000","last_undegraded":"2024-04-25T01:23:21.937996+0000","last_fullsized":"2024-04-25T01:23:21.937996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:14.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879299+0000","last_change":"2024-04-25T01:22:57.852242+0000","last_active":"2024-04-25T01:23:02.879299+0000","last_peered":"2024-04-25T01:23:02.879299+0000","last_clean":"2024-04-25T01:23:02.879299+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:23:02.879299+0000","last_undegraded":"2024-04-25T01:23:02.879299+0000","last_fullsized":"2024-04-25T01:23:02.879299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:38:02.810025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422554+0000","last_change":"2024-04-25T01:23:01.871074+0000","last_active":"2024-04-25T01:23:03.422554+0000","last_peered":"2024-04-25T01:23:03.422554+0000","last_clean":"2024-04-25T01:23:03.422554+0000","last_became_active":"2024-04-25T01:23:01.870570+0000","last_became_peered":"2024-04-25T01:23:01.870570+0000","last_unstale":"2024-04-25T01:23:03.422554+0000","last_undegraded":"2024-04-25T01:23:03.422554+0000","last_fullsized":"2024-04-25T01:23:03.422554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:14.476400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879167+0000","last_change":"2024-04-25T01:22:56.828863+0000","last_active":"2024-04-25T01:23:02.879167+0000","last_peered":"2024-04-25T01:23:02.879167+0000","last_clean":"2024-04-25T01:23:02.879167+0000","last_became_active":"2024-04-25T01:22:56.828520+0000","last_became_peered":"2024-04-25T01:22:56.828520+0000","last_unstale":"2024-04-25T01:23:02.879167+0000","last_undegraded":"2024-04-25T01:23:02.879167+0000","last_fullsized":"2024-04-25T01:23:02.879167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:10.357618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879805+0000","last_change":"2024-04-25T01:22:57.843173+0000","last_active":"2024-04-25T01:23:02.879805+0000","last_peered":"2024-04-25T01:23:02.879805+0000","last_clean":"2024-04-25T01:23:02.879805+0000","last_became_active":"2024-04-25T01:22:57.842734+0000","last_became_peered":"2024-04-25T01:22:57.842734+0000","last_unstale":"2024-04-25T01:23:02.879805+0000","last_undegraded":"2024-04-25T01:23:02.879805+0000","last_fullsized":"2024-04-25T01:23:02.879805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:33.197493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980554+0000","last_change":"2024-04-25T01:23:01.864068+0000","last_active":"2024-04-25T01:23:02.980554+0000","last_peered":"2024-04-25T01:23:02.980554+0000","last_clean":"2024-04-25T01:23:02.980554+0000","last_became_active":"2024-04-25T01:23:01.863640+0000","last_became_peered":"2024-04-25T01:23:01.863640+0000","last_unstale":"2024-04-25T01:23:02.980554+0000","last_undegraded":"2024-04-25T01:23:02.980554+0000","last_fullsized":"2024-04-25T01:23:02.980554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:26.925012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.425002+0000","last_change":"2024-04-25T01:22:56.837969+0000","last_active":"2024-04-25T01:23:03.425002+0000","last_peered":"2024-04-25T01:23:03.425002+0000","last_clean":"2024-04-25T01:23:03.425002+0000","last_became_active":"2024-04-25T01:22:56.837641+0000","last_became_peered":"2024-04-25T01:22:56.837641+0000","last_unstale":"2024-04-25T01:23:03.425002+0000","last_undegraded":"2024-04-25T01:23:03.425002+0000","last_fullsized":"2024-04-25T01:23:03.425002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:04.376038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879389+0000","last_change":"2024-04-25T01:22:57.851057+0000","last_active":"2024-04-25T01:23:02.879389+0000","last_peered":"2024-04-25T01:23:02.879389+0000","last_clean":"2024-04-25T01:23:02.879389+0000","last_became_active":"2024-04-25T01:22:57.850278+0000","last_became_peered":"2024-04-25T01:22:57.850278+0000","last_unstale":"2024-04-25T01:23:02.879389+0000","last_undegraded":"2024-04-25T01:23:02.879389+0000","last_fullsized":"2024-04-25T01:23:02.879389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:15.332193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423367+0000","last_change":"2024-04-25T01:23:01.866673+0000","last_active":"2024-04-25T01:23:03.423367+0000","last_peered":"2024-04-25T01:23:03.423367+0000","last_clean":"2024-04-25T01:23:03.423367+0000","last_became_active":"2024-04-25T01:23:01.866210+0000","last_became_peered":"2024-04-25T01:23:01.866210+0000","last_unstale":"2024-04-25T01:23:03.423367+0000","last_undegraded":"2024-04-25T01:23:03.423367+0000","last_fullsized":"2024-04-25T01:23:03.423367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:20.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827349+0000","last_change":"2024-04-25T01:22:56.844590+0000","last_active":"2024-04-25T01:23:03.827349+0000","last_peered":"2024-04-25T01:23:03.827349+0000","last_clean":"2024-04-25T01:23:03.827349+0000","last_became_active":"2024-04-25T01:22:56.844279+0000","last_became_peered":"2024-04-25T01:22:56.844279+0000","last_unstale":"2024-04-25T01:23:03.827349+0000","last_undegraded":"2024-04-25T01:23:03.827349+0000","last_fullsized":"2024-04-25T01:23:03.827349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:47.155832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879713+0000","last_change":"2024-04-25T01:22:57.858850+0000","last_active":"2024-04-25T01:23:02.879713+0000","last_peered":"2024-04-25T01:23:02.879713+0000","last_clean":"2024-04-25T01:23:02.879713+0000","last_became_active":"2024-04-25T01:22:57.858498+0000","last_became_peered":"2024-04-25T01:22:57.858498+0000","last_unstale":"2024-04-25T01:23:02.879713+0000","last_undegraded":"2024-04-25T01:23:02.879713+0000","last_fullsized":"2024-04-25T01:23:02.879713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:04.836062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422711+0000","last_change":"2024-04-25T01:22:57.843795+0000","last_active":"2024-04-25T01:23:03.422711+0000","last_peered":"2024-04-25T01:23:03.422711+0000","last_clean":"2024-04-25T01:23:03.422711+0000","last_became_active":"2024-04-25T01:22:57.843423+0000","last_became_peered":"2024-04-25T01:22:57.843423+0000","last_unstale":"2024-04-25T01:23:03.422711+0000","last_undegraded":"2024-04-25T01:23:03.422711+0000","last_fullsized":"2024-04-25T01:23:03.422711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:35.013817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879190+0000","last_change":"2024-04-25T01:22:57.844608+0000","last_active":"2024-04-25T01:23:02.879190+0000","last_peered":"2024-04-25T01:23:02.879190+0000","last_clean":"2024-04-25T01:23:02.879190+0000","last_became_active":"2024-04-25T01:22:57.844285+0000","last_became_peered":"2024-04-25T01:22:57.844285+0000","last_unstale":"2024-04-25T01:23:02.879190+0000","last_undegraded":"2024-04-25T01:23:02.879190+0000","last_fullsized":"2024-04-25T01:23:02.879190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:36.348133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.878716+0000","last_change":"2024-04-25T01:22:57.851138+0000","last_active":"2024-04-25T01:23:02.878716+0000","last_peered":"2024-04-25T01:23:02.878716+0000","last_clean":"2024-04-25T01:23:02.878716+0000","last_became_active":"2024-04-25T01:22:57.849107+0000","last_became_peered":"2024-04-25T01:22:57.849107+0000","last_unstale":"2024-04-25T01:23:02.878716+0000","last_undegraded":"2024-04-25T01:23:02.878716+0000","last_fullsized":"2024-04-25T01:23:02.878716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:47.665356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.980588+0000","last_change":"2024-04-25T01:22:57.853390+0000","last_active":"2024-04-25T01:23:02.980588+0000","last_peered":"2024-04-25T01:23:02.980588+0000","last_clean":"2024-04-25T01:23:02.980588+0000","last_became_active":"2024-04-25T01:22:57.853000+0000","last_became_peered":"2024-04-25T01:22:57.853000+0000","last_unstale":"2024-04-25T01:23:02.980588+0000","last_undegraded":"2024-04-25T01:23:02.980588+0000","last_fullsized":"2024-04-25T01:23:02.980588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:40.593133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827432+0000","last_change":"2024-04-25T01:22:57.851715+0000","last_active":"2024-04-25T01:23:03.827432+0000","last_peered":"2024-04-25T01:23:03.827432+0000","last_clean":"2024-04-25T01:23:03.827432+0000","last_became_active":"2024-04-25T01:22:57.847287+0000","last_became_peered":"2024-04-25T01:22:57.847287+0000","last_unstale":"2024-04-25T01:23:03.827432+0000","last_undegraded":"2024-04-25T01:23:03.827432+0000","last_fullsized":"2024-04-25T01:23:03.827432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.749931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423042+0000","last_change":"2024-04-25T01:22:57.846075+0000","last_active":"2024-04-25T01:23:03.423042+0000","last_peered":"2024-04-25T01:23:03.423042+0000","last_clean":"2024-04-25T01:23:03.423042+0000","last_became_active":"2024-04-25T01:22:57.845388+0000","last_became_peered":"2024-04-25T01:22:57.845388+0000","last_unstale":"2024-04-25T01:23:03.423042+0000","last_undegraded":"2024-04-25T01:23:03.423042+0000","last_fullsized":"2024-04-25T01:23:03.423042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:19.859806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:02.879517+0000","last_change":"2024-04-25T01:22:57.851446+0000","last_active":"2024-04-25T01:23:02.879517+0000","last_peered":"2024-04-25T01:23:02.879517+0000","last_clean":"2024-04-25T01:23:02.879517+0000","last_became_active":"2024-04-25T01:22:57.848729+0000","last_became_peered":"2024-04-25T01:22:57.848729+0000","last_unstale":"2024-04-25T01:23:02.879517+0000","last_undegraded":"2024-04-25T01:23:02.879517+0000","last_fullsized":"2024-04-25T01:23:02.879517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:49.721024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422849+0000","last_change":"2024-04-25T01:22:57.847598+0000","last_active":"2024-04-25T01:23:03.422849+0000","last_peered":"2024-04-25T01:23:03.422849+0000","last_clean":"2024-04-25T01:23:03.422849+0000","last_became_active":"2024-04-25T01:22:57.847232+0000","last_became_peered":"2024-04-25T01:22:57.847232+0000","last_unstale":"2024-04-25T01:23:03.422849+0000","last_undegraded":"2024-04-25T01:23:03.422849+0000","last_fullsized":"2024-04-25T01:23:03.422849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.227700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.424715+0000","last_change":"2024-04-25T01:22:57.842881+0000","last_active":"2024-04-25T01:23:03.424715+0000","last_peered":"2024-04-25T01:23:03.424715+0000","last_clean":"2024-04-25T01:23:03.424715+0000","last_became_active":"2024-04-25T01:22:57.842108+0000","last_became_peered":"2024-04-25T01:22:57.842108+0000","last_unstale":"2024-04-25T01:23:03.424715+0000","last_undegraded":"2024-04-25T01:23:03.424715+0000","last_fullsized":"2024-04-25T01:23:03.424715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:53.208188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.422167+0000","last_change":"2024-04-25T01:22:57.860047+0000","last_active":"2024-04-25T01:23:03.422167+0000","last_peered":"2024-04-25T01:23:03.422167+0000","last_clean":"2024-04-25T01:23:03.422167+0000","last_became_active":"2024-04-25T01:22:57.859702+0000","last_became_peered":"2024-04-25T01:22:57.859702+0000","last_unstale":"2024-04-25T01:23:03.422167+0000","last_undegraded":"2024-04-25T01:23:03.422167+0000","last_fullsized":"2024-04-25T01:23:03.422167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:00.684055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423064+0000","last_change":"2024-04-25T01:22:57.847996+0000","last_active":"2024-04-25T01:23:03.423064+0000","last_peered":"2024-04-25T01:23:03.423064+0000","last_clean":"2024-04-25T01:23:03.423064+0000","last_became_active":"2024-04-25T01:22:57.847561+0000","last_became_peered":"2024-04-25T01:22:57.847561+0000","last_unstale":"2024-04-25T01:23:03.423064+0000","last_undegraded":"2024-04-25T01:23:03.423064+0000","last_fullsized":"2024-04-25T01:23:03.423064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:16.327558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.827150+0000","last_change":"2024-04-25T01:22:57.845216+0000","last_active":"2024-04-25T01:23:03.827150+0000","last_peered":"2024-04-25T01:23:03.827150+0000","last_clean":"2024-04-25T01:23:03.827150+0000","last_became_active":"2024-04-25T01:22:57.844883+0000","last_became_peered":"2024-04-25T01:22:57.844883+0000","last_unstale":"2024-04-25T01:23:03.827150+0000","last_undegraded":"2024-04-25T01:23:03.827150+0000","last_fullsized":"2024-04-25T01:23:03.827150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:33.782048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:23:03.423232+0000","last_change":"2024-04-25T01:22:57.853251+0000","last_active":"2024-04-25T01:23:03.423232+0000","last_peered":"2024-04-25T01:23:03.423232+0000","last_clean":"2024-04-25T01:23:03.423232+0000","last_became_active":"2024-04-25T01:22:57.852878+0000","last_became_peered":"2024-04-25T01:22:57.852878+0000","last_unstale":"2024-04-25T01:23:03.423232+0000","last_undegraded":"2024-04-25T01:23:03.423232+0000","last_fullsized":"2024-04-25T01:23:03.423232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:42.487837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281206,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5044,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":99,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":19000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50457,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50457,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477909,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477409,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50457,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:23:27.102 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:23:27.103 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T01:23:27.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T01:23:27.178 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:26 smithi153 ceph-mon[32621]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:27.178 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:26 smithi153 ceph-mon[32621]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-04-25T01:23:27.178 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:26 smithi153 ceph-mon[32621]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:27.178 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:26 smithi153 ceph-mon[32621]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:23:27.178 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:26 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1282867695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:27.178 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:26 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2379213433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:27.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:26 smithi139 ceph-mon[32760]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:27.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:26 smithi139 ceph-mon[32760]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-04-25T01:23:27.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:26 smithi139 ceph-mon[32760]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:27.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:26 smithi139 ceph-mon[32760]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:23:27.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:26 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1282867695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:23:27.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:26 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2379213433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:27.385 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:27.793 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:23:27 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b[63104]: 2024-04-25T01:23:27.539+0000 7f4018d47700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T01:23:28.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:27 smithi027 ceph-mon[29197]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:23:28.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:27 smithi027 ceph-mon[29197]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:23:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:27 smithi153 ceph-mon[32621]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:23:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:27 smithi153 ceph-mon[32621]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:23:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:27 smithi139 ceph-mon[32760]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:23:28.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:27 smithi139 ceph-mon[32760]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:23:29.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:28 smithi027 ceph-mon[29197]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-25T01:23:29.194 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:28 smithi139 ceph-mon[32760]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-25T01:23:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:28 smithi153 ceph-mon[32621]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 -- 172.21.15.27:0/2047665379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450103570 msgr2=0x7f64501039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 --2- 172.21.15.27:0/2047665379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450103570 0x7f64501039d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6440009990 tx=0x7f6440009ca0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 -- 172.21.15.27:0/2047665379 shutdown_connections 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 --2- 172.21.15.27:0/2047665379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6450103f10 0x7f645010a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 --2- 172.21.15.27:0/2047665379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6450100590 0x7f64501009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 --2- 172.21.15.27:0/2047665379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450103570 0x7f64501039d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 -- 172.21.15.27:0/2047665379 >> 172.21.15.27:0/2047665379 conn(0x7f64500fbf10 msgr2=0x7f64500fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 -- 172.21.15.27:0/2047665379 shutdown_connections 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.307+0000 7f6455989700 1 -- 172.21.15.27:0/2047665379 wait complete. 2024-04-25T01:23:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.308+0000 7f6455989700 1 Processor -- start 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.308+0000 7f6455989700 1 -- start start 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.308+0000 7f6455989700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450100590 0x7f6450198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.308+0000 7f6455989700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6450103570 0x7f6450198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.308+0000 7f6455989700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6450103f10 0x7f645019f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f6455989700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f645010de90 con 0x7f6450100590 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f6455989700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f645010dd10 con 0x7f6450103f10 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f6455989700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f645010e010 con 0x7f6450103570 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450100590 0x7f6450198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450100590 0x7f6450198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44968/0 (socket says 172.21.15.27:44968) 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 -- 172.21.15.27:0/2009398692 learned_addr learned my addr 172.21.15.27:0/2009398692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644e7fc700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6450103570 0x7f6450198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644f7fe700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6450103f10 0x7f645019f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 -- 172.21.15.27:0/2009398692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6450103570 msgr2=0x7f6450198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6450103570 0x7f6450198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 -- 172.21.15.27:0/2009398692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6450103f10 msgr2=0x7f645019f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6450103f10 0x7f645019f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 -- 172.21.15.27:0/2009398692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645019ff70 con 0x7f6450100590 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644f7fe700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6450103f10 0x7f645019f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f644effd700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450100590 0x7f6450198310 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f644400c9c0 tx=0x7f644400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.309+0000 7f6454987700 1 -- 172.21.15.27:0/2009398692 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444007ac0 con 0x7f6450100590 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.310+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64501a0260 con 0x7f6450100590 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.310+0000 7f6454987700 1 -- 172.21.15.27:0/2009398692 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6444007c60 con 0x7f6450100590 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.310+0000 7f6454987700 1 -- 172.21.15.27:0/2009398692 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444018660 con 0x7f6450100590 2024-04-25T01:23:29.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.310+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64501a07a0 con 0x7f6450100590 2024-04-25T01:23:29.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.311+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64500507a0 con 0x7f6450100590 2024-04-25T01:23:29.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.312+0000 7f6454987700 1 -- 172.21.15.27:0/2009398692 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f644400f450 con 0x7f6450100590 2024-04-25T01:23:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.312+0000 7f6454987700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f64380a2d60 0x7f64380a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.312+0000 7f6454987700 1 -- 172.21.15.27:0/2009398692 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f64440cf450 con 0x7f6450100590 2024-04-25T01:23:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.312+0000 7f644e7fc700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f64380a2d60 0x7f64380a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.313+0000 7f644e7fc700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f64380a2d60 0x7f64380a5220 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f644000b5c0 tx=0x7f6440005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.317+0000 7f6454987700 1 -- 172.21.15.27:0/2009398692 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6444014030 con 0x7f6450100590 2024-04-25T01:23:29.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.637+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f645002db60 con 0x7f6450100590 2024-04-25T01:23:29.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.639+0000 7f6454987700 1 -- 172.21.15.27:0/2009398692 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f644409c010 con 0x7f6450100590 2024-04-25T01:23:29.621 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-04-25T01:23:29.621 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCRsClm/UUeJhAAeKaNG6M4RTc6sxYhvvheDQ== 2024-04-25T01:23:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.641+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f64380a2d60 msgr2=0x7f64380a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.641+0000 7f6455989700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f64380a2d60 0x7f64380a5220 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f644000b5c0 tx=0x7f6440005fd0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.641+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450100590 msgr2=0x7f6450198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.641+0000 7f6455989700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450100590 0x7f6450198310 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f644400c9c0 tx=0x7f644400ccd0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.642+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 shutdown_connections 2024-04-25T01:23:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.642+0000 7f6455989700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f64380a2d60 0x7f64380a5220 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.642+0000 7f6455989700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6450103f10 0x7f645019f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.642+0000 7f6455989700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6450100590 0x7f6450198310 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.642+0000 7f6455989700 1 --2- 172.21.15.27:0/2009398692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6450103570 0x7f6450198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.642+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 >> 172.21.15.27:0/2009398692 conn(0x7f64500fbf10 msgr2=0x7f64501067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.642+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 shutdown_connections 2024-04-25T01:23:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:29.642+0000 7f6455989700 1 -- 172.21.15.27:0/2009398692 wait complete. 2024-04-25T01:23:30.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:29 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2009398692' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:23:30.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:29 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2009398692' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:23:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:29 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2009398692' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:23:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:29 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2009398692' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:23:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:29 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2009398692' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:23:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:29 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2009398692' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:23:30.296 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:23:30.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T01:23:30.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T01:23:30.333 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T01:23:30.333 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T01:23:30.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T01:23:30.621 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:31.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:30 smithi027 ceph-mon[29197]: pgmap v311: 129 pgs: 129 active+clean; 625 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-04-25T01:23:31.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:30 smithi153 ceph-mon[32621]: pgmap v311: 129 pgs: 129 active+clean; 625 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-04-25T01:23:31.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:30 smithi139 ceph-mon[32760]: pgmap v311: 129 pgs: 129 active+clean; 625 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-04-25T01:23:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 -- 172.21.15.27:0/4240845300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c1011c0 msgr2=0x7f4f7c10f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/4240845300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c1011c0 0x7f4f7c10f100 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400b0a0 tx=0x7f4f7400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 -- 172.21.15.27:0/4240845300 shutdown_connections 2024-04-25T01:23:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/4240845300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c1011c0 0x7f4f7c10f100 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/4240845300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f7c1082c0 0x7f4f7c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/4240845300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c100820 0x7f4f7c100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 -- 172.21.15.27:0/4240845300 >> 172.21.15.27:0/4240845300 conn(0x7f4f7c0777a0 msgr2=0x7f4f7c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 -- 172.21.15.27:0/4240845300 shutdown_connections 2024-04-25T01:23:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.541+0000 7f4f83fb5700 1 -- 172.21.15.27:0/4240845300 wait complete. 2024-04-25T01:23:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f83fb5700 1 Processor -- start 2024-04-25T01:23:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f83fb5700 1 -- start start 2024-04-25T01:23:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f83fb5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c100820 0x7f4f7c101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f83fb5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c1011c0 0x7f4f7c101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f83fb5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f7c1082c0 0x7f4f7c102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f83fb5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c112540 con 0x7f4f7c100820 2024-04-25T01:23:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f83fb5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c1123c0 con 0x7f4f7c1082c0 2024-04-25T01:23:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f83fb5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c1126c0 con 0x7f4f7c1011c0 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f81d51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c100820 0x7f4f7c101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f81d51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c100820 0x7f4f7c101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44986/0 (socket says 172.21.15.27:44986) 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f81d51700 1 -- 172.21.15.27:0/2043787059 learned_addr learned my addr 172.21.15.27:0/2043787059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.542+0000 7f4f81550700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c1011c0 0x7f4f7c101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f82552700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f7c1082c0 0x7f4f7c102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f81d51700 1 -- 172.21.15.27:0/2043787059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c1011c0 msgr2=0x7f4f7c101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f81d51700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c1011c0 0x7f4f7c101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f81d51700 1 -- 172.21.15.27:0/2043787059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f7c1082c0 msgr2=0x7f4f7c102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f81d51700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f7c1082c0 0x7f4f7c102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f81d51700 1 -- 172.21.15.27:0/2043787059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c1a6de0 con 0x7f4f7c100820 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f81550700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c1011c0 0x7f4f7c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f82552700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f7c1082c0 0x7f4f7c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f81d51700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c100820 0x7f4f7c101a00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000c9b0 tx=0x7f4f7000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f6effd700 1 -- 172.21.15.27:0/2043787059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70007b00 con 0x7f4f7c100820 2024-04-25T01:23:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7c1a70d0 con 0x7f4f7c100820 2024-04-25T01:23:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f6effd700 1 -- 172.21.15.27:0/2043787059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f70007ca0 con 0x7f4f7c100820 2024-04-25T01:23:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f6effd700 1 -- 172.21.15.27:0/2043787059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70018780 con 0x7f4f7c100820 2024-04-25T01:23:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.543+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f7c1a7660 con 0x7f4f7c100820 2024-04-25T01:23:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.544+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f7c103b40 con 0x7f4f7c100820 2024-04-25T01:23:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.545+0000 7f4f6effd700 1 -- 172.21.15.27:0/2043787059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4f7000f450 con 0x7f4f7c100820 2024-04-25T01:23:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.546+0000 7f4f6effd700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4f680a2cb0 0x7f4f680a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.546+0000 7f4f81550700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4f680a2cb0 0x7f4f680a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.546+0000 7f4f6effd700 1 -- 172.21.15.27:0/2043787059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4f700cf490 con 0x7f4f7c100820 2024-04-25T01:23:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.546+0000 7f4f81550700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4f680a2cb0 0x7f4f680a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c107980 tx=0x7f4f7800b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.550+0000 7f4f6effd700 1 -- 172.21.15.27:0/2043787059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4f70014030 con 0x7f4f7c100820 2024-04-25T01:23:32.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.814+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f4f7c02db30 con 0x7f4f7c100820 2024-04-25T01:23:32.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.819+0000 7f4f6effd700 1 -- 172.21.15.27:0/2043787059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263201 (secure 0 0 0) 0x7f4f7009bff0 con 0x7f4f7c100820 2024-04-25T01:23:32.804 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4f680a2cb0 msgr2=0x7f4f680a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4f680a2cb0 0x7f4f680a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c107980 tx=0x7f4f7800b2c0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c100820 msgr2=0x7f4f7c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c100820 0x7f4f7c101a00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000c9b0 tx=0x7f4f7000cd70 comp rx=0 tx=0).stop 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 shutdown_connections 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4f680a2cb0 0x7f4f680a5170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4f7c1082c0 0x7f4f7c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f7c100820 0x7f4f7c101a00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 --2- 172.21.15.27:0/2043787059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c1011c0 0x7f4f7c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 >> 172.21.15.27:0/2043787059 conn(0x7f4f7c0777a0 msgr2=0x7f4f7c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 shutdown_connections 2024-04-25T01:23:32.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:32.824+0000 7f4f83fb5700 1 -- 172.21.15.27:0/2043787059 wait complete. 2024-04-25T01:23:33.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:32 smithi027 ceph-mon[29197]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-04-25T01:23:33.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:32 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2043787059' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:23:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:32 smithi153 ceph-mon[32621]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-04-25T01:23:33.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:32 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2043787059' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:23:33.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:32 smithi139 ceph-mon[32760]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-04-25T01:23:33.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:32 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2043787059' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:23:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":4052240654},{"type":"v1","addr":"172.21.15.27:6801","nonce":4052240654}]},"active_addr":"172.21.15.27:6801/4052240654","active_change":"2024-04-25T01:15:45.288620+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.27:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2504114047}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":4288305036}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3292898694}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2238094164}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2172845232}]}]} 2024-04-25T01:23:33.697 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T01:23:33.697 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T01:23:33.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:34 smithi153 ceph-mon[32621]: pgmap v313: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-25T01:23:35.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:34 smithi139 ceph-mon[32760]: pgmap v313: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-25T01:23:35.304 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:34 smithi027 ceph-mon[29197]: pgmap v313: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 -- 172.21.15.27:0/2033848778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 msgr2=0x7fa97c106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 --2- 172.21.15.27:0/2033848778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 0x7fa97c106e90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa970009990 tx=0x7fa970009ca0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 -- 172.21.15.27:0/2033848778 shutdown_connections 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 --2- 172.21.15.27:0/2033848778 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa97c107570 0x7fa97c10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 --2- 172.21.15.27:0/2033848778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c1024e0 0x7fa97c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 --2- 172.21.15.27:0/2033848778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 0x7fa97c106e90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 -- 172.21.15.27:0/2033848778 >> 172.21.15.27:0/2033848778 conn(0x7fa97c079390 msgr2=0x7fa97c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 -- 172.21.15.27:0/2033848778 shutdown_connections 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.727+0000 7fa9808ad700 1 -- 172.21.15.27:0/2033848778 wait complete. 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa9808ad700 1 Processor -- start 2024-04-25T01:23:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa9808ad700 1 -- start start 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa9808ad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c1024e0 0x7fa97c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa9808ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 0x7fa97c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa9808ad700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa97c107570 0x7fa97c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa9808ad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c112590 con 0x7fa97c102e90 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa9808ad700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c112410 con 0x7fa97c107570 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa9808ad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c112710 con 0x7fa97c1024e0 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa979d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 0x7fa97c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa979d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 0x7fa97c074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45018/0 (socket says 172.21.15.27:45018) 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa979d9b700 1 -- 172.21.15.27:0/1996834764 learned_addr learned my addr 172.21.15.27:0/1996834764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa97a59c700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c1024e0 0x7fa97c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa97ad9d700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa97c107570 0x7fa97c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa979d9b700 1 -- 172.21.15.27:0/1996834764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c1024e0 msgr2=0x7fa97c074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa979d9b700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c1024e0 0x7fa97c074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa979d9b700 1 -- 172.21.15.27:0/1996834764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa97c107570 msgr2=0x7fa97c074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa979d9b700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa97c107570 0x7fa97c074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa979d9b700 1 -- 172.21.15.27:0/1996834764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa97c1a7030 con 0x7fa97c102e90 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.728+0000 7fa97ad9d700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa97c107570 0x7fa97c074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.729+0000 7fa979d9b700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 0x7fa97c074a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9700038b0 tx=0x7fa9700058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.729+0000 7fa96b7fe700 1 -- 172.21.15.27:0/1996834764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970026070 con 0x7fa97c102e90 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.729+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa97c1a72c0 con 0x7fa97c102e90 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.729+0000 7fa96b7fe700 1 -- 172.21.15.27:0/1996834764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa970005510 con 0x7fa97c102e90 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.729+0000 7fa96b7fe700 1 -- 172.21.15.27:0/1996834764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970004960 con 0x7fa97c102e90 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.729+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa97c1a7710 con 0x7fa97c102e90 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.730+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa97c103970 con 0x7fa97c102e90 2024-04-25T01:23:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.731+0000 7fa96b7fe700 1 -- 172.21.15.27:0/1996834764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa970029070 con 0x7fa97c102e90 2024-04-25T01:23:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.731+0000 7fa96b7fe700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9640a2cb0 0x7fa9640a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.732+0000 7fa97a59c700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9640a2cb0 0x7fa9640a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.732+0000 7fa96b7fe700 1 -- 172.21.15.27:0/1996834764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa9700cfd20 con 0x7fa97c102e90 2024-04-25T01:23:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.732+0000 7fa97a59c700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9640a2cb0 0x7fa9640a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa96c006fd0 tx=0x7fa96c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.736+0000 7fa96b7fe700 1 -- 172.21.15.27:0/1996834764 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa97005d740 con 0x7fa97c102e90 2024-04-25T01:23:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.942+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa97c02dcf0 con 0x7fa97c102e90 2024-04-25T01:23:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.944+0000 7fa96b7fe700 1 -- 172.21.15.27:0/1996834764 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7fa9700bae20 con 0x7fa97c102e90 2024-04-25T01:23:35.925 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:35.926 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9640a2cb0 msgr2=0x7fa9640a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9640a2cb0 0x7fa9640a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa96c006fd0 tx=0x7fa96c008040 comp rx=0 tx=0).stop 2024-04-25T01:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 msgr2=0x7fa97c074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 0x7fa97c074a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9700038b0 tx=0x7fa9700058e0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 shutdown_connections 2024-04-25T01:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9640a2cb0 0x7fa9640a5170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa97c107570 0x7fa97c074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa97c1024e0 0x7fa97c074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 --2- 172.21.15.27:0/1996834764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97c102e90 0x7fa97c074a70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:35.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 >> 172.21.15.27:0/1996834764 conn(0x7fa97c079390 msgr2=0x7fa97c101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:35.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 shutdown_connections 2024-04-25T01:23:35.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:35.947+0000 7fa9808ad700 1 -- 172.21.15.27:0/1996834764 wait complete. 2024-04-25T01:23:36.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:35 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1996834764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:36.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:35 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1996834764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:36.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:35 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1996834764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:36.752 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T01:23:36.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:23:37.010 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:37.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:36 smithi027 ceph-mon[29197]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T01:23:37.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:36 smithi153 ceph-mon[32621]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T01:23:37.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:36 smithi139 ceph-mon[32760]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 -- 172.21.15.27:0/212256245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 msgr2=0x7fe330102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 --2- 172.21.15.27:0/212256245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 0x7fe330102740 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe320009990 tx=0x7fe320009ca0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 -- 172.21.15.27:0/212256245 shutdown_connections 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 --2- 172.21.15.27:0/212256245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe330102c80 0x7fe33010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 --2- 172.21.15.27:0/212256245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe330108360 0x7fe330108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 --2- 172.21.15.27:0/212256245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 0x7fe330102740 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 -- 172.21.15.27:0/212256245 >> 172.21.15.27:0/212256245 conn(0x7fe330079420 msgr2=0x7fe330079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 -- 172.21.15.27:0/212256245 shutdown_connections 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.933+0000 7fe33849e700 1 -- 172.21.15.27:0/212256245 wait complete. 2024-04-25T01:23:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33849e700 1 Processor -- start 2024-04-25T01:23:38.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33849e700 1 -- start start 2024-04-25T01:23:38.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33849e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 0x7fe33019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:38.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33849e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe330102c80 0x7fe33019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:38.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33849e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe330108360 0x7fe3301989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:38.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33849e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe330112680 con 0x7fe3301022c0 2024-04-25T01:23:38.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33849e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe330112500 con 0x7fe330102c80 2024-04-25T01:23:38.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33849e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe330112800 con 0x7fe330108360 2024-04-25T01:23:38.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33623a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 0x7fe33019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:38.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33623a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 0x7fe33019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54740/0 (socket says 172.21.15.27:54740) 2024-04-25T01:23:38.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33623a700 1 -- 172.21.15.27:0/3512268754 learned_addr learned my addr 172.21.15.27:0/3512268754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:38.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe335a39700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe330102c80 0x7fe33019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe336a3b700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe330108360 0x7fe3301989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33623a700 1 -- 172.21.15.27:0/3512268754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe330108360 msgr2=0x7fe3301989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33623a700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe330108360 0x7fe3301989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33623a700 1 -- 172.21.15.27:0/3512268754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe330102c80 msgr2=0x7fe33019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33623a700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe330102c80 0x7fe33019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.934+0000 7fe33623a700 1 -- 172.21.15.27:0/3512268754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe330199140 con 0x7fe3301022c0 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.935+0000 7fe33623a700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 0x7fe33019e820 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00b530 tx=0x7fe32c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.935+0000 7fe3277fe700 1 -- 172.21.15.27:0/3512268754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c015020 con 0x7fe3301022c0 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.935+0000 7fe3277fe700 1 -- 172.21.15.27:0/3512268754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe32c011d10 con 0x7fe3301022c0 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.935+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe330199430 con 0x7fe3301022c0 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.935+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3301a56f0 con 0x7fe3301022c0 2024-04-25T01:23:38.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.935+0000 7fe3277fe700 1 -- 172.21.15.27:0/3512268754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c00f2e0 con 0x7fe3301022c0 2024-04-25T01:23:38.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.936+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe33010a820 con 0x7fe3301022c0 2024-04-25T01:23:38.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.937+0000 7fe3277fe700 1 -- 172.21.15.27:0/3512268754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe32c018020 con 0x7fe3301022c0 2024-04-25T01:23:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.938+0000 7fe3277fe700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe31c0a2cb0 0x7fe31c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.938+0000 7fe335a39700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe31c0a2cb0 0x7fe31c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.938+0000 7fe3277fe700 1 -- 172.21.15.27:0/3512268754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe32c0cefe0 con 0x7fe3301022c0 2024-04-25T01:23:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.939+0000 7fe335a39700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe31c0a2cb0 0x7fe31c0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe32000b5c0 tx=0x7fe320005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:38.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:38.942+0000 7fe3277fe700 1 -- 172.21.15.27:0/3512268754 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe32c01f020 con 0x7fe3301022c0 2024-04-25T01:23:39.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.147+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3301039d0 con 0x7fe3301022c0 2024-04-25T01:23:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.149+0000 7fe3277fe700 1 -- 172.21.15.27:0/3512268754 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24124 (secure 0 0 0) 0x7fe32c09bc50 con 0x7fe3301022c0 2024-04-25T01:23:39.130 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:23:39.131 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:02.224149+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:23:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe31c0a2cb0 msgr2=0x7fe31c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe31c0a2cb0 0x7fe31c0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe32000b5c0 tx=0x7fe320005960 comp rx=0 tx=0).stop 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 msgr2=0x7fe33019e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 0x7fe33019e820 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00b530 tx=0x7fe32c00b840 comp rx=0 tx=0).stop 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 shutdown_connections 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe31c0a2cb0 0x7fe31c0a5170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe330108360 0x7fe3301989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3301022c0 0x7fe33019e820 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 --2- 172.21.15.27:0/3512268754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe330102c80 0x7fe33019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 >> 172.21.15.27:0/3512268754 conn(0x7fe330079420 msgr2=0x7fe330101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 shutdown_connections 2024-04-25T01:23:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:39.151+0000 7fe33849e700 1 -- 172.21.15.27:0/3512268754 wait complete. 2024-04-25T01:23:39.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:38 smithi027 ceph-mon[29197]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-04-25T01:23:39.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:38 smithi153 ceph-mon[32621]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-04-25T01:23:39.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:38 smithi139 ceph-mon[32760]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-04-25T01:23:39.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T01:23:39.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T01:23:39.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T01:23:39.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T01:23:39.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T01:23:39.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T01:23:39.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T01:23:39.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T01:23:39.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T01:23:39.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T01:23:39.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T01:23:39.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T01:23:40.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:39 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3512268754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:40.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:39 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3512268754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:40.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:39 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3512268754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:23:40.544 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:40.557 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:40.558 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:40.629 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:40.639 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:40.895 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:40.909 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:41.012 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:41.013 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:41.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:40 smithi027 ceph-mon[29197]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-04-25T01:23:41.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:40 smithi153 ceph-mon[32621]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-04-25T01:23:41.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:40 smithi139 ceph-mon[32760]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-04-25T01:23:43.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:42 smithi027 ceph-mon[29197]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-04-25T01:23:43.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:42 smithi153 ceph-mon[32621]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-04-25T01:23:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:43.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:42 smithi139 ceph-mon[32760]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-04-25T01:23:45.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:44 smithi027 ceph-mon[29197]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:23:45.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:44 smithi153 ceph-mon[32621]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:23:45.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:44 smithi139 ceph-mon[32760]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:23:46.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-25T01:23:46.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-25T01:23:46.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-25T01:23:46.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-04-25T01:23:46.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-25T01:23:46.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]: dispatch 2024-04-25T01:23:46.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-25T01:23:46.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-25T01:23:46.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-25T01:23:46.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:45 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-04-25T01:23:46.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-25T01:23:46.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-25T01:23:46.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-25T01:23:46.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-04-25T01:23:46.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-25T01:23:46.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]: dispatch 2024-04-25T01:23:46.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-25T01:23:46.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-25T01:23:46.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-25T01:23:46.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:45 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-04-25T01:23:46.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-25T01:23:46.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:45 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-04-25T01:23:47.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-04-25T01:23:47.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:47 smithi027 ceph-mon[29197]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:23:47.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:23:47.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-25T01:23:47.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-25T01:23:47.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-25T01:23:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-04-25T01:23:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-25T01:23:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]': finished 2024-04-25T01:23:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-25T01:23:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-25T01:23:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-25T01:23:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-04-25T01:23:47.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:47 smithi153 ceph-mon[32621]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:23:47.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:23:47.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-04-25T01:23:47.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:47 smithi139 ceph-mon[32760]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:23:48.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:48 smithi027 ceph-mon[29197]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:23:48.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:48 smithi153 ceph-mon[32621]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:23:48.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:48 smithi139 ceph-mon[32760]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:23:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 -- 172.21.15.27:0/3916775657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c1082d0 msgr2=0x7f8d3c1086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 --2- 172.21.15.27:0/3916775657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c1082d0 0x7f8d3c1086b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009990 tx=0x7f8d2c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:23:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 -- 172.21.15.27:0/3916775657 shutdown_connections 2024-04-25T01:23:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 --2- 172.21.15.27:0/3916775657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d3c06b2a0 0x7f8d3c10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 --2- 172.21.15.27:0/3916775657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c1082d0 0x7f8d3c1086b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 --2- 172.21.15.27:0/3916775657 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d3c06a900 0x7f8d3c06ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 -- 172.21.15.27:0/3916775657 >> 172.21.15.27:0/3916775657 conn(0x7f8d3c0780e0 msgr2=0x7f8d3c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 -- 172.21.15.27:0/3916775657 shutdown_connections 2024-04-25T01:23:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.343+0000 7f8d40850700 1 -- 172.21.15.27:0/3916775657 wait complete. 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d40850700 1 Processor -- start 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d40850700 1 -- start start 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d40850700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d3c06a900 0x7f8d3c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d40850700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c06b2a0 0x7f8d3c19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d40850700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d3c1082d0 0x7f8d3c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d40850700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c112530 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d40850700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c1123b0 con 0x7f8d3c06a900 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d40850700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c1126b0 con 0x7f8d3c1082d0 2024-04-25T01:23:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d33fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c06b2a0 0x7f8d3c19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d33fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c06b2a0 0x7f8d3c19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54030/0 (socket says 172.21.15.27:54030) 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d33fff700 1 -- 172.21.15.27:0/538431420 learned_addr learned my addr 172.21.15.27:0/538431420 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d3a59c700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d3c06a900 0x7f8d3c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.344+0000 7f8d3ad9d700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d3c1082d0 0x7f8d3c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d33fff700 1 -- 172.21.15.27:0/538431420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d3c1082d0 msgr2=0x7f8d3c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d33fff700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d3c1082d0 0x7f8d3c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d33fff700 1 -- 172.21.15.27:0/538431420 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d3c06a900 msgr2=0x7f8d3c19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d33fff700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d3c06a900 0x7f8d3c19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d33fff700 1 -- 172.21.15.27:0/538431420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3c198f70 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d3ad9d700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d3c1082d0 0x7f8d3c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d33fff700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c06b2a0 0x7f8d3c19eb50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400e9a0 tx=0x7f8d2400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400cbd0 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3c199260 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d2400cd70 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24018750 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.345+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d3c1a5470 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.346+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8d3c0629c0 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.347+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8d24010430 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.348+0000 7f8d337fe700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d280a2d30 0x7f8d280a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.348+0000 7f8d3a59c700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d280a2d30 0x7f8d280a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.348+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8d240cff60 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.348+0000 7f8d337fe700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] conn(0x7f8d280ade90 0x7f8d280b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.348+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 --> [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d280b0a00 con 0x7f8d280ade90 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.348+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f8d240d03b0 con 0x7f8d3c06b2a0 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.349+0000 7f8d3ad9d700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] conn(0x7f8d280ade90 0x7f8d280b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.349+0000 7f8d3a59c700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d280a2d30 0x7f8d280a51f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00b2e0 tx=0x7f8d2c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.349+0000 7f8d3ad9d700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] conn(0x7f8d280ade90 0x7f8d280b0330 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:48.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.350+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 <== osd.0 v2:172.21.15.27:6802/4134559555 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d280b0a00 con 0x7f8d280ade90 2024-04-25T01:23:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.366+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 --> [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8d3c062dc0 con 0x7f8d280ade90 2024-04-25T01:23:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d337fe700 1 -- 172.21.15.27:0/538431420 <== osd.0 v2:172.21.15.27:6802/4134559555 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8d3c062dc0 con 0x7f8d280ade90 2024-04-25T01:23:48.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] conn(0x7f8d280ade90 msgr2=0x7f8d280b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:48.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] conn(0x7f8d280ade90 0x7f8d280b0330 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:48.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d280a2d30 msgr2=0x7f8d280a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:48.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f8d280a2d30 0x7f8d280a51f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00b2e0 tx=0x7f8d2c005960 comp rx=0 tx=0).stop 2024-04-25T01:23:48.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c06b2a0 msgr2=0x7f8d3c19eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:48.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 --2- 172.21.15.27:0/538431420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d3c06b2a0 0x7f8d3c19eb50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400e9a0 tx=0x7f8d2400ecb0 comp rx=0 tx=0).stop 2024-04-25T01:23:48.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d3ad9d700 1 -- 172.21.15.27:0/538431420 reap_dead start 2024-04-25T01:23:48.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 shutdown_connections 2024-04-25T01:23:48.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 >> 172.21.15.27:0/538431420 conn(0x7f8d3c0780e0 msgr2=0x7f8d3c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:48.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 shutdown_connections 2024-04-25T01:23:48.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:48.367+0000 7f8d40850700 1 -- 172.21.15.27:0/538431420 wait complete. 2024-04-25T01:23:49.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:49 smithi027 ceph-mon[29197]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-25T01:23:49.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:49 smithi153 ceph-mon[32621]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-25T01:23:49.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:49 smithi139 ceph-mon[32760]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 -- 172.21.15.27:0/1694689013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f599810ec60 msgr2=0x7f5998111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 --2- 172.21.15.27:0/1694689013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f599810ec60 0x7f5998111050 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5988009990 tx=0x7f5988009ca0 comp rx=0 tx=0).stop 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 -- 172.21.15.27:0/1694689013 shutdown_connections 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 --2- 172.21.15.27:0/1694689013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f599810ec60 0x7f5998111050 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 --2- 172.21.15.27:0/1694689013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5998074460 0x7f5998074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 --2- 172.21.15.27:0/1694689013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5998074d80 0x7f599810e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 -- 172.21.15.27:0/1694689013 >> 172.21.15.27:0/1694689013 conn(0x7f59980793e0 msgr2=0x7f59980797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 -- 172.21.15.27:0/1694689013 shutdown_connections 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.429+0000 7f599cae5700 1 -- 172.21.15.27:0/1694689013 wait complete. 2024-04-25T01:23:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599cae5700 1 Processor -- start 2024-04-25T01:23:50.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599cae5700 1 -- start start 2024-04-25T01:23:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599cae5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5998074460 0x7f59981a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599cae5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5998074d80 0x7f59981a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599cae5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599810ec60 0x7f599819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599cae5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5998114830 con 0x7f5998074460 2024-04-25T01:23:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599cae5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59981146b0 con 0x7f5998074d80 2024-04-25T01:23:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599cae5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59981149b0 con 0x7f599810ec60 2024-04-25T01:23:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5998074460 0x7f59981a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5998074460 0x7f59981a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54056/0 (socket says 172.21.15.27:54056) 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f599659c700 1 -- 172.21.15.27:0/280169352 learned_addr learned my addr 172.21.15.27:0/280169352 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.430+0000 7f5996d9d700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599810ec60 0x7f599819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f5995d9b700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5998074d80 0x7f59981a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f599659c700 1 -- 172.21.15.27:0/280169352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599810ec60 msgr2=0x7f599819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f599659c700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f599810ec60 0x7f599819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f599659c700 1 -- 172.21.15.27:0/280169352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5998074d80 msgr2=0x7f59981a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f599659c700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5998074d80 0x7f59981a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f599659c700 1 -- 172.21.15.27:0/280169352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599819b230 con 0x7f5998074460 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f599659c700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5998074460 0x7f59981a0910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f599000ada0 tx=0x7f599000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990017020 con 0x7f5998074460 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599819b520 con 0x7f5998074460 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5990004bb0 con 0x7f5998074460 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59900054a0 con 0x7f5998074460 2024-04-25T01:23:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.431+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59981a77e0 con 0x7f5998074460 2024-04-25T01:23:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.432+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5998062940 con 0x7f5998074460 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.433+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5990005640 con 0x7f5998074460 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.434+0000 7f598f7fe700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f59840a2cb0 0x7f59840a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.434+0000 7f5995d9b700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f59840a2cb0 0x7f59840a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.434+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f59900cf6c0 con 0x7f5998074460 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.434+0000 7f598f7fe700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] conn(0x7f59840ade10 0x7f59840b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.434+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 --> [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59840b0980 con 0x7f59840ade10 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.434+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f59900cfb10 con 0x7f5998074460 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.434+0000 7f5996d9d700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] conn(0x7f59840ade10 0x7f59840b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.434+0000 7f5995d9b700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f59840a2cb0 0x7f59840a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5980006fd0 tx=0x7f5980008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.435+0000 7f5996d9d700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] conn(0x7f59840ade10 0x7f59840b02b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:50.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.435+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 <== osd.9 v2:172.21.15.27:6826/91302017 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f59840b0980 con 0x7f59840ade10 2024-04-25T01:23:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.452+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 --> [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5998062d40 con 0x7f59840ade10 2024-04-25T01:23:50.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f598f7fe700 1 -- 172.21.15.27:0/280169352 <== osd.9 v2:172.21.15.27:6826/91302017 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5998062d40 con 0x7f59840ade10 2024-04-25T01:23:50.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] conn(0x7f59840ade10 msgr2=0x7f59840b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] conn(0x7f59840ade10 0x7f59840b02b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f59840a2cb0 msgr2=0x7f59840a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f59840a2cb0 0x7f59840a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5980006fd0 tx=0x7f5980008040 comp rx=0 tx=0).stop 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5998074460 msgr2=0x7f59981a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 --2- 172.21.15.27:0/280169352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5998074460 0x7f59981a0910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f599000ada0 tx=0x7f599000c330 comp rx=0 tx=0).stop 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f5996d9d700 1 -- 172.21.15.27:0/280169352 reap_dead start 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 shutdown_connections 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 >> 172.21.15.27:0/280169352 conn(0x7f59980793e0 msgr2=0x7f59981004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 shutdown_connections 2024-04-25T01:23:50.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:50.453+0000 7f599cae5700 1 -- 172.21.15.27:0/280169352 wait complete. 2024-04-25T01:23:51.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:51 smithi027 ceph-mon[29197]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:51.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:51 smithi153 ceph-mon[32621]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:51.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:51 smithi139 ceph-mon[32760]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:53.258 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:53 smithi027 ceph-mon[29197]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:53.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:53 smithi153 ceph-mon[32621]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:53.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:53 smithi139 ceph-mon[32760]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:54.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 -- 172.21.15.27:0/3138038536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0101800 msgr2=0x7f0fc0106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 --2- 172.21.15.27:0/3138038536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0101800 0x7f0fc0106f40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40099c0 tx=0x7f0fb4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 -- 172.21.15.27:0/3138038536 shutdown_connections 2024-04-25T01:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 --2- 172.21.15.27:0/3138038536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc0107670 0x7f0fc010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 --2- 172.21.15.27:0/3138038536 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fc0100e50 0x7f0fc0101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 --2- 172.21.15.27:0/3138038536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0101800 0x7f0fc0106f40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 -- 172.21.15.27:0/3138038536 >> 172.21.15.27:0/3138038536 conn(0x7f0fc0076450 msgr2=0x7f0fc0076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 -- 172.21.15.27:0/3138038536 shutdown_connections 2024-04-25T01:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.669+0000 7f0fc5371700 1 -- 172.21.15.27:0/3138038536 wait complete. 2024-04-25T01:23:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.670+0000 7f0fc5371700 1 Processor -- start 2024-04-25T01:23:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.670+0000 7f0fc5371700 1 -- start start 2024-04-25T01:23:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.670+0000 7f0fc5371700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0100e50 0x7f0fc019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.670+0000 7f0fc5371700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc0101800 0x7f0fc019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.670+0000 7f0fc5371700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fc0107670 0x7f0fc01988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.670+0000 7f0fc5371700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0112660 con 0x7f0fc0100e50 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.670+0000 7f0fc5371700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01124e0 con 0x7f0fc0107670 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.670+0000 7f0fc5371700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01127e0 con 0x7f0fc0101800 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0100e50 0x7f0fc019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0100e50 0x7f0fc019e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54082/0 (socket says 172.21.15.27:54082) 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 -- 172.21.15.27:0/4269792921 learned_addr learned my addr 172.21.15.27:0/4269792921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbe7fc700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc0101800 0x7f0fc019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbf7fe700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fc0107670 0x7f0fc01988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 -- 172.21.15.27:0/4269792921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc0101800 msgr2=0x7f0fc019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fc0101800 0x7f0fc019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 -- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fc0107670 msgr2=0x7f0fc01988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fc0107670 0x7f0fc01988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 -- 172.21.15.27:0/4269792921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc0199020 con 0x7f0fc0100e50 2024-04-25T01:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbf7fe700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fc0107670 0x7f0fc01988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fbeffd700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0100e50 0x7f0fc019e700 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000b530 tx=0x7f0fb000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0015020 con 0x7f0fc0100e50 2024-04-25T01:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc0199310 con 0x7f0fc0100e50 2024-04-25T01:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb0011d10 con 0x7f0fc0100e50 2024-04-25T01:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb000f2e0 con 0x7f0fc0100e50 2024-04-25T01:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.671+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc01a55d0 con 0x7f0fc0100e50 2024-04-25T01:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.672+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0fc00629c0 con 0x7f0fc0100e50 2024-04-25T01:23:54.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.674+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0fb0018020 con 0x7f0fc0100e50 2024-04-25T01:23:54.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.674+0000 7f0fa7fff700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0fa80a2cb0 0x7f0fa80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:54.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.674+0000 7f0fbe7fc700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0fa80a2cb0 0x7f0fa80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:54.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.674+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0fb00cf280 con 0x7f0fc0100e50 2024-04-25T01:23:54.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.674+0000 7f0fa7fff700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] conn(0x7f0fa80ade10 0x7f0fa80b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.675+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 --> [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fb00044e0 con 0x7f0fa80ade10 2024-04-25T01:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.675+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f0fb00cf6d0 con 0x7f0fc0100e50 2024-04-25T01:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.675+0000 7f0fbe7fc700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0fa80a2cb0 0x7f0fa80a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4000c00 tx=0x7f0fb4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.675+0000 7f0fbf7fe700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] conn(0x7f0fa80ade10 0x7f0fa80b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.675+0000 7f0fbf7fe700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] conn(0x7f0fa80ade10 0x7f0fa80b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.676+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 <== osd.4 v2:172.21.15.139:6808/1457327101 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fb00044e0 con 0x7f0fa80ade10 2024-04-25T01:23:54.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.693+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 --> [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0fc0062e40 con 0x7f0fa80ade10 2024-04-25T01:23:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.693+0000 7f0fa7fff700 1 -- 172.21.15.27:0/4269792921 <== osd.4 v2:172.21.15.139:6808/1457327101 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0fc0062e40 con 0x7f0fa80ade10 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.693+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] conn(0x7f0fa80ade10 msgr2=0x7f0fa80b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.693+0000 7f0fc5371700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] conn(0x7f0fa80ade10 0x7f0fa80b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0fa80a2cb0 msgr2=0x7f0fa80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fc5371700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0fa80a2cb0 0x7f0fa80a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4000c00 tx=0x7f0fb4005fb0 comp rx=0 tx=0).stop 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0100e50 msgr2=0x7f0fc019e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fc5371700 1 --2- 172.21.15.27:0/4269792921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc0100e50 0x7f0fc019e700 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000b530 tx=0x7f0fb000b840 comp rx=0 tx=0).stop 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fbf7fe700 1 -- 172.21.15.27:0/4269792921 reap_dead start 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 shutdown_connections 2024-04-25T01:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 >> 172.21.15.27:0/4269792921 conn(0x7f0fc0076450 msgr2=0x7f0fc0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 shutdown_connections 2024-04-25T01:23:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.694+0000 7f0fc5371700 1 -- 172.21.15.27:0/4269792921 wait complete. 2024-04-25T01:23:55.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:55 smithi027 ceph-mon[29197]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:55.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:55 smithi153 ceph-mon[32621]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:55.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:55 smithi139 ceph-mon[32760]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-25T01:23:56.054 INFO:teuthology.orchestra.run.smithi027.stdout:141733920830 2024-04-25T01:23:56.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T01:23:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 -- 172.21.15.27:0/2025693318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7c107600 msgr2=0x7f0b7c10f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 --2- 172.21.15.27:0/2025693318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7c107600 0x7f0b7c10f180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400b0d0 tx=0x7f0b7400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 -- 172.21.15.27:0/2025693318 shutdown_connections 2024-04-25T01:23:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 --2- 172.21.15.27:0/2025693318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7c107600 0x7f0b7c10f180 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 --2- 172.21.15.27:0/2025693318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b7c06ab60 0x7f0b7c06af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 --2- 172.21.15.27:0/2025693318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b7c06b510 0x7f0b7c106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 -- 172.21.15.27:0/2025693318 >> 172.21.15.27:0/2025693318 conn(0x7f0b7c0780e0 msgr2=0x7f0b7c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 -- 172.21.15.27:0/2025693318 shutdown_connections 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.808+0000 7f0b8125d700 1 -- 172.21.15.27:0/2025693318 wait complete. 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.810+0000 7f0b8125d700 1 Processor -- start 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.810+0000 7f0b8125d700 1 -- start start 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.810+0000 7f0b8125d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7c06ab60 0x7f0b7c1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.810+0000 7f0b8125d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b7c06b510 0x7f0b7c101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.810+0000 7f0b8125d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b7c107600 0x7f0b7c102060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.810+0000 7f0b8125d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c112570 con 0x7f0b7c06ab60 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.810+0000 7f0b8125d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c1123f0 con 0x7f0b7c06b510 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.810+0000 7f0b8125d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c1126f0 con 0x7f0b7c107600 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7b59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b7c107600 0x7f0b7c102060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7c06ab60 0x7f0b7c1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7b59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b7c107600 0x7f0b7c102060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:48724/0 (socket says 172.21.15.27:48724) 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7b59e700 1 -- 172.21.15.27:0/1893299042 learned_addr learned my addr 172.21.15.27:0/1893299042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7a59c700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b7c06b510 0x7f0b7c101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7b59e700 1 -- 172.21.15.27:0/1893299042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b7c06b510 msgr2=0x7f0b7c101b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7b59e700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b7c06b510 0x7f0b7c101b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7b59e700 1 -- 172.21.15.27:0/1893299042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7c06ab60 msgr2=0x7f0b7c1015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7b59e700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7c06ab60 0x7f0b7c1015e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7b59e700 1 -- 172.21.15.27:0/1893299042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7c1a6e90 con 0x7f0b7c107600 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.811+0000 7f0b7ad9d700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7c06ab60 0x7f0b7c1015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.812+0000 7f0b7b59e700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b7c107600 0x7f0b7c102060 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400d010 tx=0x7f0b7400bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.812+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b74026070 con 0x7f0b7c107600 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.812+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b74024030 con 0x7f0b7c107600 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.812+0000 7f0b8125d700 1 -- 172.21.15.27:0/1893299042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7c1986a0 con 0x7f0b7c107600 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.812+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b74005610 con 0x7f0b7c107600 2024-04-25T01:23:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.813+0000 7f0b8125d700 1 -- 172.21.15.27:0/1893299042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7c198bd0 con 0x7f0b7c107600 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.814+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0b7c0629c0 con 0x7f0b7c107600 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.816+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0b74009ba0 con 0x7f0b7c107600 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.817+0000 7f0b63fff700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0b640a2d30 0x7f0b640a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.817+0000 7f0b7ad9d700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0b640a2d30 0x7f0b640a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.817+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0b7400f070 con 0x7f0b7c107600 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.817+0000 7f0b63fff700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] conn(0x7f0b640ade90 0x7f0b640b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.817+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 --> [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b74021e50 con 0x7f0b640ade90 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.818+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f0b740d8860 con 0x7f0b7c107600 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.818+0000 7f0b7a59c700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] conn(0x7f0b640ade90 0x7f0b640b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.818+0000 7f0b7ad9d700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0b640a2d30 0x7f0b640a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c007fd0 tx=0x7f0b6c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.818+0000 7f0b7a59c700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] conn(0x7f0b640ade90 0x7f0b640b0330 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.819+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 <== osd.8 v2:172.21.15.153:6816/4050878685 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0b74021e50 con 0x7f0b640ade90 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.837+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 --> [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0b7c062bd0 con 0x7f0b640ade90 2024-04-25T01:23:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.837+0000 7f0b63fff700 1 -- 172.21.15.27:0/1893299042 <== osd.8 v2:172.21.15.153:6816/4050878685 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0b7c062bd0 con 0x7f0b640ade90 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.837+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 >> [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] conn(0x7f0b640ade90 msgr2=0x7f0b640b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.837+0000 7f0b61ffb700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] conn(0x7f0b640ade90 0x7f0b640b0330 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0b640a2d30 msgr2=0x7f0b640a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b61ffb700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0b640a2d30 0x7f0b640a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c007fd0 tx=0x7f0b6c0073d0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b7c107600 msgr2=0x7f0b7c102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b61ffb700 1 --2- 172.21.15.27:0/1893299042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b7c107600 0x7f0b7c102060 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400d010 tx=0x7f0b7400bee0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b7b59e700 1 -- 172.21.15.27:0/1893299042 reap_dead start 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 shutdown_connections 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 >> 172.21.15.27:0/1893299042 conn(0x7f0b7c0780e0 msgr2=0x7f0b7c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 shutdown_connections 2024-04-25T01:23:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:56.838+0000 7f0b61ffb700 1 -- 172.21.15.27:0/1893299042 wait complete. 2024-04-25T01:23:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:57 smithi027 ceph-mon[29197]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 327 B/s wr, 6 op/s; 61 B/s, 0 objects/s recovering 2024-04-25T01:23:57.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:57 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:57.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:57 smithi153 ceph-mon[32621]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 327 B/s wr, 6 op/s; 61 B/s, 0 objects/s recovering 2024-04-25T01:23:57.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:57 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:57.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:57 smithi139 ceph-mon[32760]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 327 B/s wr, 6 op/s; 61 B/s, 0 objects/s recovering 2024-04-25T01:23:57.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:57 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:57.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 -- 172.21.15.27:0/1225560487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c1075a0 msgr2=0x7f707c10f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 --2- 172.21.15.27:0/1225560487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c1075a0 0x7f707c10f170 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f707400b0a0 tx=0x7f707400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 -- 172.21.15.27:0/1225560487 shutdown_connections 2024-04-25T01:23:57.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 --2- 172.21.15.27:0/1225560487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c1075a0 0x7f707c10f170 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 --2- 172.21.15.27:0/1225560487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f707c102540 0x7f707c102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 --2- 172.21.15.27:0/1225560487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f707c102ef0 0x7f707c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 -- 172.21.15.27:0/1225560487 >> 172.21.15.27:0/1225560487 conn(0x7f707c079310 msgr2=0x7f707c079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 -- 172.21.15.27:0/1225560487 shutdown_connections 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.006+0000 7f7080851700 1 -- 172.21.15.27:0/1225560487 wait complete. 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f7080851700 1 Processor -- start 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f7080851700 1 -- start start 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f7080851700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c102540 0x7f707c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f7080851700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f707c102ef0 0x7f707c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f7080851700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f707c1075a0 0x7f707c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f7080851700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f707c112580 con 0x7f707c102540 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f7080851700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f707c112400 con 0x7f707c102ef0 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f7080851700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f707c112700 con 0x7f707c1075a0 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f707a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c102540 0x7f707c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f707a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c102540 0x7f707c19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54072/0 (socket says 172.21.15.27:54072) 2024-04-25T01:23:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.007+0000 7f707a59c700 1 -- 172.21.15.27:0/2968511457 learned_addr learned my addr 172.21.15.27:0/2968511457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:57.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f707ad9d700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f707c1075a0 0x7f707c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f7079d9b700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f707c102ef0 0x7f707c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f707a59c700 1 -- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f707c1075a0 msgr2=0x7f707c198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f707a59c700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f707c1075a0 0x7f707c198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f707a59c700 1 -- 172.21.15.27:0/2968511457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f707c102ef0 msgr2=0x7f707c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f707a59c700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f707c102ef0 0x7f707c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f707a59c700 1 -- 172.21.15.27:0/2968511457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707c199020 con 0x7f707c102540 2024-04-25T01:23:57.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f7079d9b700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f707c102ef0 0x7f707c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f707ad9d700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f707c1075a0 0x7f707c198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f707a59c700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c102540 0x7f707c19e670 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f706c00d980 tx=0x7f706c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c016070 con 0x7f707c102540 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f707c199310 con 0x7f707c102540 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f706c004500 con 0x7f707c102540 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c004d10 con 0x7f707c102540 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.008+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f707c1a54d0 con 0x7f707c102540 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.009+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f707c0629c0 con 0x7f707c102540 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.010+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f706c00bbf0 con 0x7f707c102540 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.011+0000 7f706b7fe700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70640a2cb0 0x7f70640a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.011+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f706c0cfa20 con 0x7f707c102540 2024-04-25T01:23:57.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.011+0000 7f7079d9b700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70640a2cb0 0x7f70640a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.011+0000 7f706b7fe700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] conn(0x7f70640ade10 0x7f70640b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.011+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 --> [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f706c015ec0 con 0x7f70640ade10 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.011+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f706c0cfe70 con 0x7f707c102540 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.011+0000 7f7079d9b700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70640a2cb0 0x7f70640a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f707c079d30 tx=0x7f7070009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.012+0000 7f707ad9d700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] conn(0x7f70640ade10 0x7f70640b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.012+0000 7f707ad9d700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] conn(0x7f70640ade10 0x7f70640b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.013+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 <== osd.11 v2:172.21.15.153:6824/3171978672 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c015ec0 con 0x7f70640ade10 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.030+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 --> [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f707c062dc0 con 0x7f70640ade10 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f706b7fe700 1 -- 172.21.15.27:0/2968511457 <== osd.11 v2:172.21.15.153:6824/3171978672 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f707c062dc0 con 0x7f70640ade10 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] conn(0x7f70640ade10 msgr2=0x7f70640b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] conn(0x7f70640ade10 0x7f70640b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70640a2cb0 msgr2=0x7f70640a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70640a2cb0 0x7f70640a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f707c079d30 tx=0x7f7070009260 comp rx=0 tx=0).stop 2024-04-25T01:23:57.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c102540 msgr2=0x7f707c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 --2- 172.21.15.27:0/2968511457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f707c102540 0x7f707c19e670 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f706c00d980 tx=0x7f706c00dd40 comp rx=0 tx=0).stop 2024-04-25T01:23:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f707ad9d700 1 -- 172.21.15.27:0/2968511457 reap_dead start 2024-04-25T01:23:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 shutdown_connections 2024-04-25T01:23:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 >> 172.21.15.27:0/2968511457 conn(0x7f707c079310 msgr2=0x7f707c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 shutdown_connections 2024-04-25T01:23:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:54.031+0000 7f7080851700 1 -- 172.21.15.27:0/2968511457 wait complete. 2024-04-25T01:23:58.020 INFO:teuthology.orchestra.run.smithi027.stdout:343597383705 2024-04-25T01:23:58.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T01:23:58.542 INFO:teuthology.orchestra.run.smithi027.stdout:34359738448 2024-04-25T01:23:58.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T01:23:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 -- 172.21.15.27:0/3634320544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 msgr2=0x7fa94c10d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 --2- 172.21.15.27:0/3634320544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 0x7fa94c10d770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa93c009990 tx=0x7fa93c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 -- 172.21.15.27:0/3634320544 shutdown_connections 2024-04-25T01:23:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 --2- 172.21.15.27:0/3634320544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 0x7fa94c10d770 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 --2- 172.21.15.27:0/3634320544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa94c1068a0 0x7fa94c106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 --2- 172.21.15.27:0/3634320544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c1008f0 0x7fa94c100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 -- 172.21.15.27:0/3634320544 >> 172.21.15.27:0/3634320544 conn(0x7fa94c0777a0 msgr2=0x7fa94c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 -- 172.21.15.27:0/3634320544 shutdown_connections 2024-04-25T01:23:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.056+0000 7fa951db5700 1 -- 172.21.15.27:0/3634320544 wait complete. 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa951db5700 1 Processor -- start 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa951db5700 1 -- start start 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa951db5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c1008f0 0x7fa94c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa951db5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 0x7fa94c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa951db5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa94c1068a0 0x7fa94c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa951db5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c110c40 con 0x7fa94c1012b0 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa951db5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c110ac0 con 0x7fa94c1068a0 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa951db5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c110dc0 con 0x7fa94c1008f0 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa94affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 0x7fa94c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa94affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 0x7fa94c19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57496/0 (socket says 172.21.15.27:57496) 2024-04-25T01:23:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa94affd700 1 -- 172.21.15.27:0/735153381 learned_addr learned my addr 172.21.15.27:0/735153381 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa94b7fe700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c1008f0 0x7fa94c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.057+0000 7fa94bfff700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa94c1068a0 0x7fa94c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa94b7fe700 1 -- 172.21.15.27:0/735153381 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa94c1068a0 msgr2=0x7fa94c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa94b7fe700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa94c1068a0 0x7fa94c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa94b7fe700 1 -- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 msgr2=0x7fa94c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa94b7fe700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 0x7fa94c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa94b7fe700 1 -- 172.21.15.27:0/735153381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c198fd0 con 0x7fa94c1008f0 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa94affd700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c1012b0 0x7fa94c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa94b7fe700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c1008f0 0x7fa94c19e670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa94400ef10 tx=0x7fa94400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94400c9c0 con 0x7fa94c1008f0 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c199260 con 0x7fa94c1008f0 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c1a54d0 con 0x7fa94c1008f0 2024-04-25T01:23:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa944019070 con 0x7fa94c1008f0 2024-04-25T01:23:59.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.058+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944014480 con 0x7fa94c1008f0 2024-04-25T01:23:59.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.059+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa94c0ff7b0 con 0x7fa94c1008f0 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.060+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa944012050 con 0x7fa94c1008f0 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.061+0000 7fa948ff9700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9380a2cb0 0x7fa9380a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.061+0000 7fa94affd700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9380a2cb0 0x7fa9380a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.061+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa9440cf470 con 0x7fa94c1008f0 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.061+0000 7fa948ff9700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] conn(0x7fa9380ade10 0x7fa9380b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.061+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 --> [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9380b0980 con 0x7fa9380ade10 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.061+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa9440cf8c0 con 0x7fa94c1008f0 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.061+0000 7fa94bfff700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] conn(0x7fa9380ade10 0x7fa9380b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.062+0000 7fa94affd700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9380a2cb0 0x7fa9380a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa94c104700 tx=0x7fa934009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.062+0000 7fa94bfff700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] conn(0x7fa9380ade10 0x7fa9380b02b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.062+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 <== osd.3 v2:172.21.15.27:6810/2907204232 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9380b0980 con 0x7fa9380ade10 2024-04-25T01:23:59.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.079+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 --> [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa94c101fe0 con 0x7fa9380ade10 2024-04-25T01:23:59.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.079+0000 7fa948ff9700 1 -- 172.21.15.27:0/735153381 <== osd.3 v2:172.21.15.27:6810/2907204232 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa94c101fe0 con 0x7fa9380ade10 2024-04-25T01:23:59.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] conn(0x7fa9380ade10 msgr2=0x7fa9380b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] conn(0x7fa9380ade10 0x7fa9380b02b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9380a2cb0 msgr2=0x7fa9380a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa9380a2cb0 0x7fa9380a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa94c104700 tx=0x7fa934009260 comp rx=0 tx=0).stop 2024-04-25T01:23:59.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c1008f0 msgr2=0x7fa94c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 --2- 172.21.15.27:0/735153381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa94c1008f0 0x7fa94c19e670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa94400ef10 tx=0x7fa94400c3a0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa94bfff700 1 -- 172.21.15.27:0/735153381 reap_dead start 2024-04-25T01:23:59.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 shutdown_connections 2024-04-25T01:23:59.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 >> 172.21.15.27:0/735153381 conn(0x7fa94c0777a0 msgr2=0x7fa94c0fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:59.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 shutdown_connections 2024-04-25T01:23:59.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:23:59.080+0000 7fa951db5700 1 -- 172.21.15.27:0/735153381 wait complete. 2024-04-25T01:23:59.071 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:59 smithi027 ceph-mon[29197]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 295 B/s wr, 6 op/s; 56 B/s, 0 objects/s recovering 2024-04-25T01:23:59.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:23:59 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:59 smithi153 ceph-mon[32621]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 295 B/s wr, 6 op/s; 56 B/s, 0 objects/s recovering 2024-04-25T01:23:59.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:59 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:59 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:59 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:23:59 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:59 smithi139 ceph-mon[32760]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 295 B/s wr, 6 op/s; 56 B/s, 0 objects/s recovering 2024-04-25T01:23:59.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:59 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:59 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:59 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:23:59 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:23:59.768 INFO:teuthology.orchestra.run.smithi027.stdout:287762808867 2024-04-25T01:23:59.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T01:24:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1720493853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101130 msgr2=0x7f5de01015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 --2- 172.21.15.27:0/1720493853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101130 0x7f5de01015b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009990 tx=0x7f5dd4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1720493853 shutdown_connections 2024-04-25T01:24:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 --2- 172.21.15.27:0/1720493853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de0101af0 0x7f5de010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 --2- 172.21.15.27:0/1720493853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5de01071a0 0x7f5de0107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 --2- 172.21.15.27:0/1720493853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101130 0x7f5de01015b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1720493853 >> 172.21.15.27:0/1720493853 conn(0x7f5de00780e0 msgr2=0x7f5de00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1720493853 shutdown_connections 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1720493853 wait complete. 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.675+0000 7f5de4bf4700 1 Processor -- start 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5de4bf4700 1 -- start start 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5de4bf4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de0101130 0x7f5de01030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5de4bf4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101af0 0x7f5de01035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5de4bf4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5de01071a0 0x7f5de0103b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5de4bf4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5de0111440 con 0x7f5de0101af0 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5de4bf4700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01112c0 con 0x7f5de01071a0 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5de4bf4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01115c0 con 0x7f5de0101130 2024-04-25T01:24:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101af0 0x7f5de01035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101af0 0x7f5de01035e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57548/0 (socket says 172.21.15.27:57548) 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 -- 172.21.15.27:0/1282063204 learned_addr learned my addr 172.21.15.27:0/1282063204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dded9d700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5de01071a0 0x7f5de0103b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dde59c700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de0101130 0x7f5de01030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 -- 172.21.15.27:0/1282063204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de0101130 msgr2=0x7f5de01030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de0101130 0x7f5de01030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 -- 172.21.15.27:0/1282063204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5de01071a0 msgr2=0x7f5de0103b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5de01071a0 0x7f5de0103b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 -- 172.21.15.27:0/1282063204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de01a6f00 con 0x7f5de0101af0 2024-04-25T01:24:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.676+0000 7f5dddd9b700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101af0 0x7f5de01035e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4000c00 tx=0x7f5dd40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.677+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4027070 con 0x7f5de0101af0 2024-04-25T01:24:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.677+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd4003780 con 0x7f5de0101af0 2024-04-25T01:24:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.677+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de01a7190 con 0x7f5de0101af0 2024-04-25T01:24:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.677+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4022dd0 con 0x7f5de0101af0 2024-04-25T01:24:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.677+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de01a7640 con 0x7f5de0101af0 2024-04-25T01:24:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.678+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5de00ffd10 con 0x7f5de0101af0 2024-04-25T01:24:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.679+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5dd402a070 con 0x7f5de0101af0 2024-04-25T01:24:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.679+0000 7f5dcf7fe700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5dc80a2d30 0x7f5dc80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.680+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5dd40d0770 con 0x7f5de0101af0 2024-04-25T01:24:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.680+0000 7f5dde59c700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5dc80a2d30 0x7f5dc80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.680+0000 7f5dcf7fe700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] conn(0x7f5dc80ade90 0x7f5dc80b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.680+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 --> [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5dd40043d0 con 0x7f5dc80ade90 2024-04-25T01:24:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.680+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f5dd40d0bc0 con 0x7f5de0101af0 2024-04-25T01:24:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.680+0000 7f5dde59c700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5dc80a2d30 0x7f5dc80a51f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000dad0 tx=0x7f5dd000d4b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.680+0000 7f5dded9d700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] conn(0x7f5dc80ade90 0x7f5dc80b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.680+0000 7f5dded9d700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] conn(0x7f5dc80ade90 0x7f5dc80b0330 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.681+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 <== osd.10 v2:172.21.15.139:6824/155195264 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5dd40043d0 con 0x7f5dc80ade90 2024-04-25T01:24:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.698+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 --> [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5de0102850 con 0x7f5dc80ade90 2024-04-25T01:24:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.698+0000 7f5dcf7fe700 1 -- 172.21.15.27:0/1282063204 <== osd.10 v2:172.21.15.139:6824/155195264 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5de0102850 con 0x7f5dc80ade90 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.698+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 >> [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] conn(0x7f5dc80ade90 msgr2=0x7f5dc80b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] conn(0x7f5dc80ade90 0x7f5dc80b0330 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5dc80a2d30 msgr2=0x7f5dc80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5dc80a2d30 0x7f5dc80a51f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000dad0 tx=0x7f5dd000d4b0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101af0 msgr2=0x7f5de01035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 --2- 172.21.15.27:0/1282063204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de0101af0 0x7f5de01035e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4000c00 tx=0x7f5dd40047b0 comp rx=0 tx=0).stop 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5dded9d700 1 -- 172.21.15.27:0/1282063204 reap_dead start 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 shutdown_connections 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 >> 172.21.15.27:0/1282063204 conn(0x7f5de00780e0 msgr2=0x7f5de0105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 shutdown_connections 2024-04-25T01:24:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.699+0000 7f5de4bf4700 1 -- 172.21.15.27:0/1282063204 wait complete. 2024-04-25T01:24:01.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:01 smithi027 ceph-mon[29197]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-04-25T01:24:01.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:01 smithi153 ceph-mon[32621]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-04-25T01:24:01.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:01 smithi139 ceph-mon[32760]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 -- 172.21.15.27:0/1057069597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 msgr2=0x7f51c410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 --2- 172.21.15.27:0/1057069597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 0x7f51c410dfe0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00b0a0 tx=0x7f51bc00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 -- 172.21.15.27:0/1057069597 shutdown_connections 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 --2- 172.21.15.27:0/1057069597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 0x7f51c410dfe0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 --2- 172.21.15.27:0/1057069597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51c41071a0 0x7f51c4107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 --2- 172.21.15.27:0/1057069597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51c4101130 0x7f51c41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 -- 172.21.15.27:0/1057069597 >> 172.21.15.27:0/1057069597 conn(0x7f51c40780e0 msgr2=0x7f51c40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 -- 172.21.15.27:0/1057069597 shutdown_connections 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.304+0000 7f51c9863700 1 -- 172.21.15.27:0/1057069597 wait complete. 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c9863700 1 Processor -- start 2024-04-25T01:24:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c9863700 1 -- start start 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c9863700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51c4101130 0x7f51c4199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c9863700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 0x7f51c4199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c9863700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51c41071a0 0x7f51c41a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c9863700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4111490 con 0x7f51c4101af0 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c9863700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4111310 con 0x7f51c41071a0 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c9863700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4111610 con 0x7f51c4101130 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 0x7f51c4199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 0x7f51c4199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57574/0 (socket says 172.21.15.27:57574) 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c27fc700 1 -- 172.21.15.27:0/2331405813 learned_addr learned my addr 172.21.15.27:0/2331405813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c2ffd700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51c4101130 0x7f51c4199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.305+0000 7f51c37fe700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51c41071a0 0x7f51c41a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c27fc700 1 -- 172.21.15.27:0/2331405813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51c4101130 msgr2=0x7f51c4199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:01.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c27fc700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51c4101130 0x7f51c4199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c27fc700 1 -- 172.21.15.27:0/2331405813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51c41071a0 msgr2=0x7f51c41a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c27fc700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51c41071a0 0x7f51c41a48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c27fc700 1 -- 172.21.15.27:0/2331405813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c41a4e30 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c27fc700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 0x7f51c4199db0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f51b800e9c0 tx=0x7f51b800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b800cbd0 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b800ee60 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c41a5000 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8005630 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.306+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c41a54c0 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.307+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f51c4062900 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.308+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f51b8004750 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c8861700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f51ac0a2cb0 0x7f51ac0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f51b80cfc20 con 0x7f51c4101af0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c2ffd700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f51ac0a2cb0 0x7f51ac0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c8861700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] conn(0x7f51ac0ade10 0x7f51ac0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:01.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 --> [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51b801ee50 con 0x7f51ac0ade10 2024-04-25T01:24:01.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f51b80d0070 con 0x7f51c4101af0 2024-04-25T01:24:01.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c37fe700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] conn(0x7f51ac0ade10 0x7f51ac0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:01.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c2ffd700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f51ac0a2cb0 0x7f51ac0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f51b4007b60 tx=0x7f51b40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:01.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.309+0000 7f51c37fe700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] conn(0x7f51ac0ade10 0x7f51ac0b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:01.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.310+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 <== osd.6 v2:172.21.15.27:6818/1607133562 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f51b801ee50 con 0x7f51ac0ade10 2024-04-25T01:24:01.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.326+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 --> [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f51c4062d80 con 0x7f51ac0ade10 2024-04-25T01:24:01.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c8861700 1 -- 172.21.15.27:0/2331405813 <== osd.6 v2:172.21.15.27:6818/1607133562 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f51c4062d80 con 0x7f51ac0ade10 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] conn(0x7f51ac0ade10 msgr2=0x7f51ac0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] conn(0x7f51ac0ade10 0x7f51ac0b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f51ac0a2cb0 msgr2=0x7f51ac0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f51ac0a2cb0 0x7f51ac0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f51b4007b60 tx=0x7f51b40073d0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 msgr2=0x7f51c4199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 --2- 172.21.15.27:0/2331405813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51c4101af0 0x7f51c4199db0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f51b800e9c0 tx=0x7f51b800ecd0 comp rx=0 tx=0).stop 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c37fe700 1 -- 172.21.15.27:0/2331405813 reap_dead start 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 shutdown_connections 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 >> 172.21.15.27:0/2331405813 conn(0x7f51c40780e0 msgr2=0x7f51c40ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 shutdown_connections 2024-04-25T01:24:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:01.327+0000 7f51c9863700 1 -- 172.21.15.27:0/2331405813 wait complete. 2024-04-25T01:24:02.134 INFO:teuthology.orchestra.run.smithi027.stdout:197568495669 2024-04-25T01:24:02.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T01:24:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.210+0000 7f96748a2700 1 -- 172.21.15.27:0/2300943904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 msgr2=0x7f96701028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.210+0000 7f96748a2700 1 --2- 172.21.15.27:0/2300943904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 0x7f96701028c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9660009990 tx=0x7f9660009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.210+0000 7f96748a2700 1 -- 172.21.15.27:0/2300943904 shutdown_connections 2024-04-25T01:24:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.210+0000 7f96748a2700 1 --2- 172.21.15.27:0/2300943904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9670107570 0x7f967010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.211+0000 7f96748a2700 1 --2- 172.21.15.27:0/2300943904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 0x7f96701028c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.211+0000 7f96748a2700 1 --2- 172.21.15.27:0/2300943904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670102e90 0x7f9670106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.211+0000 7f96748a2700 1 -- 172.21.15.27:0/2300943904 >> 172.21.15.27:0/2300943904 conn(0x7f9670079390 msgr2=0x7f96700797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.211+0000 7f96748a2700 1 -- 172.21.15.27:0/2300943904 shutdown_connections 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.211+0000 7f96748a2700 1 -- 172.21.15.27:0/2300943904 wait complete. 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f96748a2700 1 Processor -- start 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f96748a2700 1 -- start start 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f96748a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 0x7f967019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f96748a2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9670102e90 0x7f967019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f96748a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670107570 0x7f9670198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f96748a2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9670112530 con 0x7f96701024e0 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f96748a2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f96701123b0 con 0x7f9670102e90 2024-04-25T01:24:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f96748a2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96701126b0 con 0x7f9670107570 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 0x7f967019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 0x7f967019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57536/0 (socket says 172.21.15.27:57536) 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966e59c700 1 -- 172.21.15.27:0/3581576745 learned_addr learned my addr 172.21.15.27:0/3581576745 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966dd9b700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9670102e90 0x7f967019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966ed9d700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670107570 0x7f9670198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966e59c700 1 -- 172.21.15.27:0/3581576745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670107570 msgr2=0x7f9670198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966e59c700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670107570 0x7f9670198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966e59c700 1 -- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9670102e90 msgr2=0x7f967019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966e59c700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9670102e90 0x7f967019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966e59c700 1 -- 172.21.15.27:0/3581576745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9670198f70 con 0x7f96701024e0 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966dd9b700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9670102e90 0x7f967019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.212+0000 7f966ed9d700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9670107570 0x7f9670198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.213+0000 7f966e59c700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 0x7f967019e610 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9660000c00 tx=0x7f9660003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.213+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660027070 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.213+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9660004190 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.213+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9670199200 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.213+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966000f460 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.213+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96701a5470 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.214+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f96700629c0 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.215+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f966002a070 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.215+0000 7f965f7fe700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f96580a2cb0 0x7f96580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.216+0000 7f966dd9b700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f96580a2cb0 0x7f96580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.216+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f96600d0b60 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.216+0000 7f965f7fe700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] conn(0x7f96580ade10 0x7f96580b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.216+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 --> [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9660001d30 con 0x7f96580ade10 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.216+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f96600d0fb0 con 0x7f96701024e0 2024-04-25T01:24:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.216+0000 7f966dd9b700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f96580a2cb0 0x7f96580a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9670108a30 tx=0x7f9664005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.216+0000 7f966ed9d700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] conn(0x7f96580ade10 0x7f96580b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.217+0000 7f966ed9d700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] conn(0x7f96580ade10 0x7f96580b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.218+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 <== osd.1 v2:172.21.15.139:6800/4036947883 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9660001d30 con 0x7f96580ade10 2024-04-25T01:24:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.234+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 --> [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9670062dc0 con 0x7f96580ade10 2024-04-25T01:24:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.234+0000 7f965f7fe700 1 -- 172.21.15.27:0/3581576745 <== osd.1 v2:172.21.15.139:6800/4036947883 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9670062dc0 con 0x7f96580ade10 2024-04-25T01:24:02.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.234+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] conn(0x7f96580ade10 msgr2=0x7f96580b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.234+0000 7f96748a2700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] conn(0x7f96580ade10 0x7f96580b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f96580a2cb0 msgr2=0x7f96580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f96748a2700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f96580a2cb0 0x7f96580a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9670108a30 tx=0x7f9664005f50 comp rx=0 tx=0).stop 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 msgr2=0x7f967019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f96748a2700 1 --2- 172.21.15.27:0/3581576745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96701024e0 0x7f967019e610 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9660000c00 tx=0x7f9660003730 comp rx=0 tx=0).stop 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f966ed9d700 1 -- 172.21.15.27:0/3581576745 reap_dead start 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 shutdown_connections 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 >> 172.21.15.27:0/3581576745 conn(0x7f9670079390 msgr2=0x7f9670101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 shutdown_connections 2024-04-25T01:24:02.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.235+0000 7f96748a2700 1 -- 172.21.15.27:0/3581576745 wait complete. 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3182406181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb81071a0 msgr2=0x7fcfb8107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 --2- 172.21.15.27:0/3182406181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb81071a0 0x7fcfb8107580 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009990 tx=0x7fcfa8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3182406181 shutdown_connections 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 --2- 172.21.15.27:0/3182406181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfb8101af0 0x7fcfb810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 --2- 172.21.15.27:0/3182406181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb81071a0 0x7fcfb8107580 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 --2- 172.21.15.27:0/3182406181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb8101130 0x7fcfb81015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3182406181 >> 172.21.15.27:0/3182406181 conn(0x7fcfb80780e0 msgr2=0x7fcfb80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3182406181 shutdown_connections 2024-04-25T01:24:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3182406181 wait complete. 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.091+0000 7fcfbdd84700 1 Processor -- start 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfbdd84700 1 -- start start 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfbdd84700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb8101130 0x7fcfb819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfbdd84700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb8101af0 0x7fcfb819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfbdd84700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfb81071a0 0x7fcfb8198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfbdd84700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb8111440 con 0x7fcfb8101130 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfbdd84700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81112c0 con 0x7fcfb81071a0 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfbdd84700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81115c0 con 0x7fcfb8101af0 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfb77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb8101130 0x7fcfb819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfb7fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfb81071a0 0x7fcfb8198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfb77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb8101130 0x7fcfb819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57518/0 (socket says 172.21.15.27:57518) 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfb77fe700 1 -- 172.21.15.27:0/3709883104 learned_addr learned my addr 172.21.15.27:0/3709883104 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.092+0000 7fcfb6ffd700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb8101af0 0x7fcfb819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.093+0000 7fcfb7fff700 1 -- 172.21.15.27:0/3709883104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb8101af0 msgr2=0x7fcfb819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.093+0000 7fcfb7fff700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb8101af0 0x7fcfb819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.093+0000 7fcfb7fff700 1 -- 172.21.15.27:0/3709883104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb8101130 msgr2=0x7fcfb819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.093+0000 7fcfb7fff700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb8101130 0x7fcfb819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.093+0000 7fcfb7fff700 1 -- 172.21.15.27:0/3709883104 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb8199010 con 0x7fcfb81071a0 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.093+0000 7fcfb7fff700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfb81071a0 0x7fcfb8198830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0007f10 tx=0x7fcfb000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.093+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0004db0 con 0x7fcfb81071a0 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.094+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0003a00 con 0x7fcfb81071a0 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.094+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb8199300 con 0x7fcfb81071a0 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.094+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0004680 con 0x7fcfb81071a0 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.094+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb8102d40 con 0x7fcfb81071a0 2024-04-25T01:24:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.095+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfa4001030 con 0x7fcfb81071a0 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.101+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcfb001c020 con 0x7fcfb81071a0 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.102+0000 7fcfb4ff9700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcfa00a2cb0 0x7fcfa00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.102+0000 7fcfb77fe700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcfa00a2cb0 0x7fcfa00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.102+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcfb00cef30 con 0x7fcfb81071a0 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.102+0000 7fcfb4ff9700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] conn(0x7fcfa00ade10 0x7fcfa00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.102+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 --> [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfa00b0980 con 0x7fcfa00ade10 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.102+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fcfb00cf380 con 0x7fcfb81071a0 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.102+0000 7fcfb77fe700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcfa00a2cb0 0x7fcfa00a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800b5c0 tx=0x7fcfa8005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.103+0000 7fcfb6ffd700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] conn(0x7fcfa00ade10 0x7fcfa00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.103+0000 7fcfb6ffd700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] conn(0x7fcfa00ade10 0x7fcfa00b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.104+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 <== osd.2 v2:172.21.15.153:6800/3400401807 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfa00b0980 con 0x7fcfa00ade10 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.120+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 --> [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcfa4002e60 con 0x7fcfa00ade10 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.121+0000 7fcfb4ff9700 1 -- 172.21.15.27:0/3709883104 <== osd.2 v2:172.21.15.153:6800/3400401807 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcfa4002e60 con 0x7fcfa00ade10 2024-04-25T01:24:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.121+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 >> [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] conn(0x7fcfa00ade10 msgr2=0x7fcfa00b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.121+0000 7fcfbdd84700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] conn(0x7fcfa00ade10 0x7fcfa00b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcfa00a2cb0 msgr2=0x7fcfa00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfbdd84700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcfa00a2cb0 0x7fcfa00a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800b5c0 tx=0x7fcfa8005be0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfb81071a0 msgr2=0x7fcfb8198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfbdd84700 1 --2- 172.21.15.27:0/3709883104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfb81071a0 0x7fcfb8198830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0007f10 tx=0x7fcfb000d3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfb7fff700 1 -- 172.21.15.27:0/3709883104 reap_dead start 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 shutdown_connections 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 >> 172.21.15.27:0/3709883104 conn(0x7fcfb80780e0 msgr2=0x7fcfb8100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 shutdown_connections 2024-04-25T01:24:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:00.122+0000 7fcfbdd84700 1 -- 172.21.15.27:0/3709883104 wait complete. 2024-04-25T01:24:02.927 INFO:teuthology.orchestra.run.smithi027.stdout:77309411401 2024-04-25T01:24:02.928 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T01:24:03.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:03 smithi027 ceph-mon[29197]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:24:03.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:03 smithi153 ceph-mon[32621]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:24:03.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:03 smithi139 ceph-mon[32760]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:24:03.890 INFO:teuthology.orchestra.run.smithi027.stdout:253403070505 2024-04-25T01:24:03.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T01:24:04.309 INFO:teuthology.orchestra.run.smithi027.stdout:107374182468 2024-04-25T01:24:04.309 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T01:24:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 -- 172.21.15.27:0/2825546621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc1024e0 msgr2=0x7f71bc1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 --2- 172.21.15.27:0/2825546621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc1024e0 0x7f71bc1028c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009990 tx=0x7f71ac009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 -- 172.21.15.27:0/2825546621 shutdown_connections 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 --2- 172.21.15.27:0/2825546621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71bc1075a0 0x7f71bc10f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 --2- 172.21.15.27:0/2825546621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc1024e0 0x7f71bc1028c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 --2- 172.21.15.27:0/2825546621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71bc102e90 0x7f71bc106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 -- 172.21.15.27:0/2825546621 >> 172.21.15.27:0/2825546621 conn(0x7f71bc0793e0 msgr2=0x7f71bc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 -- 172.21.15.27:0/2825546621 shutdown_connections 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.533+0000 7f71c1d76700 1 -- 172.21.15.27:0/2825546621 wait complete. 2024-04-25T01:24:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71c1d76700 1 Processor -- start 2024-04-25T01:24:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71c1d76700 1 -- start start 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71c1d76700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71bc1024e0 0x7f71bc074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71c1d76700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc102e90 0x7f71bc074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71c1d76700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71bc1075a0 0x7f71bc074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71c1d76700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1125b0 con 0x7f71bc102e90 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71c1d76700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc112430 con 0x7f71bc1075a0 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71c1d76700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc112730 con 0x7f71bc1024e0 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71baffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc102e90 0x7f71bc074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71baffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc102e90 0x7f71bc074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57590/0 (socket says 172.21.15.27:57590) 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.534+0000 7f71baffd700 1 -- 172.21.15.27:0/2549753790 learned_addr learned my addr 172.21.15.27:0/2549753790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71bb7fe700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71bc1024e0 0x7f71bc074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71bbfff700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71bc1075a0 0x7f71bc074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71baffd700 1 -- 172.21.15.27:0/2549753790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71bc1024e0 msgr2=0x7f71bc074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71baffd700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71bc1024e0 0x7f71bc074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71baffd700 1 -- 172.21.15.27:0/2549753790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71bc1075a0 msgr2=0x7f71bc074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71baffd700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71bc1075a0 0x7f71bc074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71baffd700 1 -- 172.21.15.27:0/2549753790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71bc1a6f90 con 0x7f71bc102e90 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71baffd700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc102e90 0x7f71bc074a70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f71b000e970 tx=0x7f71b000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0004a30 con 0x7f71bc102e90 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b000ccc0 con 0x7f71bc102e90 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0005650 con 0x7f71bc102e90 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71bc1a7280 con 0x7f71bc102e90 2024-04-25T01:24:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.535+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71bc1a7760 con 0x7f71bc102e90 2024-04-25T01:24:04.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.536+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f71bc062940 con 0x7f71bc102e90 2024-04-25T01:24:04.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.538+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f71b0004bd0 con 0x7f71bc102e90 2024-04-25T01:24:04.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.538+0000 7f71b8ff9700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f71a40a2d30 0x7f71a40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:04.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.538+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f71b00cfb30 con 0x7f71bc102e90 2024-04-25T01:24:04.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.538+0000 7f71bb7fe700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f71a40a2d30 0x7f71a40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:04.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.538+0000 7f71b8ff9700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] conn(0x7f71a40ade90 0x7f71a40b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:04.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.539+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 --> [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71a40b0a00 con 0x7f71a40ade90 2024-04-25T01:24:04.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.539+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f71b00cfee0 con 0x7f71bc102e90 2024-04-25T01:24:04.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.539+0000 7f71bb7fe700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f71a40a2d30 0x7f71a40a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00b2e0 tx=0x7f71ac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:04.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.539+0000 7f71bbfff700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] conn(0x7f71a40ade90 0x7f71a40b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:04.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.539+0000 7f71bbfff700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] conn(0x7f71a40ade90 0x7f71a40b0330 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:04.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.540+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 <== osd.7 v2:172.21.15.139:6816/3705167644 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f71a40b0a00 con 0x7f71a40ade90 2024-04-25T01:24:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.566+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 --> [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f71bc062dc0 con 0x7f71a40ade90 2024-04-25T01:24:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.566+0000 7f71b8ff9700 1 -- 172.21.15.27:0/2549753790 <== osd.7 v2:172.21.15.139:6816/3705167644 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f71bc062dc0 con 0x7f71a40ade90 2024-04-25T01:24:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 >> [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] conn(0x7f71a40ade90 msgr2=0x7f71a40b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71c1d76700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] conn(0x7f71a40ade90 0x7f71a40b0330 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f71a40a2d30 msgr2=0x7f71a40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71c1d76700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f71a40a2d30 0x7f71a40a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00b2e0 tx=0x7f71ac005fd0 comp rx=0 tx=0).stop 2024-04-25T01:24:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc102e90 msgr2=0x7f71bc074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71c1d76700 1 --2- 172.21.15.27:0/2549753790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71bc102e90 0x7f71bc074a70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f71b000e970 tx=0x7f71b000ec80 comp rx=0 tx=0).stop 2024-04-25T01:24:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71bbfff700 1 -- 172.21.15.27:0/2549753790 reap_dead start 2024-04-25T01:24:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 shutdown_connections 2024-04-25T01:24:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.567+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 >> 172.21.15.27:0/2549753790 conn(0x7f71bc0793e0 msgr2=0x7f71bc101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.568+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 shutdown_connections 2024-04-25T01:24:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:04.568+0000 7f71c1d76700 1 -- 172.21.15.27:0/2549753790 wait complete. 2024-04-25T01:24:05.181 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:05 smithi027 ceph-mon[29197]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:24:05.210 INFO:teuthology.orchestra.run.smithi027.stdout:309237645344 2024-04-25T01:24:05.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T01:24:05.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:05 smithi153 ceph-mon[32621]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:24:05.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:05 smithi139 ceph-mon[32760]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:24:05.878 INFO:teuthology.orchestra.run.smithi027.stdout:219043332144 2024-04-25T01:24:05.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 -- 172.21.15.27:0/4214669177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 msgr2=0x7f7074106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 --2- 172.21.15.27:0/4214669177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 0x7f7074106f00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f70680099c0 tx=0x7f7068009cd0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 -- 172.21.15.27:0/4214669177 shutdown_connections 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 --2- 172.21.15.27:0/4214669177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70741075e0 0x7f707410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 --2- 172.21.15.27:0/4214669177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7074102510 0x7f70741028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 --2- 172.21.15.27:0/4214669177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 0x7f7074106f00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 -- 172.21.15.27:0/4214669177 >> 172.21.15.27:0/4214669177 conn(0x7f70740793e0 msgr2=0x7f70740797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 -- 172.21.15.27:0/4214669177 shutdown_connections 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.346+0000 7f707c7fe700 1 -- 172.21.15.27:0/4214669177 wait complete. 2024-04-25T01:24:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707c7fe700 1 Processor -- start 2024-04-25T01:24:06.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707c7fe700 1 -- start start 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707c7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7074102510 0x7f7074074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707c7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 0x7f7074074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707c7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70741075e0 0x7f7074074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707c7fe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f70741125a0 con 0x7f7074102ec0 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707c7fe700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7074112420 con 0x7f70741075e0 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707c7fe700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7074112720 con 0x7f7074102510 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f7079d99700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 0x7f7074074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f7079d99700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 0x7f7074074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57604/0 (socket says 172.21.15.27:57604) 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f7079d99700 1 -- 172.21.15.27:0/2455268036 learned_addr learned my addr 172.21.15.27:0/2455268036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.347+0000 7f707a59a700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7074102510 0x7f7074074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f707ad9b700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70741075e0 0x7f7074074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f7079d99700 1 -- 172.21.15.27:0/2455268036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7074102510 msgr2=0x7f7074074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f7079d99700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7074102510 0x7f7074074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f7079d99700 1 -- 172.21.15.27:0/2455268036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70741075e0 msgr2=0x7f7074074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f7079d99700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70741075e0 0x7f7074074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f7079d99700 1 -- 172.21.15.27:0/2455268036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70741a6f90 con 0x7f7074102ec0 2024-04-25T01:24:06.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f707ad9b700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70741075e0 0x7f7074074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f7079d99700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 0x7f7074074a70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f706800fa00 tx=0x7f706800fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068027070 con 0x7f7074102ec0 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7068003680 con 0x7f7074102ec0 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70741a7220 con 0x7f7074102ec0 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068017650 con 0x7f7074102ec0 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.348+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70741a7670 con 0x7f7074102ec0 2024-04-25T01:24:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.349+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7074062940 con 0x7f7074102ec0 2024-04-25T01:24:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.350+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f706802a070 con 0x7f7074102ec0 2024-04-25T01:24:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.351+0000 7f70737fe700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f705c0a2cb0 0x7f705c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.351+0000 7f707a59a700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f705c0a2cb0 0x7f705c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.351+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f70680d0cd0 con 0x7f7074102ec0 2024-04-25T01:24:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.351+0000 7f70737fe700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] conn(0x7f705c0ade10 0x7f705c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.351+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 --> [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f705c0b0980 con 0x7f705c0ade10 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.351+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f70680d1120 con 0x7f7074102ec0 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.351+0000 7f707a59a700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f705c0a2cb0 0x7f705c0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f706c009d10 tx=0x7f706c009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.351+0000 7f707ad9b700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] conn(0x7f705c0ade10 0x7f705c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.352+0000 7f707ad9b700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] conn(0x7f705c0ade10 0x7f705c0b02b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.353+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 <== osd.5 v2:172.21.15.153:6808/4140079274 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f705c0b0980 con 0x7f705c0ade10 2024-04-25T01:24:06.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.369+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 --> [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7074062d40 con 0x7f705c0ade10 2024-04-25T01:24:06.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.370+0000 7f70737fe700 1 -- 172.21.15.27:0/2455268036 <== osd.5 v2:172.21.15.153:6808/4140079274 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7074062d40 con 0x7f705c0ade10 2024-04-25T01:24:06.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.370+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 >> [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] conn(0x7f705c0ade10 msgr2=0x7f705c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.370+0000 7f707c7fe700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] conn(0x7f705c0ade10 0x7f705c0b02b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.370+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f705c0a2cb0 msgr2=0x7f705c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.370+0000 7f707c7fe700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f705c0a2cb0 0x7f705c0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f706c009d10 tx=0x7f706c009040 comp rx=0 tx=0).stop 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.370+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 msgr2=0x7f7074074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.370+0000 7f707c7fe700 1 --2- 172.21.15.27:0/2455268036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7074102ec0 0x7f7074074a70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f706800fa00 tx=0x7f706800fae0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.370+0000 7f707ad9b700 1 -- 172.21.15.27:0/2455268036 reap_dead start 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.371+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 shutdown_connections 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.371+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 >> 172.21.15.27:0/2455268036 conn(0x7f70740793e0 msgr2=0x7f7074100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.371+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 shutdown_connections 2024-04-25T01:24:06.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:06.371+0000 7f707c7fe700 1 -- 172.21.15.27:0/2455268036 wait complete. 2024-04-25T01:24:07.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:07 smithi027 ceph-mon[29197]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:24:07.246 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:07.251 INFO:teuthology.orchestra.run.smithi027.stdout:55834574925 2024-04-25T01:24:07.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T01:24:07.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:07 smithi153 ceph-mon[32621]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:24:07.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:07 smithi139 ceph-mon[32760]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:24:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:09.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:09 smithi153 ceph-mon[32621]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:09.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:09 smithi139 ceph-mon[32760]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:09.308 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:09 smithi027 ceph-mon[29197]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:11.109 INFO:teuthology.orchestra.run.smithi027.stdout:163208757306 2024-04-25T01:24:11.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T01:24:11.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:11 smithi153 ceph-mon[32621]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:11.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:11 smithi139 ceph-mon[32760]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:11.316 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:11 smithi027 ceph-mon[29197]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 -- 172.21.15.27:0/241904566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102550 msgr2=0x7f74c0102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/241904566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102550 0x7f74c0102930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f74b00099c0 tx=0x7f74b0009cd0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 -- 172.21.15.27:0/241904566 shutdown_connections 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/241904566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74c0107540 0x7f74c010f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/241904566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102550 0x7f74c0102930 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/241904566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c0102f00 0x7f74c0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 -- 172.21.15.27:0/241904566 >> 172.21.15.27:0/241904566 conn(0x7f74c00793f0 msgr2=0x7f74c0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 -- 172.21.15.27:0/241904566 shutdown_connections 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.459+0000 7f74c6c8f700 1 -- 172.21.15.27:0/241904566 wait complete. 2024-04-25T01:24:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.460+0000 7f74c6c8f700 1 Processor -- start 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.460+0000 7f74c6c8f700 1 -- start start 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.460+0000 7f74c6c8f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c0102550 0x7f74c0074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.460+0000 7f74c6c8f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102f00 0x7f74c0074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.460+0000 7f74c6c8f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74c0107540 0x7f74c0074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.460+0000 7f74c6c8f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f74c0112540 con 0x7f74c0102f00 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.460+0000 7f74c6c8f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01123c0 con 0x7f74c0107540 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.460+0000 7f74c6c8f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74c01126c0 con 0x7f74c0102550 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102f00 0x7f74c0074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102f00 0x7f74c0074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45756/0 (socket says 172.21.15.27:45756) 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 -- 172.21.15.27:0/4197053156 learned_addr learned my addr 172.21.15.27:0/4197053156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74c522c700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74c0107540 0x7f74c0074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74c4a2b700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c0102550 0x7f74c0074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 -- 172.21.15.27:0/4197053156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c0102550 msgr2=0x7f74c0074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c0102550 0x7f74c0074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 -- 172.21.15.27:0/4197053156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74c0107540 msgr2=0x7f74c0074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74c0107540 0x7f74c0074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 -- 172.21.15.27:0/4197053156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c01a6f30 con 0x7f74c0102f00 2024-04-25T01:24:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bffff700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102f00 0x7f74c0074a70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f74b400d950 tx=0x7f74b400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:12.866 INFO:teuthology.orchestra.run.smithi027.stdout:309237645346 2024-04-25T01:24:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bdffb700 1 -- 172.21.15.27:0/4197053156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4017070 con 0x7f74c0102f00 2024-04-25T01:24:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c01a7220 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bdffb700 1 -- 172.21.15.27:0/4197053156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b4005020 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.461+0000 7f74bdffb700 1 -- 172.21.15.27:0/4197053156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b400fbc0 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.462+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c01a7710 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.462+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c0068f20 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.463+0000 7f74bdffb700 1 -- 172.21.15.27:0/4197053156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f74b400fd60 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.464+0000 7f74bdffb700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f74a80a2cb0 0x7f74a80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.464+0000 7f74bdffb700 1 -- 172.21.15.27:0/4197053156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74b40cecf0 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.464+0000 7f74c4a2b700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f74a80a2cb0 0x7f74a80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.464+0000 7f74c4a2b700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f74a80a2cb0 0x7f74a80a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009f50 tx=0x7f74b0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.469+0000 7f74bdffb700 1 -- 172.21.15.27:0/4197053156 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f74b409b790 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.686+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f74c002db30 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.687+0000 7f74bdffb700 1 -- 172.21.15.27:0/4197053156 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f74b409b090 con 0x7f74c0102f00 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f74a80a2cb0 msgr2=0x7f74a80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f74a80a2cb0 0x7f74a80a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009f50 tx=0x7f74b0005fb0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102f00 msgr2=0x7f74c0074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102f00 0x7f74c0074a70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f74b400d950 tx=0x7f74b400dc60 comp rx=0 tx=0).stop 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 shutdown_connections 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f74a80a2cb0 0x7f74a80a5170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74c0107540 0x7f74c0074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c0102550 0x7f74c0074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 --2- 172.21.15.27:0/4197053156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74c0102f00 0x7f74c0074a70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 >> 172.21.15.27:0/4197053156 conn(0x7f74c00793f0 msgr2=0x7f74c01066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 shutdown_connections 2024-04-25T01:24:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:12.689+0000 7f74c6c8f700 1 -- 172.21.15.27:0/4197053156 wait complete. 2024-04-25T01:24:13.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:13 smithi027 ceph-mon[29197]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:13.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:13 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4197053156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:24:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:13 smithi153 ceph-mon[32621]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:13.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:13 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4197053156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:24:13.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:13 smithi139 ceph-mon[32760]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:13.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:13 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4197053156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:24:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 -- 172.21.15.27:0/2718940493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101af0 msgr2=0x7f584810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 --2- 172.21.15.27:0/2718940493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101af0 0x7f584810dfe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5838009990 tx=0x7f5838009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 -- 172.21.15.27:0/2718940493 shutdown_connections 2024-04-25T01:24:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 --2- 172.21.15.27:0/2718940493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101af0 0x7f584810dfe0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 --2- 172.21.15.27:0/2718940493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58481071a0 0x7f5848107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 --2- 172.21.15.27:0/2718940493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5848101130 0x7f58481015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 -- 172.21.15.27:0/2718940493 >> 172.21.15.27:0/2718940493 conn(0x7f58480780e0 msgr2=0x7f58480784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 -- 172.21.15.27:0/2718940493 shutdown_connections 2024-04-25T01:24:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.324+0000 7f584d633700 1 -- 172.21.15.27:0/2718940493 wait complete. 2024-04-25T01:24:13.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f584d633700 1 Processor -- start 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f584d633700 1 -- start start 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f584d633700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101130 0x7f58481997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f584d633700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5848101af0 0x7f5848199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f584d633700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58481071a0 0x7f58481a4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f584d633700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5848111410 con 0x7f5848101130 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f584d633700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5848111290 con 0x7f58481071a0 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f584d633700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5848111590 con 0x7f5848101af0 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f5846ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101130 0x7f58481997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f5846ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101130 0x7f58481997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45782/0 (socket says 172.21.15.27:45782) 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.325+0000 7f5846ffd700 1 -- 172.21.15.27:0/3717673961 learned_addr learned my addr 172.21.15.27:0/3717673961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f58477fe700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58481071a0 0x7f58481a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f58467fc700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5848101af0 0x7f5848199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f5846ffd700 1 -- 172.21.15.27:0/3717673961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5848101af0 msgr2=0x7f5848199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f5846ffd700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5848101af0 0x7f5848199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f5846ffd700 1 -- 172.21.15.27:0/3717673961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58481071a0 msgr2=0x7f58481a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f5846ffd700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58481071a0 0x7f58481a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f5846ffd700 1 -- 172.21.15.27:0/3717673961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58481a4db0 con 0x7f5848101130 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f5846ffd700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101130 0x7f58481997f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f58400060b0 tx=0x7f584000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f583ffff700 1 -- 172.21.15.27:0/3717673961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584001a070 con 0x7f5848101130 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58481a4f80 con 0x7f5848101130 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f583ffff700 1 -- 172.21.15.27:0/3717673961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f584000ddd0 con 0x7f5848101130 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f583ffff700 1 -- 172.21.15.27:0/3717673961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58400147a0 con 0x7f5848101130 2024-04-25T01:24:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.326+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58481a5430 con 0x7f5848101130 2024-04-25T01:24:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.327+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5848068f20 con 0x7f5848101130 2024-04-25T01:24:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.328+0000 7f583ffff700 1 -- 172.21.15.27:0/3717673961 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f584001d070 con 0x7f5848101130 2024-04-25T01:24:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.329+0000 7f583ffff700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f58340a2cb0 0x7f58340a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.329+0000 7f58467fc700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f58340a2cb0 0x7f58340a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.329+0000 7f583ffff700 1 -- 172.21.15.27:0/3717673961 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f58400ced50 con 0x7f5848101130 2024-04-25T01:24:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.329+0000 7f58467fc700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f58340a2cb0 0x7f58340a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5830005950 tx=0x7f58300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:13.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.333+0000 7f583ffff700 1 -- 172.21.15.27:0/3717673961 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5840010070 con 0x7f5848101130 2024-04-25T01:24:13.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.549+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f584802db30 con 0x7f5848101130 2024-04-25T01:24:13.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.550+0000 7f583ffff700 1 -- 172.21.15.27:0/3717673961 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f584009b7f0 con 0x7f5848101130 2024-04-25T01:24:13.531 INFO:teuthology.orchestra.run.smithi027.stdout:34359738452 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f58340a2cb0 msgr2=0x7f58340a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f58340a2cb0 0x7f58340a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5830005950 tx=0x7f58300058e0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101130 msgr2=0x7f58481997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101130 0x7f58481997f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f58400060b0 tx=0x7f584000d3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 shutdown_connections 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f58340a2cb0 0x7f58340a5170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58481071a0 0x7f58481a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5848101130 0x7f58481997f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 --2- 172.21.15.27:0/3717673961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5848101af0 0x7f5848199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 >> 172.21.15.27:0/3717673961 conn(0x7f58480780e0 msgr2=0x7f58480ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 shutdown_connections 2024-04-25T01:24:13.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:13.552+0000 7f584d633700 1 -- 172.21.15.27:0/3717673961 wait complete. 2024-04-25T01:24:13.830 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645344 got 309237645346 for osd.10 2024-04-25T01:24:13.830 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:13.856 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:14.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:14 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3717673961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:24:14.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:14 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3717673961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:24:14.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:14 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3717673961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:24:14.908 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:14.909 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:14.909 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:14.910 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:15.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:15 smithi027 ceph-mon[29197]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 5 op/s 2024-04-25T01:24:15.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:15 smithi139 ceph-mon[32760]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 5 op/s 2024-04-25T01:24:15.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738448 got 34359738452 for osd.0 2024-04-25T01:24:15.306 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:15.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:15 smithi153 ceph-mon[32621]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 5 op/s 2024-04-25T01:24:17.251 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:17 smithi027 ceph-mon[29197]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-25T01:24:17.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:17 smithi139 ceph-mon[32760]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-25T01:24:17.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:17 smithi153 ceph-mon[32621]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-25T01:24:17.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.932+0000 7fb185dcb700 1 -- 172.21.15.27:0/4134003540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180102e90 msgr2=0x7fb180106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.932+0000 7fb185dcb700 1 --2- 172.21.15.27:0/4134003540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180102e90 0x7fb180106e90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb174009990 tx=0x7fb174009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.932+0000 7fb185dcb700 1 -- 172.21.15.27:0/4134003540 shutdown_connections 2024-04-25T01:24:17.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.932+0000 7fb185dcb700 1 --2- 172.21.15.27:0/4134003540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb180107570 0x7fb18010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.932+0000 7fb185dcb700 1 --2- 172.21.15.27:0/4134003540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1801024e0 0x7fb1801028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.932+0000 7fb185dcb700 1 --2- 172.21.15.27:0/4134003540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180102e90 0x7fb180106e90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.932+0000 7fb185dcb700 1 -- 172.21.15.27:0/4134003540 >> 172.21.15.27:0/4134003540 conn(0x7fb180079390 msgr2=0x7fb1800797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:17.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.932+0000 7fb185dcb700 1 -- 172.21.15.27:0/4134003540 shutdown_connections 2024-04-25T01:24:17.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.933+0000 7fb185dcb700 1 -- 172.21.15.27:0/4134003540 wait complete. 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.933+0000 7fb185dcb700 1 Processor -- start 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.933+0000 7fb185dcb700 1 -- start start 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb185dcb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1801024e0 0x7fb180074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb185dcb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb180102e90 0x7fb180074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb185dcb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180107570 0x7fb180074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb185dcb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb180112510 con 0x7fb180107570 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb185dcb700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb180112390 con 0x7fb180102e90 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb185dcb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb180112690 con 0x7fb1801024e0 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180107570 0x7fb180074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180107570 0x7fb180074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43760/0 (socket says 172.21.15.27:43760) 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 -- 172.21.15.27:0/1846033392 learned_addr learned my addr 172.21.15.27:0/1846033392 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17effd700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb180102e90 0x7fb180074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17f7fe700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1801024e0 0x7fb180074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 -- 172.21.15.27:0/1846033392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1801024e0 msgr2=0x7fb180074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1801024e0 0x7fb180074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 -- 172.21.15.27:0/1846033392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb180102e90 msgr2=0x7fb180074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb180102e90 0x7fb180074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 -- 172.21.15.27:0/1846033392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1801a6f50 con 0x7fb180107570 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17ffff700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180107570 0x7fb180074fb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb178007ee0 tx=0x7fb17800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17cff9700 1 -- 172.21.15.27:0/1846033392 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17800dbd0 con 0x7fb180107570 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb17cff9700 1 -- 172.21.15.27:0/1846033392 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb17800dd70 con 0x7fb180107570 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.934+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1801a7240 con 0x7fb180107570 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.935+0000 7fb17cff9700 1 -- 172.21.15.27:0/1846033392 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178005a60 con 0x7fb180107570 2024-04-25T01:24:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.935+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1801a7700 con 0x7fb180107570 2024-04-25T01:24:17.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.936+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb180068f20 con 0x7fb180107570 2024-04-25T01:24:17.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.937+0000 7fb17cff9700 1 -- 172.21.15.27:0/1846033392 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb17801a020 con 0x7fb180107570 2024-04-25T01:24:17.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.937+0000 7fb17cff9700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb1680a2cb0 0x7fb1680a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:17.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.937+0000 7fb17f7fe700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb1680a2cb0 0x7fb1680a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.937+0000 7fb17cff9700 1 -- 172.21.15.27:0/1846033392 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb1780ce630 con 0x7fb180107570 2024-04-25T01:24:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.938+0000 7fb17f7fe700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb1680a2cb0 0x7fb1680a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb170005fd0 tx=0x7fb17000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:17.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:17.942+0000 7fb17cff9700 1 -- 172.21.15.27:0/1846033392 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb178010070 con 0x7fb180107570 2024-04-25T01:24:18.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.161+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fb18002db30 con 0x7fb180107570 2024-04-25T01:24:18.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.161+0000 7fb17cff9700 1 -- 172.21.15.27:0/1846033392 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb17809b0d0 con 0x7fb180107570 2024-04-25T01:24:18.142 INFO:teuthology.orchestra.run.smithi027.stdout:253403070509 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb1680a2cb0 msgr2=0x7fb1680a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb1680a2cb0 0x7fb1680a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb170005fd0 tx=0x7fb17000a560 comp rx=0 tx=0).stop 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180107570 msgr2=0x7fb180074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180107570 0x7fb180074fb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb178007ee0 tx=0x7fb17800d3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 shutdown_connections 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb1680a2cb0 0x7fb1680a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb180107570 0x7fb180074fb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1801024e0 0x7fb180074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 --2- 172.21.15.27:0/1846033392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb180102e90 0x7fb180074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.163+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 >> 172.21.15.27:0/1846033392 conn(0x7fb180079390 msgr2=0x7fb180101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.164+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 shutdown_connections 2024-04-25T01:24:18.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.164+0000 7fb185dcb700 1 -- 172.21.15.27:0/1846033392 wait complete. 2024-04-25T01:24:19.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:19 smithi139 ceph-mon[32760]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:19.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:19 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1846033392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:24:19.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:19 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1947128578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:24:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 -- 172.21.15.27:0/47511883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 msgr2=0x7f2fa010e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/47511883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 0x7f2fa010e020 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800b0d0 tx=0x7f2f9800b3e0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 -- 172.21.15.27:0/47511883 shutdown_connections 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/47511883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 0x7f2fa010e020 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/47511883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa01013c0 0x7f2fa01017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/47511883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fa0101d70 0x7f2fa0105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 -- 172.21.15.27:0/47511883 >> 172.21.15.27:0/47511883 conn(0x7f2fa0078100 msgr2=0x7f2fa0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 -- 172.21.15.27:0/47511883 shutdown_connections 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.720+0000 7f2fa4fae700 1 -- 172.21.15.27:0/47511883 wait complete. 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.721+0000 7f2fa4fae700 1 Processor -- start 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.721+0000 7f2fa4fae700 1 -- start start 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.721+0000 7f2fa4fae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa01013c0 0x7f2fa019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2fa4fae700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fa0101d70 0x7f2fa019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2fa4fae700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 0x7f2fa0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2fa4fae700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0111440 con 0x7f2fa0106450 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2fa4fae700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa01112c0 con 0x7f2fa0101d70 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2fa4fae700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa01115c0 con 0x7f2fa01013c0 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 0x7f2fa0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 0x7f2fa0198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43772/0 (socket says 172.21.15.27:43772) 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 -- 172.21.15.27:0/1947128578 learned_addr learned my addr 172.21.15.27:0/1947128578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9e59c700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa01013c0 0x7f2fa019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9dd9b700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fa0101d70 0x7f2fa019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 -- 172.21.15.27:0/1947128578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa01013c0 msgr2=0x7f2fa019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa01013c0 0x7f2fa019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 -- 172.21.15.27:0/1947128578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fa0101d70 msgr2=0x7f2fa019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fa0101d70 0x7f2fa019ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 -- 172.21.15.27:0/1947128578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa0199030 con 0x7f2fa0106450 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f9ed9d700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 0x7f2fa0198750 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f98006070 tx=0x7f2f98004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f977fe700 1 -- 172.21.15.27:0/1947128578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98021070 con 0x7f2fa0106450 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa01992c0 con 0x7f2fa0106450 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f977fe700 1 -- 172.21.15.27:0/1947128578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f98007b80 con 0x7f2fa0106450 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2f977fe700 1 -- 172.21.15.27:0/1947128578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98005330 con 0x7f2fa0106450 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.722+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa01a5600 con 0x7f2fa0106450 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.723+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa0102850 con 0x7f2fa0106450 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.725+0000 7f2f977fe700 1 -- 172.21.15.27:0/1947128578 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2f98024070 con 0x7f2fa0106450 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.725+0000 7f2f977fe700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2f8c0a2cb0 0x7f2f8c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.725+0000 7f2f9e59c700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2f8c0a2cb0 0x7f2f8c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.725+0000 7f2f977fe700 1 -- 172.21.15.27:0/1947128578 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2f9800f020 con 0x7f2fa0106450 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stdout:197568495672 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.726+0000 7f2f9e59c700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2f8c0a2cb0 0x7f2f8c0a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800ac50 tx=0x7f2f8800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.731+0000 7f2f977fe700 1 -- 172.21.15.27:0/1947128578 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2f9802b020 con 0x7f2fa0106450 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.940+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2fa002db30 con 0x7f2fa0106450 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.940+0000 7f2f977fe700 1 -- 172.21.15.27:0/1947128578 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2f9809d620 con 0x7f2fa0106450 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.942+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2f8c0a2cb0 msgr2=0x7f2f8c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.942+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2f8c0a2cb0 0x7f2f8c0a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800ac50 tx=0x7f2f8800a380 comp rx=0 tx=0).stop 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 msgr2=0x7f2fa0198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 0x7f2fa0198750 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f98006070 tx=0x7f2f98004d00 comp rx=0 tx=0).stop 2024-04-25T01:24:19.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 shutdown_connections 2024-04-25T01:24:19.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2f8c0a2cb0 0x7f2f8c0a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fa0106450 0x7f2fa0198750 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa01013c0 0x7f2fa019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 --2- 172.21.15.27:0/1947128578 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fa0101d70 0x7f2fa019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:19.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 >> 172.21.15.27:0/1947128578 conn(0x7f2fa0078100 msgr2=0x7f2fa006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:19.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 shutdown_connections 2024-04-25T01:24:19.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:18.943+0000 7f2fa4fae700 1 -- 172.21.15.27:0/1947128578 wait complete. 2024-04-25T01:24:19.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:19 smithi027 ceph-mon[29197]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:19.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:19 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1846033392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:24:19.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:19 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1947128578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:24:19.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:19 smithi153 ceph-mon[32621]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:19.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:19 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1846033392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:24:19.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:19 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1947128578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:24:21.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:21 smithi139 ceph-mon[32760]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:21.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:21 smithi027 ceph-mon[29197]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:21.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:21 smithi153 ceph-mon[32621]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:22.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 -- 172.21.15.27:0/2477795311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a8101af0 msgr2=0x7f17a810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 --2- 172.21.15.27:0/2477795311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a8101af0 0x7f17a810dfe0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009990 tx=0x7f17a0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 -- 172.21.15.27:0/2477795311 shutdown_connections 2024-04-25T01:24:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 --2- 172.21.15.27:0/2477795311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a8101af0 0x7f17a810dfe0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 --2- 172.21.15.27:0/2477795311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a81071a0 0x7f17a8107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 --2- 172.21.15.27:0/2477795311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a8101130 0x7f17a81015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 -- 172.21.15.27:0/2477795311 >> 172.21.15.27:0/2477795311 conn(0x7f17a80780e0 msgr2=0x7f17a80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 -- 172.21.15.27:0/2477795311 shutdown_connections 2024-04-25T01:24:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.722+0000 7f17afe07700 1 -- 172.21.15.27:0/2477795311 wait complete. 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.723+0000 7f17afe07700 1 Processor -- start 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.723+0000 7f17afe07700 1 -- start start 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.723+0000 7f17afe07700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a8101130 0x7f17a819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.723+0000 7f17afe07700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a8101af0 0x7f17a819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17afe07700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a81071a0 0x7f17a81987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17afe07700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17a8111430 con 0x7f17a81071a0 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17afe07700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81112b0 con 0x7f17a8101af0 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17afe07700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81115b0 con 0x7f17a8101130 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a81071a0 0x7f17a81987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a81071a0 0x7f17a81987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43778/0 (socket says 172.21.15.27:43778) 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17adba3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a8101130 0x7f17a819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 -- 172.21.15.27:0/4224997398 learned_addr learned my addr 172.21.15.27:0/4224997398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ad3a2700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a8101af0 0x7f17a819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 -- 172.21.15.27:0/4224997398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a8101130 msgr2=0x7f17a819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a8101130 0x7f17a819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 -- 172.21.15.27:0/4224997398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a8101af0 msgr2=0x7f17a819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a8101af0 0x7f17a819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 -- 172.21.15.27:0/4224997398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a8198f80 con 0x7f17a81071a0 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ad3a2700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a8101af0 0x7f17a819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17ae3a4700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a81071a0 0x7f17a81987a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f17a00047b0 tx=0x7f17a0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17a6ffd700 1 -- 172.21.15.27:0/4224997398 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0027070 con 0x7f17a81071a0 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17a6ffd700 1 -- 172.21.15.27:0/4224997398 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17a0003e40 con 0x7f17a81071a0 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a8199210 con 0x7f17a81071a0 2024-04-25T01:24:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.724+0000 7f17a6ffd700 1 -- 172.21.15.27:0/4224997398 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0022c50 con 0x7f17a81071a0 2024-04-25T01:24:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.725+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a81a5480 con 0x7f17a81071a0 2024-04-25T01:24:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.725+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a8102850 con 0x7f17a81071a0 2024-04-25T01:24:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.727+0000 7f17a6ffd700 1 -- 172.21.15.27:0/4224997398 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f17a002a070 con 0x7f17a81071a0 2024-04-25T01:24:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.728+0000 7f17a6ffd700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17900a2d30 0x7f17900a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.728+0000 7f17adba3700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17900a2d30 0x7f17900a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.728+0000 7f17a6ffd700 1 -- 172.21.15.27:0/4224997398 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f17a00d0400 con 0x7f17a81071a0 2024-04-25T01:24:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.730+0000 7f17adba3700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17900a2d30 0x7f17900a51f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1798003f10 tx=0x7f1798009180 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.736+0000 7f17a6ffd700 1 -- 172.21.15.27:0/4224997398 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f17a005e740 con 0x7f17a81071a0 2024-04-25T01:24:22.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.945+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f17a81a5650 con 0x7f17a81071a0 2024-04-25T01:24:22.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.946+0000 7f17a6ffd700 1 -- 172.21.15.27:0/4224997398 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f17a0013070 con 0x7f17a81071a0 2024-04-25T01:24:22.927 INFO:teuthology.orchestra.run.smithi027.stdout:287762808873 2024-04-25T01:24:22.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.948+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17900a2d30 msgr2=0x7f17900a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.948+0000 7f17afe07700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17900a2d30 0x7f17900a51f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1798003f10 tx=0x7f1798009180 comp rx=0 tx=0).stop 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.948+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a81071a0 msgr2=0x7f17a81987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.948+0000 7f17afe07700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a81071a0 0x7f17a81987a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f17a00047b0 tx=0x7f17a0005e80 comp rx=0 tx=0).stop 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.948+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 shutdown_connections 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.948+0000 7f17afe07700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17900a2d30 0x7f17900a51f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.948+0000 7f17afe07700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a81071a0 0x7f17a81987a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.949+0000 7f17afe07700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a8101130 0x7f17a819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.949+0000 7f17afe07700 1 --2- 172.21.15.27:0/4224997398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a8101af0 0x7f17a819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.949+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 >> 172.21.15.27:0/4224997398 conn(0x7f17a80780e0 msgr2=0x7f17a8100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.949+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 shutdown_connections 2024-04-25T01:24:22.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:22.949+0000 7f17afe07700 1 -- 172.21.15.27:0/4224997398 wait complete. 2024-04-25T01:24:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:23 smithi027 ceph-mon[29197]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:23.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:23 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4224997398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:24:23.361 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070505 got 253403070509 for osd.8 2024-04-25T01:24:23.361 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:23.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:23 smithi153 ceph-mon[32621]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:23.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:23 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4224997398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:24:23.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:23 smithi139 ceph-mon[32760]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:23.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:23 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4224997398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:24:24.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808873 for osd.9 2024-04-25T01:24:24.658 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:25.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:25 smithi027 ceph-mon[29197]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:25.420 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495669 got 197568495672 for osd.6 2024-04-25T01:24:25.420 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:25 smithi153 ceph-mon[32621]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:25 smithi139 ceph-mon[32760]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.743+0000 7f85c397c700 1 -- 172.21.15.27:0/2945715908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 msgr2=0x7f85bc106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.743+0000 7f85c397c700 1 --2- 172.21.15.27:0/2945715908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 0x7f85bc106ed0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009990 tx=0x7f85ac009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 -- 172.21.15.27:0/2945715908 shutdown_connections 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 --2- 172.21.15.27:0/2945715908 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f85bc107600 0x7f85bc10f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 --2- 172.21.15.27:0/2945715908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc1009e0 0x7f85bc100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 --2- 172.21.15.27:0/2945715908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 0x7f85bc106ed0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 -- 172.21.15.27:0/2945715908 >> 172.21.15.27:0/2945715908 conn(0x7f85bc077730 msgr2=0x7f85bc0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 -- 172.21.15.27:0/2945715908 shutdown_connections 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 -- 172.21.15.27:0/2945715908 wait complete. 2024-04-25T01:24:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 Processor -- start 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.744+0000 7f85c397c700 1 -- start start 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c397c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f85bc1009e0 0x7f85bc19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c397c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 0x7f85bc19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c397c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc107600 0x7f85bc198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c397c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc112550 con 0x7f85bc101390 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c397c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc1123d0 con 0x7f85bc1009e0 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c397c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc1126d0 con 0x7f85bc107600 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 0x7f85bc19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 0x7f85bc19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43800/0 (socket says 172.21.15.27:43800) 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 -- 172.21.15.27:0/1057762173 learned_addr learned my addr 172.21.15.27:0/1057762173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c1f19700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc107600 0x7f85bc198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c1718700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f85bc1009e0 0x7f85bc19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 -- 172.21.15.27:0/1057762173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc107600 msgr2=0x7f85bc198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc107600 0x7f85bc198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 -- 172.21.15.27:0/1057762173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f85bc1009e0 msgr2=0x7f85bc19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f85bc1009e0 0x7f85bc19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 -- 172.21.15.27:0/1057762173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85bc198ff0 con 0x7f85bc101390 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c0f17700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 0x7f85bc19eb80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f85ac006010 tx=0x7f85ac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85b27fc700 1 -- 172.21.15.27:0/1057762173 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac027070 con 0x7f85bc101390 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85bc199280 con 0x7f85bc101390 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85b27fc700 1 -- 172.21.15.27:0/1057762173 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85ac004050 con 0x7f85bc101390 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.745+0000 7f85b27fc700 1 -- 172.21.15.27:0/1057762173 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac022c00 con 0x7f85bc101390 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.746+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85bc1a54a0 con 0x7f85bc101390 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.746+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85bc103aa0 con 0x7f85bc101390 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.748+0000 7f85b27fc700 1 -- 172.21.15.27:0/1057762173 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f85ac02a070 con 0x7f85bc101390 2024-04-25T01:24:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.748+0000 7f85b27fc700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85a80a2cb0 0x7f85a80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stdout:141733920836 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.748+0000 7f85c1718700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85a80a2cb0 0x7f85a80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.748+0000 7f85b27fc700 1 -- 172.21.15.27:0/1057762173 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f85ac0d09f0 con 0x7f85bc101390 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.749+0000 7f85c1718700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85a80a2cb0 0x7f85a80a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f85b8005950 tx=0x7f85b80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.752+0000 7f85b27fc700 1 -- 172.21.15.27:0/1057762173 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f85ac031070 con 0x7f85bc101390 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.968+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f85bc199cf0 con 0x7f85bc101390 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.968+0000 7f85b27fc700 1 -- 172.21.15.27:0/1057762173 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f85ac013070 con 0x7f85bc101390 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85a80a2cb0 msgr2=0x7f85a80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85a80a2cb0 0x7f85a80a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f85b8005950 tx=0x7f85b80058e0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 msgr2=0x7f85bc19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 0x7f85bc19eb80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f85ac006010 tx=0x7f85ac0047b0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 shutdown_connections 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85a80a2cb0 0x7f85a80a5170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc107600 0x7f85bc198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f85bc1009e0 0x7f85bc19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 --2- 172.21.15.27:0/1057762173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85bc101390 0x7f85bc19eb80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.970+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 >> 172.21.15.27:0/1057762173 conn(0x7f85bc077730 msgr2=0x7f85bc0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.971+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 shutdown_connections 2024-04-25T01:24:25.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:25.971+0000 7f85c397c700 1 -- 172.21.15.27:0/1057762173 wait complete. 2024-04-25T01:24:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 -- 172.21.15.27:0/2895073005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d41082b0 msgr2=0x7fd2d4108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 --2- 172.21.15.27:0/2895073005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d41082b0 0x7fd2d4108690 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009990 tx=0x7fd2c4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 -- 172.21.15.27:0/2895073005 shutdown_connections 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 --2- 172.21.15.27:0/2895073005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d406a8e0 0x7fd2d410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 --2- 172.21.15.27:0/2895073005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d41082b0 0x7fd2d4108690 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 --2- 172.21.15.27:0/2895073005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d4069f40 0x7fd2d406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 -- 172.21.15.27:0/2895073005 >> 172.21.15.27:0/2895073005 conn(0x7fd2d40780e0 msgr2=0x7fd2d40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 -- 172.21.15.27:0/2895073005 shutdown_connections 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.206+0000 7fd2da861700 1 -- 172.21.15.27:0/2895073005 wait complete. 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2da861700 1 Processor -- start 2024-04-25T01:24:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2da861700 1 -- start start 2024-04-25T01:24:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2da861700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d4069f40 0x7fd2d41a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2da861700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d406a8e0 0x7fd2d41a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2da861700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d41082b0 0x7fd2d419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2da861700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d41124f0 con 0x7fd2d4069f40 2024-04-25T01:24:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2da861700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d4112370 con 0x7fd2d41082b0 2024-04-25T01:24:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2da861700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d4112670 con 0x7fd2d406a8e0 2024-04-25T01:24:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2d3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d4069f40 0x7fd2d41a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2d3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d4069f40 0x7fd2d41a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43818/0 (socket says 172.21.15.27:43818) 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2d3fff700 1 -- 172.21.15.27:0/2502065024 learned_addr learned my addr 172.21.15.27:0/2502065024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.207+0000 7fd2d37fe700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d406a8e0 0x7fd2d41a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d8dfe700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d41082b0 0x7fd2d419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d3fff700 1 -- 172.21.15.27:0/2502065024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d406a8e0 msgr2=0x7fd2d41a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d3fff700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d406a8e0 0x7fd2d41a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d3fff700 1 -- 172.21.15.27:0/2502065024 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d41082b0 msgr2=0x7fd2d419aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d3fff700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d41082b0 0x7fd2d419aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d3fff700 1 -- 172.21.15.27:0/2502065024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d419b160 con 0x7fd2d4069f40 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d3fff700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d4069f40 0x7fd2d41a0840 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40038b0 tx=0x7fd2c4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d17fa700 1 -- 172.21.15.27:0/2502065024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4027070 con 0x7fd2d4069f40 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d419b3f0 con 0x7fd2d4069f40 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d17fa700 1 -- 172.21.15.27:0/2502065024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2c4004050 con 0x7fd2d4069f40 2024-04-25T01:24:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2d17fa700 1 -- 172.21.15.27:0/2502065024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4022c00 con 0x7fd2d4069f40 2024-04-25T01:24:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.208+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d41a7710 con 0x7fd2d4069f40 2024-04-25T01:24:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.209+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d4068f20 con 0x7fd2d4069f40 2024-04-25T01:24:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.210+0000 7fd2d17fa700 1 -- 172.21.15.27:0/2502065024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd2c402a070 con 0x7fd2d4069f40 2024-04-25T01:24:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.211+0000 7fd2d17fa700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2bc0a2cb0 0x7fd2bc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:26.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.211+0000 7fd2d37fe700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2bc0a2cb0 0x7fd2bc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:26.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.211+0000 7fd2d17fa700 1 -- 172.21.15.27:0/2502065024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd2c40d0bb0 con 0x7fd2d4069f40 2024-04-25T01:24:26.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.212+0000 7fd2d37fe700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2bc0a2cb0 0x7fd2bc0a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4103230 tx=0x7fd2c80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.215+0000 7fd2d17fa700 1 -- 172.21.15.27:0/2502065024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd2c4031020 con 0x7fd2d4069f40 2024-04-25T01:24:26.393 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:26 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1057762173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:24:26.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.429+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd2d41a78e0 con 0x7fd2d4069f40 2024-04-25T01:24:26.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.429+0000 7fd2d17fa700 1 -- 172.21.15.27:0/2502065024 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd2c4013070 con 0x7fd2d4069f40 2024-04-25T01:24:26.409 INFO:teuthology.orchestra.run.smithi027.stdout:77309411406 2024-04-25T01:24:26.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2bc0a2cb0 msgr2=0x7fd2bc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2bc0a2cb0 0x7fd2bc0a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4103230 tx=0x7fd2c80073d0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d4069f40 msgr2=0x7fd2d41a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d4069f40 0x7fd2d41a0840 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40038b0 tx=0x7fd2c4009f50 comp rx=0 tx=0).stop 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 shutdown_connections 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2bc0a2cb0 0x7fd2bc0a5170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d41082b0 0x7fd2d419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2d4069f40 0x7fd2d41a0840 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 --2- 172.21.15.27:0/2502065024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2d406a8e0 0x7fd2d41a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.432+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 >> 172.21.15.27:0/2502065024 conn(0x7fd2d40780e0 msgr2=0x7fd2d410d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.433+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 shutdown_connections 2024-04-25T01:24:26.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.433+0000 7fd2da861700 1 -- 172.21.15.27:0/2502065024 wait complete. 2024-04-25T01:24:26.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:26 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1057762173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:24:26.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:26 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1057762173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:24:27.232 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:27 smithi027 ceph-mon[29197]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:27.232 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:27 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2502065024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:24:27.232 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:27 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1041390961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:24:27.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411401 got 77309411406 for osd.2 2024-04-25T01:24:27.233 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 -- 172.21.15.27:0/1166572866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 msgr2=0x7f27e4108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 --2- 172.21.15.27:0/1166572866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 0x7f27e4108740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009990 tx=0x7f27e0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 -- 172.21.15.27:0/1166572866 shutdown_connections 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 --2- 172.21.15.27:0/1166572866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27e4102c80 0x7f27e410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 --2- 172.21.15.27:0/1166572866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 0x7f27e4108740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 --2- 172.21.15.27:0/1166572866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e41022c0 0x7f27e4102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 -- 172.21.15.27:0/1166572866 >> 172.21.15.27:0/1166572866 conn(0x7f27e4079420 msgr2=0x7f27e4079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 -- 172.21.15.27:0/1166572866 shutdown_connections 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.413+0000 7f27ec730700 1 -- 172.21.15.27:0/1166572866 wait complete. 2024-04-25T01:24:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ec730700 1 Processor -- start 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ec730700 1 -- start start 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ec730700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e41022c0 0x7f27e419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ec730700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27e4102c80 0x7f27e419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ec730700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 0x7f27e4198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ec730700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f27e41125a0 con 0x7f27e4108360 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27eaccd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 0x7f27e4198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27eaccd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 0x7f27e4198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43836/0 (socket says 172.21.15.27:43836) 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27eaccd700 1 -- 172.21.15.27:0/1041390961 learned_addr learned my addr 172.21.15.27:0/1041390961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f27e4112420 con 0x7f27e4102c80 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27e4112720 con 0x7f27e41022c0 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27e9ccb700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27e4102c80 0x7f27e419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.414+0000 7f27ea4cc700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e41022c0 0x7f27e419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27eaccd700 1 -- 172.21.15.27:0/1041390961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e41022c0 msgr2=0x7f27e419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27eaccd700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e41022c0 0x7f27e419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27eaccd700 1 -- 172.21.15.27:0/1041390961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27e4102c80 msgr2=0x7f27e419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27eaccd700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27e4102c80 0x7f27e419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27eaccd700 1 -- 172.21.15.27:0/1041390961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e4199020 con 0x7f27e4108360 2024-04-25T01:24:27.323 INFO:teuthology.orchestra.run.smithi027.stdout:107374182473 2024-04-25T01:24:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27e9ccb700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27e4102c80 0x7f27e419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27eaccd700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 0x7f27e4198840 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f27dc00ef30 tx=0x7f27dc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27db7fe700 1 -- 172.21.15.27:0/1041390961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc0092e0 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27db7fe700 1 -- 172.21.15.27:0/1041390961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27dc004bb0 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e41992b0 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27db7fe700 1 -- 172.21.15.27:0/1041390961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc004280 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.415+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e41a5520 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.416+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e410a820 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.418+0000 7f27db7fe700 1 -- 172.21.15.27:0/1041390961 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f27dc01b030 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.419+0000 7f27db7fe700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f27d00a2cb0 0x7f27d00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.419+0000 7f27db7fe700 1 -- 172.21.15.27:0/1041390961 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f27dc0cf040 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.419+0000 7f27ea4cc700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f27d00a2cb0 0x7f27d00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.419+0000 7f27ea4cc700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f27d00a2cb0 0x7f27d00a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f27e000b2e0 tx=0x7f27e0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.423+0000 7f27db7fe700 1 -- 172.21.15.27:0/1041390961 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f27dc022020 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.650+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f27e4199d70 con 0x7f27e4108360 2024-04-25T01:24:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.650+0000 7f27db7fe700 1 -- 172.21.15.27:0/1041390961 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f27dc010040 con 0x7f27e4108360 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f27d00a2cb0 msgr2=0x7f27d00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f27d00a2cb0 0x7f27d00a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f27e000b2e0 tx=0x7f27e0005960 comp rx=0 tx=0).stop 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 msgr2=0x7f27e4198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 0x7f27e4198840 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f27dc00ef30 tx=0x7f27dc00c3a0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 shutdown_connections 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f27d00a2cb0 0x7f27d00a5170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e4108360 0x7f27e4198840 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e41022c0 0x7f27e419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 --2- 172.21.15.27:0/1041390961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27e4102c80 0x7f27e419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 >> 172.21.15.27:0/1041390961 conn(0x7f27e4079420 msgr2=0x7f27e41010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 shutdown_connections 2024-04-25T01:24:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:26.652+0000 7f27ec730700 1 -- 172.21.15.27:0/1041390961 wait complete. 2024-04-25T01:24:27.427 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:27 smithi153 ceph-mon[32621]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:27.427 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:27 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2502065024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:24:27.427 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:27 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1041390961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:24:27.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:27 smithi139 ceph-mon[32760]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:27.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:27 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2502065024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:24:27.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:27 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1041390961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:24:28.002 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182468 got 107374182473 for osd.3 2024-04-25T01:24:28.002 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:28.489 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920836 for osd.4 2024-04-25T01:24:28.489 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:28.659 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:28.663 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.683+0000 7f2ee7483700 1 -- 172.21.15.27:0/3596315214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee0101af0 msgr2=0x7f2ee010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.683+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3596315214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee0101af0 0x7f2ee010dfe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800b0a0 tx=0x7f2ed800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.683+0000 7f2ee7483700 1 -- 172.21.15.27:0/3596315214 shutdown_connections 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.683+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3596315214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee0101af0 0x7f2ee010dfe0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.683+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3596315214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ee01071a0 0x7f2ee0107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.683+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3596315214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee0101130 0x7f2ee01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.683+0000 7f2ee7483700 1 -- 172.21.15.27:0/3596315214 >> 172.21.15.27:0/3596315214 conn(0x7f2ee00780e0 msgr2=0x7f2ee00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.684+0000 7f2ee7483700 1 -- 172.21.15.27:0/3596315214 shutdown_connections 2024-04-25T01:24:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.684+0000 7f2ee7483700 1 -- 172.21.15.27:0/3596315214 wait complete. 2024-04-25T01:24:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee7483700 1 Processor -- start 2024-04-25T01:24:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee7483700 1 -- start start 2024-04-25T01:24:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee7483700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee0101130 0x7f2ee0199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee7483700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ee0101af0 0x7f2ee0199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee7483700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee01071a0 0x7f2ee01a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee7483700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee0111410 con 0x7f2ee01071a0 2024-04-25T01:24:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee7483700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee0111290 con 0x7f2ee0101af0 2024-04-25T01:24:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee7483700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee0111590 con 0x7f2ee0101130 2024-04-25T01:24:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee4a1e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ee0101af0 0x7f2ee0199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee4a1e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ee0101af0 0x7f2ee0199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:42450/0 (socket says 172.21.15.27:42450) 2024-04-25T01:24:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee4a1e700 1 -- 172.21.15.27:0/3602857648 learned_addr learned my addr 172.21.15.27:0/3602857648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.685+0000 7f2ee521f700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee0101130 0x7f2ee0199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.686+0000 7f2ee5a20700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee01071a0 0x7f2ee01a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.686+0000 7f2ee4a1e700 1 -- 172.21.15.27:0/3602857648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee0101130 msgr2=0x7f2ee0199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.686+0000 7f2ee4a1e700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee0101130 0x7f2ee0199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.686+0000 7f2ee4a1e700 1 -- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee01071a0 msgr2=0x7f2ee01a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.686+0000 7f2ee4a1e700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee01071a0 0x7f2ee01a4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.686+0000 7f2ee4a1e700 1 -- 172.21.15.27:0/3602857648 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee01a4dd0 con 0x7f2ee0101af0 2024-04-25T01:24:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.687+0000 7f2ee4a1e700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ee0101af0 0x7f2ee0199d50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000bf20 tx=0x7f2ed000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.687+0000 7f2ed67fc700 1 -- 172.21.15.27:0/3602857648 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0012c90 con 0x7f2ee0101af0 2024-04-25T01:24:28.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.687+0000 7f2ed67fc700 1 -- 172.21.15.27:0/3602857648 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ed0012e30 con 0x7f2ee0101af0 2024-04-25T01:24:28.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.687+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee0197660 con 0x7f2ee0101af0 2024-04-25T01:24:28.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.687+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee0197bf0 con 0x7f2ee0101af0 2024-04-25T01:24:28.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.687+0000 7f2ed67fc700 1 -- 172.21.15.27:0/3602857648 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00098d0 con 0x7f2ee0101af0 2024-04-25T01:24:28.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.689+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec4005330 con 0x7f2ee0101af0 2024-04-25T01:24:28.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.691+0000 7f2ed67fc700 1 -- 172.21.15.27:0/3602857648 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2ed0049b60 con 0x7f2ee0101af0 2024-04-25T01:24:28.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.692+0000 7f2ed67fc700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2ecc0a2d30 0x7f2ecc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:28.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.692+0000 7f2ee521f700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2ecc0a2d30 0x7f2ecc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:28.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.692+0000 7f2ed67fc700 1 -- 172.21.15.27:0/3602857648 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2ed00d3420 con 0x7f2ee0101af0 2024-04-25T01:24:28.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.692+0000 7f2ee521f700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2ecc0a2d30 0x7f2ecc0a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2edc007b60 tx=0x7f2edc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:28.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.694+0000 7f2ed67fc700 1 -- 172.21.15.27:0/3602857648 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2ed009ffd0 con 0x7f2ee0101af0 2024-04-25T01:24:28.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.913+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2ec40058f0 con 0x7f2ee0101af0 2024-04-25T01:24:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.913+0000 7f2ed67fc700 1 -- 172.21.15.27:0/3602857648 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2ed009f8d0 con 0x7f2ee0101af0 2024-04-25T01:24:28.893 INFO:teuthology.orchestra.run.smithi027.stdout:219043332148 2024-04-25T01:24:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2ecc0a2d30 msgr2=0x7f2ecc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2ecc0a2d30 0x7f2ecc0a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2edc007b60 tx=0x7f2edc0073d0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ee0101af0 msgr2=0x7f2ee0199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ee0101af0 0x7f2ee0199d50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000bf20 tx=0x7f2ed000dc30 comp rx=0 tx=0).stop 2024-04-25T01:24:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 shutdown_connections 2024-04-25T01:24:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2ecc0a2d30 0x7f2ecc0a51f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee01071a0 0x7f2ee01a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ee0101130 0x7f2ee0199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 --2- 172.21.15.27:0/3602857648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2ee0101af0 0x7f2ee0199d50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 >> 172.21.15.27:0/3602857648 conn(0x7f2ee00780e0 msgr2=0x7f2ee00ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 shutdown_connections 2024-04-25T01:24:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:28.915+0000 7f2ee7483700 1 -- 172.21.15.27:0/3602857648 wait complete. 2024-04-25T01:24:29.345 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:29 smithi139 ceph-mon[32760]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:29.345 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:29 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3602857648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:24:29.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:29 smithi027 ceph-mon[29197]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:29.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:29 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3602857648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:24:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:29 smithi153 ceph-mon[32621]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:29.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:29 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3602857648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:24:29.881 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332144 got 219043332148 for osd.7 2024-04-25T01:24:29.882 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:31.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:31 smithi027 ceph-mon[29197]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:31.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:31 smithi153 ceph-mon[32621]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/963187754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb810eac0 msgr2=0x7f5fb8110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/963187754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb810eac0 0x7f5fb8110eb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000b0a0 tx=0x7f5fb000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/963187754 shutdown_connections 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/963187754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb810eac0 0x7f5fb8110eb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/963187754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb8100f10 0x7f5fb81012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/963187754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fb81018c0 0x7f5fb810e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/963187754 >> 172.21.15.27:0/963187754 conn(0x7f5fb80fcb50 msgr2=0x7f5fb80fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/963187754 shutdown_connections 2024-04-25T01:24:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/963187754 wait complete. 2024-04-25T01:24:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 Processor -- start 2024-04-25T01:24:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.556+0000 7f5fbdd9d700 1 -- start start 2024-04-25T01:24:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbdd9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb8100f10 0x7f5fb81a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbdd9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb81018c0 0x7f5fb81a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbdd9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fb810eac0 0x7f5fb819a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbdd9d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8114690 con 0x7f5fb8100f10 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbdd9d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8114510 con 0x7f5fb810eac0 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbdd9d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8114810 con 0x7f5fb81018c0 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbcd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb8100f10 0x7f5fb81a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbcd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb8100f10 0x7f5fb81a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55286/0 (socket says 172.21.15.27:55286) 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbcd9b700 1 -- 172.21.15.27:0/848325848 learned_addr learned my addr 172.21.15.27:0/848325848 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbd59c700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fb810eac0 0x7f5fb819a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fb7fff700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb81018c0 0x7f5fb81a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbd59c700 1 -- 172.21.15.27:0/848325848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb81018c0 msgr2=0x7f5fb81a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbd59c700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb81018c0 0x7f5fb81a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbd59c700 1 -- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb8100f10 msgr2=0x7f5fb81a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbd59c700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb8100f10 0x7f5fb81a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbd59c700 1 -- 172.21.15.27:0/848325848 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb819aff0 con 0x7f5fb810eac0 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.557+0000 7f5fbcd9b700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb8100f10 0x7f5fb81a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.558+0000 7f5fbd59c700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fb810eac0 0x7f5fb819a890 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000b0a0 tx=0x7f5fb0004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.558+0000 7f5fb5ffb700 1 -- 172.21.15.27:0/848325848 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0021070 con 0x7f5fb810eac0 2024-04-25T01:24:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.558+0000 7f5fb5ffb700 1 -- 172.21.15.27:0/848325848 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb0007b80 con 0x7f5fb810eac0 2024-04-25T01:24:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.558+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb819b280 con 0x7f5fb810eac0 2024-04-25T01:24:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.558+0000 7f5fb5ffb700 1 -- 172.21.15.27:0/848325848 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00052a0 con 0x7f5fb810eac0 2024-04-25T01:24:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.558+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb81a75a0 con 0x7f5fb810eac0 2024-04-25T01:24:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.559+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb8068f20 con 0x7f5fb810eac0 2024-04-25T01:24:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.560+0000 7f5fb5ffb700 1 -- 172.21.15.27:0/848325848 <== mon.2 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5fb0024070 con 0x7f5fb810eac0 2024-04-25T01:24:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.561+0000 7f5fb5ffb700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5fa80a2cb0 0x7f5fa80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.561+0000 7f5fbcd9b700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5fa80a2cb0 0x7f5fa80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.561+0000 7f5fb5ffb700 1 -- 172.21.15.27:0/848325848 <== mon.2 v2:172.21.15.139:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5fb000f020 con 0x7f5fb810eac0 2024-04-25T01:24:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.562+0000 7f5fbcd9b700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5fa80a2cb0 0x7f5fa80a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5fb819b9d0 tx=0x7f5fac00b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:31.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:31 smithi139 ceph-mon[32760]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:31.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.565+0000 7f5fb5ffb700 1 -- 172.21.15.27:0/848325848 <== mon.2 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5fb002b020 con 0x7f5fb810eac0 2024-04-25T01:24:31.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.778+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f5fb819bdc0 con 0x7f5fb810eac0 2024-04-25T01:24:31.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.779+0000 7f5fb5ffb700 1 -- 172.21.15.27:0/848325848 <== mon.2 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f5fb009d9c0 con 0x7f5fb810eac0 2024-04-25T01:24:31.759 INFO:teuthology.orchestra.run.smithi027.stdout:343597383712 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5fa80a2cb0 msgr2=0x7f5fa80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5fa80a2cb0 0x7f5fa80a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5fb819b9d0 tx=0x7f5fac00b480 comp rx=0 tx=0).stop 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fb810eac0 msgr2=0x7f5fb819a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fb810eac0 0x7f5fb819a890 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000b0a0 tx=0x7f5fb0004d00 comp rx=0 tx=0).stop 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 shutdown_connections 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5fa80a2cb0 0x7f5fa80a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fb810eac0 0x7f5fb819a890 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fb8100f10 0x7f5fb81a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 --2- 172.21.15.27:0/848325848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb81018c0 0x7f5fb81a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 >> 172.21.15.27:0/848325848 conn(0x7f5fb80fcb50 msgr2=0x7f5fb810fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:31.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 shutdown_connections 2024-04-25T01:24:31.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:31.781+0000 7f5fbdd9d700 1 -- 172.21.15.27:0/848325848 wait complete. 2024-04-25T01:24:32.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:32 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/848325848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:24:32.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:32 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/848325848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:24:32.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:32 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/848325848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:24:33.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:33 smithi027 ceph-mon[29197]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:33.363 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383705 got 343597383712 for osd.11 2024-04-25T01:24:33.363 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:33.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:33 smithi153 ceph-mon[32621]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:33.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:33 smithi139 ceph-mon[32760]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 -- 172.21.15.27:0/2688331382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 msgr2=0x7fd12410f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 --2- 172.21.15.27:0/2688331382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 0x7fd12410f170 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00b0a0 tx=0x7fd11c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 -- 172.21.15.27:0/2688331382 shutdown_connections 2024-04-25T01:24:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 --2- 172.21.15.27:0/2688331382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 0x7fd12410f170 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 --2- 172.21.15.27:0/2688331382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd124102540 0x7fd124102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 --2- 172.21.15.27:0/2688331382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd124102ef0 0x7fd124106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 -- 172.21.15.27:0/2688331382 >> 172.21.15.27:0/2688331382 conn(0x7fd124079310 msgr2=0x7fd124079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 -- 172.21.15.27:0/2688331382 shutdown_connections 2024-04-25T01:24:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.904+0000 7fd129ac7700 1 -- 172.21.15.27:0/2688331382 wait complete. 2024-04-25T01:24:33.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd129ac7700 1 Processor -- start 2024-04-25T01:24:33.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd129ac7700 1 -- start start 2024-04-25T01:24:33.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd129ac7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd124102540 0x7fd124074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:33.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd129ac7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd124102ef0 0x7fd124074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd129ac7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 0x7fd124074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd129ac7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd1241125d0 con 0x7fd1241075a0 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd129ac7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd124112450 con 0x7fd124102540 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd129ac7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd124112750 con 0x7fd124102ef0 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd123fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 0x7fd124074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd123fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 0x7fd124074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55302/0 (socket says 172.21.15.27:55302) 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd123fff700 1 -- 172.21.15.27:0/3713104713 learned_addr learned my addr 172.21.15.27:0/3713104713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.905+0000 7fd122ffd700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd124102ef0 0x7fd124074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd1237fe700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd124102540 0x7fd124074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd123fff700 1 -- 172.21.15.27:0/3713104713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd124102ef0 msgr2=0x7fd124074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd123fff700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd124102ef0 0x7fd124074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd123fff700 1 -- 172.21.15.27:0/3713104713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd124102540 msgr2=0x7fd124074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd123fff700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd124102540 0x7fd124074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd123fff700 1 -- 172.21.15.27:0/3713104713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1241a6f00 con 0x7fd1241075a0 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd122ffd700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd124102ef0 0x7fd124074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd123fff700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 0x7fd124074fb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00d010 tx=0x7fd11c004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd120ff9700 1 -- 172.21.15.27:0/3713104713 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c021070 con 0x7fd1241075a0 2024-04-25T01:24:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd120ff9700 1 -- 172.21.15.27:0/3713104713 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd11c003c00 con 0x7fd1241075a0 2024-04-25T01:24:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1241a7190 con 0x7fd1241075a0 2024-04-25T01:24:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd120ff9700 1 -- 172.21.15.27:0/3713104713 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c004850 con 0x7fd1241075a0 2024-04-25T01:24:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.906+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1241a7640 con 0x7fd1241075a0 2024-04-25T01:24:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.908+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd124068f20 con 0x7fd1241075a0 2024-04-25T01:24:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.909+0000 7fd120ff9700 1 -- 172.21.15.27:0/3713104713 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd11c003db0 con 0x7fd1241075a0 2024-04-25T01:24:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.910+0000 7fd120ff9700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd10c0a2cb0 0x7fd10c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.910+0000 7fd1237fe700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd10c0a2cb0 0x7fd10c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.910+0000 7fd120ff9700 1 -- 172.21.15.27:0/3713104713 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd11c00f020 con 0x7fd1241075a0 2024-04-25T01:24:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.910+0000 7fd1237fe700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd10c0a2cb0 0x7fd10c0a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd114005950 tx=0x7fd11400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:33.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:33.914+0000 7fd120ff9700 1 -- 172.21.15.27:0/3713104713 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd11c02a020 con 0x7fd1241075a0 2024-04-25T01:24:34.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.130+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd12402db30 con 0x7fd1241075a0 2024-04-25T01:24:34.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.130+0000 7fd120ff9700 1 -- 172.21.15.27:0/3713104713 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd11c09d4d0 con 0x7fd1241075a0 2024-04-25T01:24:34.110 INFO:teuthology.orchestra.run.smithi027.stdout:55834574932 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd10c0a2cb0 msgr2=0x7fd10c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd10c0a2cb0 0x7fd10c0a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd114005950 tx=0x7fd11400b410 comp rx=0 tx=0).stop 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 msgr2=0x7fd124074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 0x7fd124074fb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00d010 tx=0x7fd11c004d50 comp rx=0 tx=0).stop 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 shutdown_connections 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd10c0a2cb0 0x7fd10c0a5170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1241075a0 0x7fd124074fb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd124102540 0x7fd124074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 --2- 172.21.15.27:0/3713104713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd124102ef0 0x7fd124074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 >> 172.21.15.27:0/3713104713 conn(0x7fd124079310 msgr2=0x7fd124100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:34.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 shutdown_connections 2024-04-25T01:24:34.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:34.132+0000 7fd129ac7700 1 -- 172.21.15.27:0/3713104713 wait complete. 2024-04-25T01:24:34.873 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574925 got 55834574932 for osd.1 2024-04-25T01:24:34.873 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:35.096 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:35 smithi027 ceph-mon[29197]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:35.097 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:35 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3713104713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:24:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 -- 172.21.15.27:0/4053892247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 msgr2=0x7ff81c107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 --2- 172.21.15.27:0/4053892247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 0x7ff81c107580 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff81400b0a0 tx=0x7ff81400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 -- 172.21.15.27:0/4053892247 shutdown_connections 2024-04-25T01:24:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 --2- 172.21.15.27:0/4053892247 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff81c101af0 0x7ff81c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 --2- 172.21.15.27:0/4053892247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 0x7ff81c107580 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 --2- 172.21.15.27:0/4053892247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff81c101130 0x7ff81c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 -- 172.21.15.27:0/4053892247 >> 172.21.15.27:0/4053892247 conn(0x7ff81c0780e0 msgr2=0x7ff81c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 -- 172.21.15.27:0/4053892247 shutdown_connections 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 -- 172.21.15.27:0/4053892247 wait complete. 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.116+0000 7ff821ee0700 1 Processor -- start 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff821ee0700 1 -- start start 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff821ee0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff81c101130 0x7ff81c102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff821ee0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff81c101af0 0x7ff81c103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff821ee0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 0x7ff81c103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff821ee0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c111440 con 0x7ff81c1071a0 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff821ee0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1112c0 con 0x7ff81c101af0 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff821ee0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1115c0 con 0x7ff81c101130 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff81bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 0x7ff81c103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff81bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 0x7ff81c103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55306/0 (socket says 172.21.15.27:55306) 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff81bfff700 1 -- 172.21.15.27:0/2944454256 learned_addr learned my addr 172.21.15.27:0/2944454256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff81b7fe700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff81c101130 0x7ff81c102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:35.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.117+0000 7ff81affd700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff81c101af0 0x7ff81c103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff81bfff700 1 -- 172.21.15.27:0/2944454256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff81c101130 msgr2=0x7ff81c102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff81bfff700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff81c101130 0x7ff81c102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff81bfff700 1 -- 172.21.15.27:0/2944454256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff81c101af0 msgr2=0x7ff81c103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff81bfff700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff81c101af0 0x7ff81c103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff81bfff700 1 -- 172.21.15.27:0/2944454256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c1a6ec0 con 0x7ff81c1071a0 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff81affd700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff81c101af0 0x7ff81c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff81bfff700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 0x7ff81c103890 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00c900 tx=0x7ff80c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff818ff9700 1 -- 172.21.15.27:0/2944454256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c007b00 con 0x7ff81c1071a0 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff818ff9700 1 -- 172.21.15.27:0/2944454256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff80c00ce50 con 0x7ff81c1071a0 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81c1a71b0 con 0x7ff81c1071a0 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff818ff9700 1 -- 172.21.15.27:0/2944454256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c018660 con 0x7ff81c1071a0 2024-04-25T01:24:35.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.118+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81c1a76a0 con 0x7ff81c1071a0 2024-04-25T01:24:35.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.120+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff81c068f20 con 0x7ff81c1071a0 2024-04-25T01:24:35.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.120+0000 7ff818ff9700 1 -- 172.21.15.27:0/2944454256 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff80c00f450 con 0x7ff81c1071a0 2024-04-25T01:24:35.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.121+0000 7ff818ff9700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff8080a2cb0 0x7ff8080a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:35.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.121+0000 7ff81b7fe700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff8080a2cb0 0x7ff8080a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:35.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.121+0000 7ff818ff9700 1 -- 172.21.15.27:0/2944454256 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff80c0cf470 con 0x7ff81c1071a0 2024-04-25T01:24:35.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.122+0000 7ff81b7fe700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff8080a2cb0 0x7ff8080a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff814009e60 tx=0x7ff81400b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:35.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.125+0000 7ff818ff9700 1 -- 172.21.15.27:0/2944454256 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff80c014030 con 0x7ff81c1071a0 2024-04-25T01:24:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.344+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ff81c1a7900 con 0x7ff81c1071a0 2024-04-25T01:24:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.345+0000 7ff818ff9700 1 -- 172.21.15.27:0/2944454256 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff80c09bf10 con 0x7ff81c1071a0 2024-04-25T01:24:35.325 INFO:teuthology.orchestra.run.smithi027.stdout:163208757312 2024-04-25T01:24:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff8080a2cb0 msgr2=0x7ff8080a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff8080a2cb0 0x7ff8080a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff814009e60 tx=0x7ff81400b7b0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 msgr2=0x7ff81c103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 0x7ff81c103890 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00c900 tx=0x7ff80c00cc10 comp rx=0 tx=0).stop 2024-04-25T01:24:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 shutdown_connections 2024-04-25T01:24:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff8080a2cb0 0x7ff8080a5170 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff81c1071a0 0x7ff81c103890 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff81c101130 0x7ff81c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 --2- 172.21.15.27:0/2944454256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff81c101af0 0x7ff81c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 >> 172.21.15.27:0/2944454256 conn(0x7ff81c0780e0 msgr2=0x7ff81c100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 shutdown_connections 2024-04-25T01:24:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:35.347+0000 7ff821ee0700 1 -- 172.21.15.27:0/2944454256 wait complete. 2024-04-25T01:24:35.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:35 smithi153 ceph-mon[32621]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:35.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:35 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3713104713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:24:35.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:35 smithi139 ceph-mon[32760]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:35.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:35 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3713104713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:24:36.038 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757306 got 163208757312 for osd.5 2024-04-25T01:24:36.039 DEBUG:teuthology.parallel:result is None 2024-04-25T01:24:36.039 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T01:24:36.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:24:36.293 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:36.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:36.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:36.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:36 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2944454256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:24:36.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:36 smithi027 ceph-mon[29197]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:36.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:36.577 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:36 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:36.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:36.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:36.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:36 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2944454256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:24:36.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:36 smithi153 ceph-mon[32621]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:36.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:36.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:36 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:36.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:36.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:36.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:36 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2944454256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:24:36.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:36 smithi139 ceph-mon[32760]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:36.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:36.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:36 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' 2024-04-25T01:24:38.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 -- 172.21.15.27:0/3066352502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc101290 msgr2=0x7fdffc10d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 --2- 172.21.15.27:0/3066352502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc101290 0x7fdffc10d7a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdff400b0a0 tx=0x7fdff400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 -- 172.21.15.27:0/3066352502 shutdown_connections 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 --2- 172.21.15.27:0/3066352502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc101290 0x7fdffc10d7a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 --2- 172.21.15.27:0/3066352502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdffc1068d0 0x7fdffc106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 --2- 172.21.15.27:0/3066352502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc1008d0 0x7fdffc100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 -- 172.21.15.27:0/3066352502 >> 172.21.15.27:0/3066352502 conn(0x7fdffc077730 msgr2=0x7fdffc0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 -- 172.21.15.27:0/3066352502 shutdown_connections 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 -- 172.21.15.27:0/3066352502 wait complete. 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.150+0000 7fe000924700 1 Processor -- start 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fe000924700 1 -- start start 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fe000924700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc1008d0 0x7fdffc19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fe000924700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdffc101290 0x7fdffc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fe000924700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc1068d0 0x7fdffc1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fe000924700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc110390 con 0x7fdffc1008d0 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fe000924700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc110210 con 0x7fdffc101290 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fe000924700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc110510 con 0x7fdffc1068d0 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc1008d0 0x7fdffc19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc1008d0 0x7fdffc19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49648/0 (socket says 172.21.15.27:49648) 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 -- 172.21.15.27:0/2084004693 learned_addr learned my addr 172.21.15.27:0/2084004693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdff9d9b700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdffc101290 0x7fdffc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffad9d700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc1068d0 0x7fdffc1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 -- 172.21.15.27:0/2084004693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc1068d0 msgr2=0x7fdffc1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc1068d0 0x7fdffc1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 -- 172.21.15.27:0/2084004693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdffc101290 msgr2=0x7fdffc19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdffc101290 0x7fdffc19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 -- 172.21.15.27:0/2084004693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc198fc0 con 0x7fdffc1008d0 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdff9d9b700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdffc101290 0x7fdffc19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.151+0000 7fdffa59c700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc1008d0 0x7fdffc19e610 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400c9b0 tx=0x7fdfe400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:38.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.152+0000 7fdff37fe700 1 -- 172.21.15.27:0/2084004693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4007b00 con 0x7fdffc1008d0 2024-04-25T01:24:38.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.152+0000 7fdff37fe700 1 -- 172.21.15.27:0/2084004693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfe4007ca0 con 0x7fdffc1008d0 2024-04-25T01:24:38.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.152+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdffc1992b0 con 0x7fdffc1008d0 2024-04-25T01:24:38.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.152+0000 7fdff37fe700 1 -- 172.21.15.27:0/2084004693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4018780 con 0x7fdffc1008d0 2024-04-25T01:24:38.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.152+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdffc1a5470 con 0x7fdffc1008d0 2024-04-25T01:24:38.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.153+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdffc102010 con 0x7fdffc1008d0 2024-04-25T01:24:38.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.154+0000 7fdff37fe700 1 -- 172.21.15.27:0/2084004693 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fdfe400f450 con 0x7fdffc1008d0 2024-04-25T01:24:38.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.154+0000 7fdff37fe700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdfe80a2cb0 0x7fdfe80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:38.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.155+0000 7fdff9d9b700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdfe80a2cb0 0x7fdfe80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:38.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.155+0000 7fdff37fe700 1 -- 172.21.15.27:0/2084004693 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdfe40cf550 con 0x7fdffc1008d0 2024-04-25T01:24:38.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.155+0000 7fdff9d9b700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdfe80a2cb0 0x7fdfe80a5170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdffc104ec0 tx=0x7fdfec00b360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:38.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.159+0000 7fdff37fe700 1 -- 172.21.15.27:0/2084004693 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdfe4014030 con 0x7fdffc1008d0 2024-04-25T01:24:38.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.359+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdffc199cc0 con 0x7fdfe80a2cb0 2024-04-25T01:24:38.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.369+0000 7fdff37fe700 1 -- 172.21.15.27:0/2084004693 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340668 (secure 0 0 0) 0x7fdffc199cc0 con 0x7fdfe80a2cb0 2024-04-25T01:24:38.354 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:38.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdfe80a2cb0 msgr2=0x7fdfe80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:38.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdfe80a2cb0 0x7fdfe80a5170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdffc104ec0 tx=0x7fdfec00b360 comp rx=0 tx=0).stop 2024-04-25T01:24:38.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc1008d0 msgr2=0x7fdffc19e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:38.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc1008d0 0x7fdffc19e610 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400c9b0 tx=0x7fdfe400cd70 comp rx=0 tx=0).stop 2024-04-25T01:24:38.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 shutdown_connections 2024-04-25T01:24:38.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fdfe80a2cb0 0x7fdfe80a5170 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc1068d0 0x7fdffc1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdffc1008d0 0x7fdffc19e610 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 --2- 172.21.15.27:0/2084004693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdffc101290 0x7fdffc19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:38.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 >> 172.21.15.27:0/2084004693 conn(0x7fdffc077730 msgr2=0x7fdffc103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:38.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 shutdown_connections 2024-04-25T01:24:38.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:38.372+0000 7fe000924700 1 -- 172.21.15.27:0/2084004693 wait complete. 2024-04-25T01:24:38.358 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T01:24:38.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:38 smithi027 ceph-mon[29197]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:38.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:38 smithi153 ceph-mon[32621]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:38.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:38 smithi139 ceph-mon[32760]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-04-25T01:24:37.399926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657959,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":584,"num_read_kb":413,"num_write":344,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361356,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580468,"statfs":{"total":1151940427776,"available":1151570399232,"internally_reserved":0,"allocated":12443648,"data_stored":7959587,"data_compressed":360240,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003808"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005231+0000","last_change":"2024-04-25T01:22:57.847700+0000","last_active":"2024-04-25T01:23:47.005231+0000","last_peered":"2024-04-25T01:23:47.005231+0000","last_clean":"2024-04-25T01:23:47.005231+0000","last_became_active":"2024-04-25T01:22:57.846877+0000","last_became_peered":"2024-04-25T01:22:57.846877+0000","last_unstale":"2024-04-25T01:23:47.005231+0000","last_undegraded":"2024-04-25T01:23:47.005231+0000","last_fullsized":"2024-04-25T01:23:47.005231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:05.918261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002513+0000","last_change":"2024-04-25T01:22:57.848396+0000","last_active":"2024-04-25T01:23:47.002513+0000","last_peered":"2024-04-25T01:23:47.002513+0000","last_clean":"2024-04-25T01:23:47.002513+0000","last_became_active":"2024-04-25T01:22:57.847971+0000","last_became_peered":"2024-04-25T01:22:57.847971+0000","last_unstale":"2024-04-25T01:23:47.002513+0000","last_undegraded":"2024-04-25T01:23:47.002513+0000","last_fullsized":"2024-04-25T01:23:47.002513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:13.143828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002268+0000","last_change":"2024-04-25T01:22:57.856908+0000","last_active":"2024-04-25T01:23:47.002268+0000","last_peered":"2024-04-25T01:23:47.002268+0000","last_clean":"2024-04-25T01:23:47.002268+0000","last_became_active":"2024-04-25T01:22:57.856218+0000","last_became_peered":"2024-04-25T01:22:57.856218+0000","last_unstale":"2024-04-25T01:23:47.002268+0000","last_undegraded":"2024-04-25T01:23:47.002268+0000","last_fullsized":"2024-04-25T01:23:47.002268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:51.971222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002971+0000","last_change":"2024-04-25T01:22:57.849022+0000","last_active":"2024-04-25T01:23:47.002971+0000","last_peered":"2024-04-25T01:23:47.002971+0000","last_clean":"2024-04-25T01:23:47.002971+0000","last_became_active":"2024-04-25T01:22:57.848303+0000","last_became_peered":"2024-04-25T01:22:57.848303+0000","last_unstale":"2024-04-25T01:23:47.002971+0000","last_undegraded":"2024-04-25T01:23:47.002971+0000","last_fullsized":"2024-04-25T01:23:47.002971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:52.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488312+0000","last_change":"2024-04-25T01:23:47.488312+0000","last_active":"2024-04-25T01:23:47.488312+0000","last_peered":"2024-04-25T01:23:47.488312+0000","last_clean":"2024-04-25T01:23:47.488312+0000","last_became_active":"2024-04-25T01:23:47.487923+0000","last_became_peered":"2024-04-25T01:23:47.487923+0000","last_unstale":"2024-04-25T01:23:47.488312+0000","last_undegraded":"2024-04-25T01:23:47.488312+0000","last_fullsized":"2024-04-25T01:23:47.488312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:30.276853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002739+0000","last_change":"2024-04-25T01:22:57.855733+0000","last_active":"2024-04-25T01:23:47.002739+0000","last_peered":"2024-04-25T01:23:47.002739+0000","last_clean":"2024-04-25T01:23:47.002739+0000","last_became_active":"2024-04-25T01:22:57.855075+0000","last_became_peered":"2024-04-25T01:22:57.855075+0000","last_unstale":"2024-04-25T01:23:47.002739+0000","last_undegraded":"2024-04-25T01:23:47.002739+0000","last_fullsized":"2024-04-25T01:23:47.002739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:24.563461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487370+0000","last_change":"2024-04-25T01:22:57.849693+0000","last_active":"2024-04-25T01:23:47.487370+0000","last_peered":"2024-04-25T01:23:47.487370+0000","last_clean":"2024-04-25T01:23:47.487370+0000","last_became_active":"2024-04-25T01:22:57.849324+0000","last_became_peered":"2024-04-25T01:22:57.849324+0000","last_unstale":"2024-04-25T01:23:47.487370+0000","last_undegraded":"2024-04-25T01:23:47.487370+0000","last_fullsized":"2024-04-25T01:23:47.487370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:09.936496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147048+0000","last_change":"2024-04-25T01:22:57.847962+0000","last_active":"2024-04-25T01:23:47.147048+0000","last_peered":"2024-04-25T01:23:47.147048+0000","last_clean":"2024-04-25T01:23:47.147048+0000","last_became_active":"2024-04-25T01:22:57.847592+0000","last_became_peered":"2024-04-25T01:22:57.847592+0000","last_unstale":"2024-04-25T01:23:47.147048+0000","last_undegraded":"2024-04-25T01:23:47.147048+0000","last_fullsized":"2024-04-25T01:23:47.147048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:45.594118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487589+0000","last_change":"2024-04-25T01:22:57.860711+0000","last_active":"2024-04-25T01:23:47.487589+0000","last_peered":"2024-04-25T01:23:47.487589+0000","last_clean":"2024-04-25T01:23:47.487589+0000","last_became_active":"2024-04-25T01:22:57.860393+0000","last_became_peered":"2024-04-25T01:22:57.860393+0000","last_unstale":"2024-04-25T01:23:47.487589+0000","last_undegraded":"2024-04-25T01:23:47.487589+0000","last_fullsized":"2024-04-25T01:23:47.487589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:31.068095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004059+0000","last_change":"2024-04-25T01:22:57.853741+0000","last_active":"2024-04-25T01:23:47.004059+0000","last_peered":"2024-04-25T01:23:47.004059+0000","last_clean":"2024-04-25T01:23:47.004059+0000","last_became_active":"2024-04-25T01:22:57.851015+0000","last_became_peered":"2024-04-25T01:22:57.851015+0000","last_unstale":"2024-04-25T01:23:47.004059+0000","last_undegraded":"2024-04-25T01:23:47.004059+0000","last_fullsized":"2024-04-25T01:23:47.004059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:45.831335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488219+0000","last_change":"2024-04-25T01:22:57.853354+0000","last_active":"2024-04-25T01:23:47.488219+0000","last_peered":"2024-04-25T01:23:47.488219+0000","last_clean":"2024-04-25T01:23:47.488219+0000","last_became_active":"2024-04-25T01:22:57.853043+0000","last_became_peered":"2024-04-25T01:22:57.853043+0000","last_unstale":"2024-04-25T01:23:47.488219+0000","last_undegraded":"2024-04-25T01:23:47.488219+0000","last_fullsized":"2024-04-25T01:23:47.488219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:54.378751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003089+0000","last_change":"2024-04-25T01:22:57.855542+0000","last_active":"2024-04-25T01:23:47.003089+0000","last_peered":"2024-04-25T01:23:47.003089+0000","last_clean":"2024-04-25T01:23:47.003089+0000","last_became_active":"2024-04-25T01:22:57.855196+0000","last_became_peered":"2024-04-25T01:22:57.855196+0000","last_unstale":"2024-04-25T01:23:47.003089+0000","last_undegraded":"2024-04-25T01:23:47.003089+0000","last_fullsized":"2024-04-25T01:23:47.003089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:56.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487277+0000","last_change":"2024-04-25T01:22:57.829353+0000","last_active":"2024-04-25T01:23:47.487277+0000","last_peered":"2024-04-25T01:23:47.487277+0000","last_clean":"2024-04-25T01:23:47.487277+0000","last_became_active":"2024-04-25T01:22:57.828937+0000","last_became_peered":"2024-04-25T01:22:57.828937+0000","last_unstale":"2024-04-25T01:23:47.487277+0000","last_undegraded":"2024-04-25T01:23:47.487277+0000","last_fullsized":"2024-04-25T01:23:47.487277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.541556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002018+0000","last_change":"2024-04-25T01:22:57.854635+0000","last_active":"2024-04-25T01:23:47.002018+0000","last_peered":"2024-04-25T01:23:47.002018+0000","last_clean":"2024-04-25T01:23:47.002018+0000","last_became_active":"2024-04-25T01:22:57.854292+0000","last_became_peered":"2024-04-25T01:22:57.854292+0000","last_unstale":"2024-04-25T01:23:47.002018+0000","last_undegraded":"2024-04-25T01:23:47.002018+0000","last_fullsized":"2024-04-25T01:23:47.002018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:48.209059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002065+0000","last_change":"2024-04-25T01:22:57.852210+0000","last_active":"2024-04-25T01:23:47.002065+0000","last_peered":"2024-04-25T01:23:47.002065+0000","last_clean":"2024-04-25T01:23:47.002065+0000","last_became_active":"2024-04-25T01:22:57.851833+0000","last_became_peered":"2024-04-25T01:22:57.851833+0000","last_unstale":"2024-04-25T01:23:47.002065+0000","last_undegraded":"2024-04-25T01:23:47.002065+0000","last_fullsized":"2024-04-25T01:23:47.002065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:14.664785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.486765+0000","last_change":"2024-04-25T01:22:57.849357+0000","last_active":"2024-04-25T01:23:47.486765+0000","last_peered":"2024-04-25T01:23:47.486765+0000","last_clean":"2024-04-25T01:23:47.486765+0000","last_became_active":"2024-04-25T01:22:57.849050+0000","last_became_peered":"2024-04-25T01:22:57.849050+0000","last_unstale":"2024-04-25T01:23:47.486765+0000","last_undegraded":"2024-04-25T01:23:47.486765+0000","last_fullsized":"2024-04-25T01:23:47.486765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:37.669424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005156+0000","last_change":"2024-04-25T01:22:57.844790+0000","last_active":"2024-04-25T01:23:47.005156+0000","last_peered":"2024-04-25T01:23:47.005156+0000","last_clean":"2024-04-25T01:23:47.005156+0000","last_became_active":"2024-04-25T01:22:57.844442+0000","last_became_peered":"2024-04-25T01:22:57.844442+0000","last_unstale":"2024-04-25T01:23:47.005156+0000","last_undegraded":"2024-04-25T01:23:47.005156+0000","last_fullsized":"2024-04-25T01:23:47.005156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:33.766034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999905+0000","last_change":"2024-04-25T01:22:57.844430+0000","last_active":"2024-04-25T01:23:46.999905+0000","last_peered":"2024-04-25T01:23:46.999905+0000","last_clean":"2024-04-25T01:23:46.999905+0000","last_became_active":"2024-04-25T01:22:57.844110+0000","last_became_peered":"2024-04-25T01:22:57.844110+0000","last_unstale":"2024-04-25T01:23:46.999905+0000","last_undegraded":"2024-04-25T01:23:46.999905+0000","last_fullsized":"2024-04-25T01:23:46.999905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:50.348249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002121+0000","last_change":"2024-04-25T01:22:57.852705+0000","last_active":"2024-04-25T01:23:47.002121+0000","last_peered":"2024-04-25T01:23:47.002121+0000","last_clean":"2024-04-25T01:23:47.002121+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:23:47.002121+0000","last_undegraded":"2024-04-25T01:23:47.002121+0000","last_fullsized":"2024-04-25T01:23:47.002121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:10.320459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.001099+0000","last_change":"2024-04-25T01:23:01.879341+0000","last_active":"2024-04-25T01:23:47.001099+0000","last_peered":"2024-04-25T01:23:47.001099+0000","last_clean":"2024-04-25T01:23:47.001099+0000","last_became_active":"2024-04-25T01:23:01.878881+0000","last_became_peered":"2024-04-25T01:23:01.878881+0000","last_unstale":"2024-04-25T01:23:47.001099+0000","last_undegraded":"2024-04-25T01:23:47.001099+0000","last_fullsized":"2024-04-25T01:23:47.001099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:46.036868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999778+0000","last_change":"2024-04-25T01:22:56.839867+0000","last_active":"2024-04-25T01:23:46.999778+0000","last_peered":"2024-04-25T01:23:46.999778+0000","last_clean":"2024-04-25T01:23:46.999778+0000","last_became_active":"2024-04-25T01:22:56.839521+0000","last_became_peered":"2024-04-25T01:22:56.839521+0000","last_unstale":"2024-04-25T01:23:46.999778+0000","last_undegraded":"2024-04-25T01:23:46.999778+0000","last_fullsized":"2024-04-25T01:23:46.999778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:00.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147533+0000","last_change":"2024-04-25T01:22:57.855922+0000","last_active":"2024-04-25T01:23:47.147533+0000","last_peered":"2024-04-25T01:23:47.147533+0000","last_clean":"2024-04-25T01:23:47.147533+0000","last_became_active":"2024-04-25T01:22:57.855584+0000","last_became_peered":"2024-04-25T01:22:57.855584+0000","last_unstale":"2024-04-25T01:23:47.147533+0000","last_undegraded":"2024-04-25T01:23:47.147533+0000","last_fullsized":"2024-04-25T01:23:47.147533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:02.791269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002057+0000","last_change":"2024-04-25T01:23:01.872340+0000","last_active":"2024-04-25T01:23:47.002057+0000","last_peered":"2024-04-25T01:23:47.002057+0000","last_clean":"2024-04-25T01:23:47.002057+0000","last_became_active":"2024-04-25T01:23:01.871900+0000","last_became_peered":"2024-04-25T01:23:01.871900+0000","last_unstale":"2024-04-25T01:23:47.002057+0000","last_undegraded":"2024-04-25T01:23:47.002057+0000","last_fullsized":"2024-04-25T01:23:47.002057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:11.918022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147541+0000","last_change":"2024-04-25T01:22:56.843851+0000","last_active":"2024-04-25T01:23:47.147541+0000","last_peered":"2024-04-25T01:23:47.147541+0000","last_clean":"2024-04-25T01:23:47.147541+0000","last_became_active":"2024-04-25T01:22:56.843232+0000","last_became_peered":"2024-04-25T01:22:56.843232+0000","last_unstale":"2024-04-25T01:23:47.147541+0000","last_undegraded":"2024-04-25T01:23:47.147541+0000","last_fullsized":"2024-04-25T01:23:47.147541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:13.758328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002472+0000","last_change":"2024-04-25T01:22:57.857686+0000","last_active":"2024-04-25T01:23:47.002472+0000","last_peered":"2024-04-25T01:23:47.002472+0000","last_clean":"2024-04-25T01:23:47.002472+0000","last_became_active":"2024-04-25T01:22:57.857175+0000","last_became_peered":"2024-04-25T01:22:57.857175+0000","last_unstale":"2024-04-25T01:23:47.002472+0000","last_undegraded":"2024-04-25T01:23:47.002472+0000","last_fullsized":"2024-04-25T01:23:47.002472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:24.777861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002310+0000","last_change":"2024-04-25T01:23:01.867914+0000","last_active":"2024-04-25T01:23:47.002310+0000","last_peered":"2024-04-25T01:23:47.002310+0000","last_clean":"2024-04-25T01:23:47.002310+0000","last_became_active":"2024-04-25T01:23:01.867103+0000","last_became_peered":"2024-04-25T01:23:01.867103+0000","last_unstale":"2024-04-25T01:23:47.002310+0000","last_undegraded":"2024-04-25T01:23:47.002310+0000","last_fullsized":"2024-04-25T01:23:47.002310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:26.425433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004565+0000","last_change":"2024-04-25T01:22:56.841310+0000","last_active":"2024-04-25T01:23:47.004565+0000","last_peered":"2024-04-25T01:23:47.004565+0000","last_clean":"2024-04-25T01:23:47.004565+0000","last_became_active":"2024-04-25T01:22:56.840882+0000","last_became_peered":"2024-04-25T01:22:56.840882+0000","last_unstale":"2024-04-25T01:23:47.004565+0000","last_undegraded":"2024-04-25T01:23:47.004565+0000","last_fullsized":"2024-04-25T01:23:47.004565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:29.971448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487298+0000","last_change":"2024-04-25T01:22:57.842810+0000","last_active":"2024-04-25T01:23:47.487298+0000","last_peered":"2024-04-25T01:23:47.487298+0000","last_clean":"2024-04-25T01:23:47.487298+0000","last_became_active":"2024-04-25T01:22:57.842457+0000","last_became_peered":"2024-04-25T01:22:57.842457+0000","last_unstale":"2024-04-25T01:23:47.487298+0000","last_undegraded":"2024-04-25T01:23:47.487298+0000","last_fullsized":"2024-04-25T01:23:47.487298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:45.512037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002379+0000","last_change":"2024-04-25T01:23:01.877298+0000","last_active":"2024-04-25T01:23:47.002379+0000","last_peered":"2024-04-25T01:23:47.002379+0000","last_clean":"2024-04-25T01:23:47.002379+0000","last_became_active":"2024-04-25T01:23:01.876874+0000","last_became_peered":"2024-04-25T01:23:01.876874+0000","last_unstale":"2024-04-25T01:23:47.002379+0000","last_undegraded":"2024-04-25T01:23:47.002379+0000","last_fullsized":"2024-04-25T01:23:47.002379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:35:08.221388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003630+0000","last_change":"2024-04-25T01:22:56.838914+0000","last_active":"2024-04-25T01:23:47.003630+0000","last_peered":"2024-04-25T01:23:47.003630+0000","last_clean":"2024-04-25T01:23:47.003630+0000","last_became_active":"2024-04-25T01:22:56.838616+0000","last_became_peered":"2024-04-25T01:22:56.838616+0000","last_unstale":"2024-04-25T01:23:47.003630+0000","last_undegraded":"2024-04-25T01:23:47.003630+0000","last_fullsized":"2024-04-25T01:23:47.003630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:20.761770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037545+0000","last_change":"2024-04-25T01:22:57.851210+0000","last_active":"2024-04-25T01:23:47.037545+0000","last_peered":"2024-04-25T01:23:47.037545+0000","last_clean":"2024-04-25T01:23:47.037545+0000","last_became_active":"2024-04-25T01:22:57.850867+0000","last_became_peered":"2024-04-25T01:22:57.850867+0000","last_unstale":"2024-04-25T01:23:47.037545+0000","last_undegraded":"2024-04-25T01:23:47.037545+0000","last_fullsized":"2024-04-25T01:23:47.037545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:15.998433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.489730+0000","last_change":"2024-04-25T01:23:47.489730+0000","last_active":"2024-04-25T01:23:47.489730+0000","last_peered":"2024-04-25T01:23:47.489730+0000","last_clean":"2024-04-25T01:23:47.489730+0000","last_became_active":"2024-04-25T01:23:47.489190+0000","last_became_peered":"2024-04-25T01:23:47.489190+0000","last_unstale":"2024-04-25T01:23:47.489730+0000","last_undegraded":"2024-04-25T01:23:47.489730+0000","last_fullsized":"2024-04-25T01:23:47.489730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.101073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:35.046632+0000","last_change":"2024-04-25T01:22:56.834741+0000","last_active":"2024-04-25T01:24:35.046632+0000","last_peered":"2024-04-25T01:24:35.046632+0000","last_clean":"2024-04-25T01:24:35.046632+0000","last_became_active":"2024-04-25T01:22:56.834296+0000","last_became_peered":"2024-04-25T01:22:56.834296+0000","last_unstale":"2024-04-25T01:24:35.046632+0000","last_undegraded":"2024-04-25T01:24:35.046632+0000","last_fullsized":"2024-04-25T01:24:35.046632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:38.940603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002414+0000","last_change":"2024-04-25T01:22:57.854646+0000","last_active":"2024-04-25T01:23:47.002414+0000","last_peered":"2024-04-25T01:23:47.002414+0000","last_clean":"2024-04-25T01:23:47.002414+0000","last_became_active":"2024-04-25T01:22:57.853839+0000","last_became_peered":"2024-04-25T01:22:57.853839+0000","last_unstale":"2024-04-25T01:23:47.002414+0000","last_undegraded":"2024-04-25T01:23:47.002414+0000","last_fullsized":"2024-04-25T01:23:47.002414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:36.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000919+0000","last_change":"2024-04-25T01:23:01.875478+0000","last_active":"2024-04-25T01:23:47.000919+0000","last_peered":"2024-04-25T01:23:47.000919+0000","last_clean":"2024-04-25T01:23:47.000919+0000","last_became_active":"2024-04-25T01:23:01.874643+0000","last_became_peered":"2024-04-25T01:23:01.874643+0000","last_unstale":"2024-04-25T01:23:47.000919+0000","last_undegraded":"2024-04-25T01:23:47.000919+0000","last_fullsized":"2024-04-25T01:23:47.000919+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.953507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002703+0000","last_change":"2024-04-25T01:22:56.843118+0000","last_active":"2024-04-25T01:23:47.002703+0000","last_peered":"2024-04-25T01:23:47.002703+0000","last_clean":"2024-04-25T01:23:47.002703+0000","last_became_active":"2024-04-25T01:22:56.842798+0000","last_became_peered":"2024-04-25T01:22:56.842798+0000","last_unstale":"2024-04-25T01:23:47.002703+0000","last_undegraded":"2024-04-25T01:23:47.002703+0000","last_fullsized":"2024-04-25T01:23:47.002703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:00.054913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037229+0000","last_change":"2024-04-25T01:22:57.850265+0000","last_active":"2024-04-25T01:23:47.037229+0000","last_peered":"2024-04-25T01:23:47.037229+0000","last_clean":"2024-04-25T01:23:47.037229+0000","last_became_active":"2024-04-25T01:22:57.849480+0000","last_became_peered":"2024-04-25T01:22:57.849480+0000","last_unstale":"2024-04-25T01:23:47.037229+0000","last_undegraded":"2024-04-25T01:23:47.037229+0000","last_fullsized":"2024-04-25T01:23:47.037229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:48.577252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005121+0000","last_change":"2024-04-25T01:23:01.865369+0000","last_active":"2024-04-25T01:23:47.005121+0000","last_peered":"2024-04-25T01:23:47.005121+0000","last_clean":"2024-04-25T01:23:47.005121+0000","last_became_active":"2024-04-25T01:23:01.864872+0000","last_became_peered":"2024-04-25T01:23:01.864872+0000","last_unstale":"2024-04-25T01:23:47.005121+0000","last_undegraded":"2024-04-25T01:23:47.005121+0000","last_fullsized":"2024-04-25T01:23:47.005121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:56.266514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487008+0000","last_change":"2024-04-25T01:22:56.842576+0000","last_active":"2024-04-25T01:23:47.487008+0000","last_peered":"2024-04-25T01:23:47.487008+0000","last_clean":"2024-04-25T01:23:47.487008+0000","last_became_active":"2024-04-25T01:22:56.841081+0000","last_became_peered":"2024-04-25T01:22:56.841081+0000","last_unstale":"2024-04-25T01:23:47.487008+0000","last_undegraded":"2024-04-25T01:23:47.487008+0000","last_fullsized":"2024-04-25T01:23:47.487008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:08.174426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.146949+0000","last_change":"2024-04-25T01:22:57.842812+0000","last_active":"2024-04-25T01:23:47.146949+0000","last_peered":"2024-04-25T01:23:47.146949+0000","last_clean":"2024-04-25T01:23:47.146949+0000","last_became_active":"2024-04-25T01:22:57.842462+0000","last_became_peered":"2024-04-25T01:22:57.842462+0000","last_unstale":"2024-04-25T01:23:47.146949+0000","last_undegraded":"2024-04-25T01:23:47.146949+0000","last_fullsized":"2024-04-25T01:23:47.146949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:14:13.444850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002382+0000","last_change":"2024-04-25T01:23:01.870401+0000","last_active":"2024-04-25T01:23:47.002382+0000","last_peered":"2024-04-25T01:23:47.002382+0000","last_clean":"2024-04-25T01:23:47.002382+0000","last_became_active":"2024-04-25T01:23:01.869932+0000","last_became_peered":"2024-04-25T01:23:01.869932+0000","last_unstale":"2024-04-25T01:23:47.002382+0000","last_undegraded":"2024-04-25T01:23:47.002382+0000","last_fullsized":"2024-04-25T01:23:47.002382+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:13.323209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036874+0000","last_change":"2024-04-25T01:22:56.835946+0000","last_active":"2024-04-25T01:23:47.036874+0000","last_peered":"2024-04-25T01:23:47.036874+0000","last_clean":"2024-04-25T01:23:47.036874+0000","last_became_active":"2024-04-25T01:22:56.835630+0000","last_became_peered":"2024-04-25T01:22:56.835630+0000","last_unstale":"2024-04-25T01:23:47.036874+0000","last_undegraded":"2024-04-25T01:23:47.036874+0000","last_fullsized":"2024-04-25T01:23:47.036874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:02.128275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999518+0000","last_change":"2024-04-25T01:22:57.844462+0000","last_active":"2024-04-25T01:23:46.999518+0000","last_peered":"2024-04-25T01:23:46.999518+0000","last_clean":"2024-04-25T01:23:46.999518+0000","last_became_active":"2024-04-25T01:22:57.843426+0000","last_became_peered":"2024-04-25T01:22:57.843426+0000","last_unstale":"2024-04-25T01:23:46.999518+0000","last_undegraded":"2024-04-25T01:23:46.999518+0000","last_fullsized":"2024-04-25T01:23:46.999518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:42.896622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147275+0000","last_change":"2024-04-25T01:23:01.871387+0000","last_active":"2024-04-25T01:23:47.147275+0000","last_peered":"2024-04-25T01:23:47.147275+0000","last_clean":"2024-04-25T01:23:47.147275+0000","last_became_active":"2024-04-25T01:23:01.870901+0000","last_became_peered":"2024-04-25T01:23:01.870901+0000","last_unstale":"2024-04-25T01:23:47.147275+0000","last_undegraded":"2024-04-25T01:23:47.147275+0000","last_fullsized":"2024-04-25T01:23:47.147275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:49.582515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:32.041922+0000","last_change":"2024-04-25T01:22:56.834440+0000","last_active":"2024-04-25T01:24:32.041922+0000","last_peered":"2024-04-25T01:24:32.041922+0000","last_clean":"2024-04-25T01:24:32.041922+0000","last_became_active":"2024-04-25T01:22:56.834116+0000","last_became_peered":"2024-04-25T01:22:56.834116+0000","last_unstale":"2024-04-25T01:24:32.041922+0000","last_undegraded":"2024-04-25T01:24:32.041922+0000","last_fullsized":"2024-04-25T01:24:32.041922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:01.176601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487920+0000","last_change":"2024-04-25T01:22:57.853560+0000","last_active":"2024-04-25T01:23:47.487920+0000","last_peered":"2024-04-25T01:23:47.487920+0000","last_clean":"2024-04-25T01:23:47.487920+0000","last_became_active":"2024-04-25T01:22:57.853234+0000","last_became_peered":"2024-04-25T01:22:57.853234+0000","last_unstale":"2024-04-25T01:23:47.487920+0000","last_undegraded":"2024-04-25T01:23:47.487920+0000","last_fullsized":"2024-04-25T01:23:47.487920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:51.139371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999402+0000","last_change":"2024-04-25T01:23:01.873856+0000","last_active":"2024-04-25T01:23:46.999402+0000","last_peered":"2024-04-25T01:23:46.999402+0000","last_clean":"2024-04-25T01:23:46.999402+0000","last_became_active":"2024-04-25T01:23:01.873413+0000","last_became_peered":"2024-04-25T01:23:01.873413+0000","last_unstale":"2024-04-25T01:23:46.999402+0000","last_undegraded":"2024-04-25T01:23:46.999402+0000","last_fullsized":"2024-04-25T01:23:46.999402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:11.802479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487630+0000","last_change":"2024-04-25T01:22:56.831080+0000","last_active":"2024-04-25T01:23:47.487630+0000","last_peered":"2024-04-25T01:23:47.487630+0000","last_clean":"2024-04-25T01:23:47.487630+0000","last_became_active":"2024-04-25T01:22:56.830735+0000","last_became_peered":"2024-04-25T01:22:56.830735+0000","last_unstale":"2024-04-25T01:23:47.487630+0000","last_undegraded":"2024-04-25T01:23:47.487630+0000","last_fullsized":"2024-04-25T01:23:47.487630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:38.418439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002431+0000","last_change":"2024-04-25T01:22:57.853838+0000","last_active":"2024-04-25T01:23:47.002431+0000","last_peered":"2024-04-25T01:23:47.002431+0000","last_clean":"2024-04-25T01:23:47.002431+0000","last_became_active":"2024-04-25T01:22:57.853023+0000","last_became_peered":"2024-04-25T01:22:57.853023+0000","last_unstale":"2024-04-25T01:23:47.002431+0000","last_undegraded":"2024-04-25T01:23:47.002431+0000","last_fullsized":"2024-04-25T01:23:47.002431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:18.515831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003385+0000","last_change":"2024-04-25T01:23:01.865446+0000","last_active":"2024-04-25T01:23:47.003385+0000","last_peered":"2024-04-25T01:23:47.003385+0000","last_clean":"2024-04-25T01:23:47.003385+0000","last_became_active":"2024-04-25T01:23:01.864937+0000","last_became_peered":"2024-04-25T01:23:01.864937+0000","last_unstale":"2024-04-25T01:23:47.003385+0000","last_undegraded":"2024-04-25T01:23:47.003385+0000","last_fullsized":"2024-04-25T01:23:47.003385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:18.041430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037405+0000","last_change":"2024-04-25T01:22:56.836055+0000","last_active":"2024-04-25T01:23:47.037405+0000","last_peered":"2024-04-25T01:23:47.037405+0000","last_clean":"2024-04-25T01:23:47.037405+0000","last_became_active":"2024-04-25T01:22:56.835760+0000","last_became_peered":"2024-04-25T01:22:56.835760+0000","last_unstale":"2024-04-25T01:23:47.037405+0000","last_undegraded":"2024-04-25T01:23:47.037405+0000","last_fullsized":"2024-04-25T01:23:47.037405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:59.249996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487483+0000","last_change":"2024-04-25T01:22:57.852444+0000","last_active":"2024-04-25T01:23:47.487483+0000","last_peered":"2024-04-25T01:23:47.487483+0000","last_clean":"2024-04-25T01:23:47.487483+0000","last_became_active":"2024-04-25T01:22:57.852127+0000","last_became_peered":"2024-04-25T01:22:57.852127+0000","last_unstale":"2024-04-25T01:23:47.487483+0000","last_undegraded":"2024-04-25T01:23:47.487483+0000","last_fullsized":"2024-04-25T01:23:47.487483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:11.718512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.039488+0000","last_change":"2024-04-25T01:23:47.039057+0000","last_active":"2024-04-25T01:23:47.039488+0000","last_peered":"2024-04-25T01:23:47.039488+0000","last_clean":"2024-04-25T01:23:47.039488+0000","last_became_active":"2024-04-25T01:23:47.038568+0000","last_became_peered":"2024-04-25T01:23:47.038568+0000","last_unstale":"2024-04-25T01:23:47.039488+0000","last_undegraded":"2024-04-25T01:23:47.039488+0000","last_fullsized":"2024-04-25T01:23:47.039488+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:23.828975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'33","reported_seq":208,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:33.043126+0000","last_change":"2024-04-25T01:22:56.834125+0000","last_active":"2024-04-25T01:24:33.043126+0000","last_peered":"2024-04-25T01:24:33.043126+0000","last_clean":"2024-04-25T01:24:33.043126+0000","last_became_active":"2024-04-25T01:22:56.833817+0000","last_became_peered":"2024-04-25T01:22:56.833817+0000","last_unstale":"2024-04-25T01:24:33.043126+0000","last_undegraded":"2024-04-25T01:24:33.043126+0000","last_fullsized":"2024-04-25T01:24:33.043126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:58.451250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":148,"num_read_kb":136,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004854+0000","last_change":"2024-04-25T01:22:57.854554+0000","last_active":"2024-04-25T01:23:47.004854+0000","last_peered":"2024-04-25T01:23:47.004854+0000","last_clean":"2024-04-25T01:23:47.004854+0000","last_became_active":"2024-04-25T01:22:57.853328+0000","last_became_peered":"2024-04-25T01:22:57.853328+0000","last_unstale":"2024-04-25T01:23:47.004854+0000","last_undegraded":"2024-04-25T01:23:47.004854+0000","last_fullsized":"2024-04-25T01:23:47.004854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:21.774148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037782+0000","last_change":"2024-04-25T01:23:01.870571+0000","last_active":"2024-04-25T01:23:47.037782+0000","last_peered":"2024-04-25T01:23:47.037782+0000","last_clean":"2024-04-25T01:23:47.037782+0000","last_became_active":"2024-04-25T01:23:01.869982+0000","last_became_peered":"2024-04-25T01:23:01.869982+0000","last_unstale":"2024-04-25T01:23:47.037782+0000","last_undegraded":"2024-04-25T01:23:47.037782+0000","last_fullsized":"2024-04-25T01:23:47.037782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:17.294254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000939+0000","last_change":"2024-04-25T01:22:56.835134+0000","last_active":"2024-04-25T01:23:47.000939+0000","last_peered":"2024-04-25T01:23:47.000939+0000","last_clean":"2024-04-25T01:23:47.000939+0000","last_became_active":"2024-04-25T01:22:56.834207+0000","last_became_peered":"2024-04-25T01:22:56.834207+0000","last_unstale":"2024-04-25T01:23:47.000939+0000","last_undegraded":"2024-04-25T01:23:47.000939+0000","last_fullsized":"2024-04-25T01:23:47.000939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:07.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487006+0000","last_change":"2024-04-25T01:22:57.847693+0000","last_active":"2024-04-25T01:23:47.487006+0000","last_peered":"2024-04-25T01:23:47.487006+0000","last_clean":"2024-04-25T01:23:47.487006+0000","last_became_active":"2024-04-25T01:22:57.847211+0000","last_became_peered":"2024-04-25T01:22:57.847211+0000","last_unstale":"2024-04-25T01:23:47.487006+0000","last_undegraded":"2024-04-25T01:23:47.487006+0000","last_fullsized":"2024-04-25T01:23:47.487006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:48.295608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004806+0000","last_change":"2024-04-25T01:23:01.870340+0000","last_active":"2024-04-25T01:23:47.004806+0000","last_peered":"2024-04-25T01:23:47.004806+0000","last_clean":"2024-04-25T01:23:47.004806+0000","last_became_active":"2024-04-25T01:23:01.869662+0000","last_became_peered":"2024-04-25T01:23:01.869662+0000","last_unstale":"2024-04-25T01:23:47.004806+0000","last_undegraded":"2024-04-25T01:23:47.004806+0000","last_fullsized":"2024-04-25T01:23:47.004806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:05:57.313991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037246+0000","last_change":"2024-04-25T01:22:56.838146+0000","last_active":"2024-04-25T01:23:47.037246+0000","last_peered":"2024-04-25T01:23:47.037246+0000","last_clean":"2024-04-25T01:23:47.037246+0000","last_became_active":"2024-04-25T01:22:56.837815+0000","last_became_peered":"2024-04-25T01:22:56.837815+0000","last_unstale":"2024-04-25T01:23:47.037246+0000","last_undegraded":"2024-04-25T01:23:47.037246+0000","last_fullsized":"2024-04-25T01:23:47.037246+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:14.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488024+0000","last_change":"2024-04-25T01:22:57.850840+0000","last_active":"2024-04-25T01:23:47.488024+0000","last_peered":"2024-04-25T01:23:47.488024+0000","last_clean":"2024-04-25T01:23:47.488024+0000","last_became_active":"2024-04-25T01:22:57.850255+0000","last_became_peered":"2024-04-25T01:22:57.850255+0000","last_unstale":"2024-04-25T01:23:47.488024+0000","last_undegraded":"2024-04-25T01:23:47.488024+0000","last_fullsized":"2024-04-25T01:23:47.488024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:25.887235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999607+0000","last_change":"2024-04-25T01:23:01.869396+0000","last_active":"2024-04-25T01:23:46.999607+0000","last_peered":"2024-04-25T01:23:46.999607+0000","last_clean":"2024-04-25T01:23:46.999607+0000","last_became_active":"2024-04-25T01:23:01.868777+0000","last_became_peered":"2024-04-25T01:23:01.868777+0000","last_unstale":"2024-04-25T01:23:46.999607+0000","last_undegraded":"2024-04-25T01:23:46.999607+0000","last_fullsized":"2024-04-25T01:23:46.999607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:37.276122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:31.041088+0000","last_change":"2024-04-25T01:22:56.833289+0000","last_active":"2024-04-25T01:24:31.041088+0000","last_peered":"2024-04-25T01:24:31.041088+0000","last_clean":"2024-04-25T01:24:31.041088+0000","last_became_active":"2024-04-25T01:22:56.832353+0000","last_became_peered":"2024-04-25T01:22:56.832353+0000","last_unstale":"2024-04-25T01:24:31.041088+0000","last_undegraded":"2024-04-25T01:24:31.041088+0000","last_fullsized":"2024-04-25T01:24:31.041088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:18.251018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487480+0000","last_change":"2024-04-25T01:22:57.851974+0000","last_active":"2024-04-25T01:23:47.487480+0000","last_peered":"2024-04-25T01:23:47.487480+0000","last_clean":"2024-04-25T01:23:47.487480+0000","last_became_active":"2024-04-25T01:22:57.851573+0000","last_became_peered":"2024-04-25T01:22:57.851573+0000","last_unstale":"2024-04-25T01:23:47.487480+0000","last_undegraded":"2024-04-25T01:23:47.487480+0000","last_fullsized":"2024-04-25T01:23:47.487480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:27.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487521+0000","last_change":"2024-04-25T01:23:01.876980+0000","last_active":"2024-04-25T01:23:47.487521+0000","last_peered":"2024-04-25T01:23:47.487521+0000","last_clean":"2024-04-25T01:23:47.487521+0000","last_became_active":"2024-04-25T01:23:01.876548+0000","last_became_peered":"2024-04-25T01:23:01.876548+0000","last_unstale":"2024-04-25T01:23:47.487521+0000","last_undegraded":"2024-04-25T01:23:47.487521+0000","last_fullsized":"2024-04-25T01:23:47.487521+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:13.470736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.001830+0000","last_change":"2024-04-25T01:22:56.839694+0000","last_active":"2024-04-25T01:23:47.001830+0000","last_peered":"2024-04-25T01:23:47.001830+0000","last_clean":"2024-04-25T01:23:47.001830+0000","last_became_active":"2024-04-25T01:22:56.839367+0000","last_became_peered":"2024-04-25T01:22:56.839367+0000","last_unstale":"2024-04-25T01:23:47.001830+0000","last_undegraded":"2024-04-25T01:23:47.001830+0000","last_fullsized":"2024-04-25T01:23:47.001830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:49.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036903+0000","last_change":"2024-04-25T01:22:57.841526+0000","last_active":"2024-04-25T01:23:47.036903+0000","last_peered":"2024-04-25T01:23:47.036903+0000","last_clean":"2024-04-25T01:23:47.036903+0000","last_became_active":"2024-04-25T01:22:57.841192+0000","last_became_peered":"2024-04-25T01:22:57.841192+0000","last_unstale":"2024-04-25T01:23:47.036903+0000","last_undegraded":"2024-04-25T01:23:47.036903+0000","last_fullsized":"2024-04-25T01:23:47.036903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:38.958901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036829+0000","last_change":"2024-04-25T01:23:01.865486+0000","last_active":"2024-04-25T01:23:47.036829+0000","last_peered":"2024-04-25T01:23:47.036829+0000","last_clean":"2024-04-25T01:23:47.036829+0000","last_became_active":"2024-04-25T01:23:01.865068+0000","last_became_peered":"2024-04-25T01:23:01.865068+0000","last_unstale":"2024-04-25T01:23:47.036829+0000","last_undegraded":"2024-04-25T01:23:47.036829+0000","last_fullsized":"2024-04-25T01:23:47.036829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:50.432724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005728+0000","last_change":"2024-04-25T01:22:56.841507+0000","last_active":"2024-04-25T01:23:47.005728+0000","last_peered":"2024-04-25T01:23:47.005728+0000","last_clean":"2024-04-25T01:23:47.005728+0000","last_became_active":"2024-04-25T01:22:56.839970+0000","last_became_peered":"2024-04-25T01:22:56.839970+0000","last_unstale":"2024-04-25T01:23:47.005728+0000","last_undegraded":"2024-04-25T01:23:47.005728+0000","last_fullsized":"2024-04-25T01:23:47.005728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:37.670052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003598+0000","last_change":"2024-04-25T01:22:57.850853+0000","last_active":"2024-04-25T01:23:47.003598+0000","last_peered":"2024-04-25T01:23:47.003598+0000","last_clean":"2024-04-25T01:23:47.003598+0000","last_became_active":"2024-04-25T01:22:57.850498+0000","last_became_peered":"2024-04-25T01:22:57.850498+0000","last_unstale":"2024-04-25T01:23:47.003598+0000","last_undegraded":"2024-04-25T01:23:47.003598+0000","last_fullsized":"2024-04-25T01:23:47.003598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:13.271048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002278+0000","last_change":"2024-04-25T01:23:01.874254+0000","last_active":"2024-04-25T01:23:47.002278+0000","last_peered":"2024-04-25T01:23:47.002278+0000","last_clean":"2024-04-25T01:23:47.002278+0000","last_became_active":"2024-04-25T01:23:01.873711+0000","last_became_peered":"2024-04-25T01:23:01.873711+0000","last_unstale":"2024-04-25T01:23:47.002278+0000","last_undegraded":"2024-04-25T01:23:47.002278+0000","last_fullsized":"2024-04-25T01:23:47.002278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:13.510061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999705+0000","last_change":"2024-04-25T01:22:56.827137+0000","last_active":"2024-04-25T01:23:46.999705+0000","last_peered":"2024-04-25T01:23:46.999705+0000","last_clean":"2024-04-25T01:23:46.999705+0000","last_became_active":"2024-04-25T01:22:56.826769+0000","last_became_peered":"2024-04-25T01:22:56.826769+0000","last_unstale":"2024-04-25T01:23:46.999705+0000","last_undegraded":"2024-04-25T01:23:46.999705+0000","last_fullsized":"2024-04-25T01:23:46.999705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:21.072643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003833+0000","last_change":"2024-04-25T01:22:57.843713+0000","last_active":"2024-04-25T01:23:47.003833+0000","last_peered":"2024-04-25T01:23:47.003833+0000","last_clean":"2024-04-25T01:23:47.003833+0000","last_became_active":"2024-04-25T01:22:57.843384+0000","last_became_peered":"2024-04-25T01:22:57.843384+0000","last_unstale":"2024-04-25T01:23:47.003833+0000","last_undegraded":"2024-04-25T01:23:47.003833+0000","last_fullsized":"2024-04-25T01:23:47.003833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:18.772379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487286+0000","last_change":"2024-04-25T01:23:01.874150+0000","last_active":"2024-04-25T01:23:47.487286+0000","last_peered":"2024-04-25T01:23:47.487286+0000","last_clean":"2024-04-25T01:23:47.487286+0000","last_became_active":"2024-04-25T01:23:01.873718+0000","last_became_peered":"2024-04-25T01:23:01.873718+0000","last_unstale":"2024-04-25T01:23:47.487286+0000","last_undegraded":"2024-04-25T01:23:47.487286+0000","last_fullsized":"2024-04-25T01:23:47.487286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:26.317703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487645+0000","last_change":"2024-04-25T01:22:56.843605+0000","last_active":"2024-04-25T01:23:47.487645+0000","last_peered":"2024-04-25T01:23:47.487645+0000","last_clean":"2024-04-25T01:23:47.487645+0000","last_became_active":"2024-04-25T01:22:56.843313+0000","last_became_peered":"2024-04-25T01:22:56.843313+0000","last_unstale":"2024-04-25T01:23:47.487645+0000","last_undegraded":"2024-04-25T01:23:47.487645+0000","last_fullsized":"2024-04-25T01:23:47.487645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:41.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005378+0000","last_change":"2024-04-25T01:23:47.005378+0000","last_active":"2024-04-25T01:23:47.005378+0000","last_peered":"2024-04-25T01:23:47.005378+0000","last_clean":"2024-04-25T01:23:47.005378+0000","last_became_active":"2024-04-25T01:23:47.004962+0000","last_became_peered":"2024-04-25T01:23:47.004962+0000","last_unstale":"2024-04-25T01:23:47.005378+0000","last_undegraded":"2024-04-25T01:23:47.005378+0000","last_fullsized":"2024-04-25T01:23:47.005378+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:28.548285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487520+0000","last_change":"2024-04-25T01:23:01.872792+0000","last_active":"2024-04-25T01:23:47.487520+0000","last_peered":"2024-04-25T01:23:47.487520+0000","last_clean":"2024-04-25T01:23:47.487520+0000","last_became_active":"2024-04-25T01:23:01.872391+0000","last_became_peered":"2024-04-25T01:23:01.872391+0000","last_unstale":"2024-04-25T01:23:47.487520+0000","last_undegraded":"2024-04-25T01:23:47.487520+0000","last_fullsized":"2024-04-25T01:23:47.487520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:13.088996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037123+0000","last_change":"2024-04-25T01:22:57.843131+0000","last_active":"2024-04-25T01:23:47.037123+0000","last_peered":"2024-04-25T01:23:47.037123+0000","last_clean":"2024-04-25T01:23:47.037123+0000","last_became_active":"2024-04-25T01:22:57.842540+0000","last_became_peered":"2024-04-25T01:22:57.842540+0000","last_unstale":"2024-04-25T01:23:47.037123+0000","last_undegraded":"2024-04-25T01:23:47.037123+0000","last_fullsized":"2024-04-25T01:23:47.037123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:51.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:32.041800+0000","last_change":"2024-04-25T01:22:56.844610+0000","last_active":"2024-04-25T01:24:32.041800+0000","last_peered":"2024-04-25T01:24:32.041800+0000","last_clean":"2024-04-25T01:24:32.041800+0000","last_became_active":"2024-04-25T01:22:56.844237+0000","last_became_peered":"2024-04-25T01:22:56.844237+0000","last_unstale":"2024-04-25T01:24:32.041800+0000","last_undegraded":"2024-04-25T01:24:32.041800+0000","last_fullsized":"2024-04-25T01:24:32.041800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:09.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488111+0000","last_change":"2024-04-25T01:23:01.865711+0000","last_active":"2024-04-25T01:23:47.488111+0000","last_peered":"2024-04-25T01:23:47.488111+0000","last_clean":"2024-04-25T01:23:47.488111+0000","last_became_active":"2024-04-25T01:23:01.865317+0000","last_became_peered":"2024-04-25T01:23:01.865317+0000","last_unstale":"2024-04-25T01:23:47.488111+0000","last_undegraded":"2024-04-25T01:23:47.488111+0000","last_fullsized":"2024-04-25T01:23:47.488111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:08.995490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147119+0000","last_change":"2024-04-25T01:22:57.851623+0000","last_active":"2024-04-25T01:23:47.147119+0000","last_peered":"2024-04-25T01:23:47.147119+0000","last_clean":"2024-04-25T01:23:47.147119+0000","last_became_active":"2024-04-25T01:22:57.850616+0000","last_became_peered":"2024-04-25T01:22:57.850616+0000","last_unstale":"2024-04-25T01:23:47.147119+0000","last_undegraded":"2024-04-25T01:23:47.147119+0000","last_fullsized":"2024-04-25T01:23:47.147119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:27.447996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004047+0000","last_change":"2024-04-25T01:22:56.832094+0000","last_active":"2024-04-25T01:23:47.004047+0000","last_peered":"2024-04-25T01:23:47.004047+0000","last_clean":"2024-04-25T01:23:47.004047+0000","last_became_active":"2024-04-25T01:22:56.831743+0000","last_became_peered":"2024-04-25T01:22:56.831743+0000","last_unstale":"2024-04-25T01:23:47.004047+0000","last_undegraded":"2024-04-25T01:23:47.004047+0000","last_fullsized":"2024-04-25T01:23:47.004047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:06.116081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487298+0000","last_change":"2024-04-25T01:21:40.899517+0000","last_active":"2024-04-25T01:23:47.487298+0000","last_peered":"2024-04-25T01:23:47.487298+0000","last_clean":"2024-04-25T01:23:47.487298+0000","last_became_active":"2024-04-25T01:21:40.899043+0000","last_became_peered":"2024-04-25T01:21:40.899043+0000","last_unstale":"2024-04-25T01:23:47.487298+0000","last_undegraded":"2024-04-25T01:23:47.487298+0000","last_fullsized":"2024-04-25T01:23:47.487298+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_clean_scrub_stamp":"2024-04-25T01:18:12.053452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:41.234852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037363+0000","last_change":"2024-04-25T01:23:01.878334+0000","last_active":"2024-04-25T01:23:47.037363+0000","last_peered":"2024-04-25T01:23:47.037363+0000","last_clean":"2024-04-25T01:23:47.037363+0000","last_became_active":"2024-04-25T01:23:01.877955+0000","last_became_peered":"2024-04-25T01:23:01.877955+0000","last_unstale":"2024-04-25T01:23:47.037363+0000","last_undegraded":"2024-04-25T01:23:47.037363+0000","last_fullsized":"2024-04-25T01:23:47.037363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:33.339430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.486850+0000","last_change":"2024-04-25T01:22:57.843020+0000","last_active":"2024-04-25T01:23:47.486850+0000","last_peered":"2024-04-25T01:23:47.486850+0000","last_clean":"2024-04-25T01:23:47.486850+0000","last_became_active":"2024-04-25T01:22:57.842671+0000","last_became_peered":"2024-04-25T01:22:57.842671+0000","last_unstale":"2024-04-25T01:23:47.486850+0000","last_undegraded":"2024-04-25T01:23:47.486850+0000","last_fullsized":"2024-04-25T01:23:47.486850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:53.831432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487219+0000","last_change":"2024-04-25T01:22:56.836106+0000","last_active":"2024-04-25T01:23:47.487219+0000","last_peered":"2024-04-25T01:23:47.487219+0000","last_clean":"2024-04-25T01:23:47.487219+0000","last_became_active":"2024-04-25T01:22:56.835814+0000","last_became_peered":"2024-04-25T01:22:56.835814+0000","last_unstale":"2024-04-25T01:23:47.487219+0000","last_undegraded":"2024-04-25T01:23:47.487219+0000","last_fullsized":"2024-04-25T01:23:47.487219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:18.004312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004645+0000","last_change":"2024-04-25T01:23:01.870723+0000","last_active":"2024-04-25T01:23:47.004645+0000","last_peered":"2024-04-25T01:23:47.004645+0000","last_clean":"2024-04-25T01:23:47.004645+0000","last_became_active":"2024-04-25T01:23:01.870205+0000","last_became_peered":"2024-04-25T01:23:01.870205+0000","last_unstale":"2024-04-25T01:23:47.004645+0000","last_undegraded":"2024-04-25T01:23:47.004645+0000","last_fullsized":"2024-04-25T01:23:47.004645+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:47.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004743+0000","last_change":"2024-04-25T01:22:56.839180+0000","last_active":"2024-04-25T01:23:47.004743+0000","last_peered":"2024-04-25T01:23:47.004743+0000","last_clean":"2024-04-25T01:23:47.004743+0000","last_became_active":"2024-04-25T01:22:56.838897+0000","last_became_peered":"2024-04-25T01:22:56.838897+0000","last_unstale":"2024-04-25T01:23:47.004743+0000","last_undegraded":"2024-04-25T01:23:47.004743+0000","last_fullsized":"2024-04-25T01:23:47.004743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:27.176333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487420+0000","last_change":"2024-04-25T01:22:57.841727+0000","last_active":"2024-04-25T01:23:47.487420+0000","last_peered":"2024-04-25T01:23:47.487420+0000","last_clean":"2024-04-25T01:23:47.487420+0000","last_became_active":"2024-04-25T01:22:57.841366+0000","last_became_peered":"2024-04-25T01:22:57.841366+0000","last_unstale":"2024-04-25T01:23:47.487420+0000","last_undegraded":"2024-04-25T01:23:47.487420+0000","last_fullsized":"2024-04-25T01:23:47.487420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:20.585916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.146941+0000","last_change":"2024-04-25T01:23:01.869814+0000","last_active":"2024-04-25T01:23:47.146941+0000","last_peered":"2024-04-25T01:23:47.146941+0000","last_clean":"2024-04-25T01:23:47.146941+0000","last_became_active":"2024-04-25T01:23:01.869387+0000","last_became_peered":"2024-04-25T01:23:01.869387+0000","last_unstale":"2024-04-25T01:23:47.146941+0000","last_undegraded":"2024-04-25T01:23:47.146941+0000","last_fullsized":"2024-04-25T01:23:47.146941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:18:45.098295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999944+0000","last_change":"2024-04-25T01:22:56.838506+0000","last_active":"2024-04-25T01:23:46.999944+0000","last_peered":"2024-04-25T01:23:46.999944+0000","last_clean":"2024-04-25T01:23:46.999944+0000","last_became_active":"2024-04-25T01:22:56.838185+0000","last_became_peered":"2024-04-25T01:22:56.838185+0000","last_unstale":"2024-04-25T01:23:46.999944+0000","last_undegraded":"2024-04-25T01:23:46.999944+0000","last_fullsized":"2024-04-25T01:23:46.999944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:35.306231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147665+0000","last_change":"2024-04-25T01:22:57.846544+0000","last_active":"2024-04-25T01:23:47.147665+0000","last_peered":"2024-04-25T01:23:47.147665+0000","last_clean":"2024-04-25T01:23:47.147665+0000","last_became_active":"2024-04-25T01:22:57.843507+0000","last_became_peered":"2024-04-25T01:22:57.843507+0000","last_unstale":"2024-04-25T01:23:47.147665+0000","last_undegraded":"2024-04-25T01:23:47.147665+0000","last_fullsized":"2024-04-25T01:23:47.147665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:30.600317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000574+0000","last_change":"2024-04-25T01:23:01.863986+0000","last_active":"2024-04-25T01:23:47.000574+0000","last_peered":"2024-04-25T01:23:47.000574+0000","last_clean":"2024-04-25T01:23:47.000574+0000","last_became_active":"2024-04-25T01:23:01.863397+0000","last_became_peered":"2024-04-25T01:23:01.863397+0000","last_unstale":"2024-04-25T01:23:47.000574+0000","last_undegraded":"2024-04-25T01:23:47.000574+0000","last_fullsized":"2024-04-25T01:23:47.000574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:45:56.229349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002538+0000","last_change":"2024-04-25T01:22:56.838201+0000","last_active":"2024-04-25T01:23:47.002538+0000","last_peered":"2024-04-25T01:23:47.002538+0000","last_clean":"2024-04-25T01:23:47.002538+0000","last_became_active":"2024-04-25T01:22:56.837890+0000","last_became_peered":"2024-04-25T01:22:56.837890+0000","last_unstale":"2024-04-25T01:23:47.002538+0000","last_undegraded":"2024-04-25T01:23:47.002538+0000","last_fullsized":"2024-04-25T01:23:47.002538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:17.418112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002385+0000","last_change":"2024-04-25T01:22:57.853923+0000","last_active":"2024-04-25T01:23:47.002385+0000","last_peered":"2024-04-25T01:23:47.002385+0000","last_clean":"2024-04-25T01:23:47.002385+0000","last_became_active":"2024-04-25T01:22:57.853566+0000","last_became_peered":"2024-04-25T01:22:57.853566+0000","last_unstale":"2024-04-25T01:23:47.002385+0000","last_undegraded":"2024-04-25T01:23:47.002385+0000","last_fullsized":"2024-04-25T01:23:47.002385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:36.509650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.038657+0000","last_change":"2024-04-25T01:23:47.038657+0000","last_active":"2024-04-25T01:23:47.038657+0000","last_peered":"2024-04-25T01:23:47.038657+0000","last_clean":"2024-04-25T01:23:47.038657+0000","last_became_active":"2024-04-25T01:23:47.038168+0000","last_became_peered":"2024-04-25T01:23:47.038168+0000","last_unstale":"2024-04-25T01:23:47.038657+0000","last_undegraded":"2024-04-25T01:23:47.038657+0000","last_fullsized":"2024-04-25T01:23:47.038657+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:59.844333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.486808+0000","last_change":"2024-04-25T01:22:56.828651+0000","last_active":"2024-04-25T01:23:47.486808+0000","last_peered":"2024-04-25T01:23:47.486808+0000","last_clean":"2024-04-25T01:23:47.486808+0000","last_became_active":"2024-04-25T01:22:56.828186+0000","last_became_peered":"2024-04-25T01:22:56.828186+0000","last_unstale":"2024-04-25T01:23:47.486808+0000","last_undegraded":"2024-04-25T01:23:47.486808+0000","last_fullsized":"2024-04-25T01:23:47.486808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:56.020822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.001971+0000","last_change":"2024-04-25T01:22:57.849768+0000","last_active":"2024-04-25T01:23:47.001971+0000","last_peered":"2024-04-25T01:23:47.001971+0000","last_clean":"2024-04-25T01:23:47.001971+0000","last_became_active":"2024-04-25T01:22:57.849418+0000","last_became_peered":"2024-04-25T01:22:57.849418+0000","last_unstale":"2024-04-25T01:23:47.001971+0000","last_undegraded":"2024-04-25T01:23:47.001971+0000","last_fullsized":"2024-04-25T01:23:47.001971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:41.292915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.149351+0000","last_change":"2024-04-25T01:23:47.148790+0000","last_active":"2024-04-25T01:23:47.149351+0000","last_peered":"2024-04-25T01:23:47.149351+0000","last_clean":"2024-04-25T01:23:47.149351+0000","last_became_active":"2024-04-25T01:23:47.148180+0000","last_became_peered":"2024-04-25T01:23:47.148180+0000","last_unstale":"2024-04-25T01:23:47.149351+0000","last_undegraded":"2024-04-25T01:23:47.149351+0000","last_fullsized":"2024-04-25T01:23:47.149351+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:08.296696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147048+0000","last_change":"2024-04-25T01:22:56.844672+0000","last_active":"2024-04-25T01:23:47.147048+0000","last_peered":"2024-04-25T01:23:47.147048+0000","last_clean":"2024-04-25T01:23:47.147048+0000","last_became_active":"2024-04-25T01:22:56.844141+0000","last_became_peered":"2024-04-25T01:22:56.844141+0000","last_unstale":"2024-04-25T01:23:47.147048+0000","last_undegraded":"2024-04-25T01:23:47.147048+0000","last_fullsized":"2024-04-25T01:23:47.147048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:14.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004864+0000","last_change":"2024-04-25T01:22:57.845006+0000","last_active":"2024-04-25T01:23:47.004864+0000","last_peered":"2024-04-25T01:23:47.004864+0000","last_clean":"2024-04-25T01:23:47.004864+0000","last_became_active":"2024-04-25T01:22:57.844672+0000","last_became_peered":"2024-04-25T01:22:57.844672+0000","last_unstale":"2024-04-25T01:23:47.004864+0000","last_undegraded":"2024-04-25T01:23:47.004864+0000","last_fullsized":"2024-04-25T01:23:47.004864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:18.238512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003702+0000","last_change":"2024-04-25T01:23:01.869438+0000","last_active":"2024-04-25T01:23:47.003702+0000","last_peered":"2024-04-25T01:23:47.003702+0000","last_clean":"2024-04-25T01:23:47.003702+0000","last_became_active":"2024-04-25T01:23:01.868985+0000","last_became_peered":"2024-04-25T01:23:01.868985+0000","last_unstale":"2024-04-25T01:23:47.003702+0000","last_undegraded":"2024-04-25T01:23:47.003702+0000","last_fullsized":"2024-04-25T01:23:47.003702+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:35.509098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000742+0000","last_change":"2024-04-25T01:22:56.834475+0000","last_active":"2024-04-25T01:23:47.000742+0000","last_peered":"2024-04-25T01:23:47.000742+0000","last_clean":"2024-04-25T01:23:47.000742+0000","last_became_active":"2024-04-25T01:22:56.834076+0000","last_became_peered":"2024-04-25T01:22:56.834076+0000","last_unstale":"2024-04-25T01:23:47.000742+0000","last_undegraded":"2024-04-25T01:23:47.000742+0000","last_fullsized":"2024-04-25T01:23:47.000742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:55:47.464009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147329+0000","last_change":"2024-04-25T01:22:57.860549+0000","last_active":"2024-04-25T01:23:47.147329+0000","last_peered":"2024-04-25T01:23:47.147329+0000","last_clean":"2024-04-25T01:23:47.147329+0000","last_became_active":"2024-04-25T01:22:57.860226+0000","last_became_peered":"2024-04-25T01:22:57.860226+0000","last_unstale":"2024-04-25T01:23:47.147329+0000","last_undegraded":"2024-04-25T01:23:47.147329+0000","last_fullsized":"2024-04-25T01:23:47.147329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:10.663508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002194+0000","last_change":"2024-04-25T01:23:01.867602+0000","last_active":"2024-04-25T01:23:47.002194+0000","last_peered":"2024-04-25T01:23:47.002194+0000","last_clean":"2024-04-25T01:23:47.002194+0000","last_became_active":"2024-04-25T01:23:01.867150+0000","last_became_peered":"2024-04-25T01:23:01.867150+0000","last_unstale":"2024-04-25T01:23:47.002194+0000","last_undegraded":"2024-04-25T01:23:47.002194+0000","last_fullsized":"2024-04-25T01:23:47.002194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:34.571031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002827+0000","last_change":"2024-04-25T01:22:56.841082+0000","last_active":"2024-04-25T01:23:47.002827+0000","last_peered":"2024-04-25T01:23:47.002827+0000","last_clean":"2024-04-25T01:23:47.002827+0000","last_became_active":"2024-04-25T01:22:56.839856+0000","last_became_peered":"2024-04-25T01:22:56.839856+0000","last_unstale":"2024-04-25T01:23:47.002827+0000","last_undegraded":"2024-04-25T01:23:47.002827+0000","last_fullsized":"2024-04-25T01:23:47.002827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:14.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487775+0000","last_change":"2024-04-25T01:22:57.852242+0000","last_active":"2024-04-25T01:23:47.487775+0000","last_peered":"2024-04-25T01:23:47.487775+0000","last_clean":"2024-04-25T01:23:47.487775+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:23:47.487775+0000","last_undegraded":"2024-04-25T01:23:47.487775+0000","last_fullsized":"2024-04-25T01:23:47.487775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:38:02.810025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.486894+0000","last_change":"2024-04-25T01:23:01.871074+0000","last_active":"2024-04-25T01:23:47.486894+0000","last_peered":"2024-04-25T01:23:47.486894+0000","last_clean":"2024-04-25T01:23:47.486894+0000","last_became_active":"2024-04-25T01:23:01.870570+0000","last_became_peered":"2024-04-25T01:23:01.870570+0000","last_unstale":"2024-04-25T01:23:47.486894+0000","last_undegraded":"2024-04-25T01:23:47.486894+0000","last_fullsized":"2024-04-25T01:23:47.486894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:14.476400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.011645+0000","last_change":"2024-04-25T01:23:47.011202+0000","last_active":"2024-04-25T01:23:47.011645+0000","last_peered":"2024-04-25T01:23:47.011645+0000","last_clean":"2024-04-25T01:23:47.011645+0000","last_became_active":"2024-04-25T01:23:47.006425+0000","last_became_peered":"2024-04-25T01:23:47.006425+0000","last_unstale":"2024-04-25T01:23:47.011645+0000","last_undegraded":"2024-04-25T01:23:47.011645+0000","last_fullsized":"2024-04-25T01:23:47.011645+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:51:46.791184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488241+0000","last_change":"2024-04-25T01:22:57.843173+0000","last_active":"2024-04-25T01:23:47.488241+0000","last_peered":"2024-04-25T01:23:47.488241+0000","last_clean":"2024-04-25T01:23:47.488241+0000","last_became_active":"2024-04-25T01:22:57.842734+0000","last_became_peered":"2024-04-25T01:22:57.842734+0000","last_unstale":"2024-04-25T01:23:47.488241+0000","last_undegraded":"2024-04-25T01:23:47.488241+0000","last_fullsized":"2024-04-25T01:23:47.488241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:33.197493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002039+0000","last_change":"2024-04-25T01:23:01.864068+0000","last_active":"2024-04-25T01:23:47.002039+0000","last_peered":"2024-04-25T01:23:47.002039+0000","last_clean":"2024-04-25T01:23:47.002039+0000","last_became_active":"2024-04-25T01:23:01.863640+0000","last_became_peered":"2024-04-25T01:23:01.863640+0000","last_unstale":"2024-04-25T01:23:47.002039+0000","last_undegraded":"2024-04-25T01:23:47.002039+0000","last_fullsized":"2024-04-25T01:23:47.002039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:26.925012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037647+0000","last_change":"2024-04-25T01:22:56.837969+0000","last_active":"2024-04-25T01:23:47.037647+0000","last_peered":"2024-04-25T01:23:47.037647+0000","last_clean":"2024-04-25T01:23:47.037647+0000","last_became_active":"2024-04-25T01:22:56.837641+0000","last_became_peered":"2024-04-25T01:22:56.837641+0000","last_unstale":"2024-04-25T01:23:47.037647+0000","last_undegraded":"2024-04-25T01:23:47.037647+0000","last_fullsized":"2024-04-25T01:23:47.037647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:04.376038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487604+0000","last_change":"2024-04-25T01:22:57.851057+0000","last_active":"2024-04-25T01:23:47.487604+0000","last_peered":"2024-04-25T01:23:47.487604+0000","last_clean":"2024-04-25T01:23:47.487604+0000","last_became_active":"2024-04-25T01:22:57.850278+0000","last_became_peered":"2024-04-25T01:22:57.850278+0000","last_unstale":"2024-04-25T01:23:47.487604+0000","last_undegraded":"2024-04-25T01:23:47.487604+0000","last_fullsized":"2024-04-25T01:23:47.487604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:15.332193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036445+0000","last_change":"2024-04-25T01:23:01.866673+0000","last_active":"2024-04-25T01:23:47.036445+0000","last_peered":"2024-04-25T01:23:47.036445+0000","last_clean":"2024-04-25T01:23:47.036445+0000","last_became_active":"2024-04-25T01:23:01.866210+0000","last_became_peered":"2024-04-25T01:23:01.866210+0000","last_unstale":"2024-04-25T01:23:47.036445+0000","last_undegraded":"2024-04-25T01:23:47.036445+0000","last_fullsized":"2024-04-25T01:23:47.036445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:20.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487558+0000","last_change":"2024-04-25T01:22:56.844590+0000","last_active":"2024-04-25T01:23:47.487558+0000","last_peered":"2024-04-25T01:23:47.487558+0000","last_clean":"2024-04-25T01:23:47.487558+0000","last_became_active":"2024-04-25T01:22:56.844279+0000","last_became_peered":"2024-04-25T01:22:56.844279+0000","last_unstale":"2024-04-25T01:23:47.487558+0000","last_undegraded":"2024-04-25T01:23:47.487558+0000","last_fullsized":"2024-04-25T01:23:47.487558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:47.155832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487881+0000","last_change":"2024-04-25T01:22:57.858850+0000","last_active":"2024-04-25T01:23:47.487881+0000","last_peered":"2024-04-25T01:23:47.487881+0000","last_clean":"2024-04-25T01:23:47.487881+0000","last_became_active":"2024-04-25T01:22:57.858498+0000","last_became_peered":"2024-04-25T01:22:57.858498+0000","last_unstale":"2024-04-25T01:23:47.487881+0000","last_undegraded":"2024-04-25T01:23:47.487881+0000","last_fullsized":"2024-04-25T01:23:47.487881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:04.836062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.007111+0000","last_change":"2024-04-25T01:23:47.007111+0000","last_active":"2024-04-25T01:23:47.007111+0000","last_peered":"2024-04-25T01:23:47.007111+0000","last_clean":"2024-04-25T01:23:47.007111+0000","last_became_active":"2024-04-25T01:23:47.006699+0000","last_became_peered":"2024-04-25T01:23:47.006699+0000","last_unstale":"2024-04-25T01:23:47.007111+0000","last_undegraded":"2024-04-25T01:23:47.007111+0000","last_fullsized":"2024-04-25T01:23:47.007111+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:57:43.608183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488742+0000","last_change":"2024-04-25T01:23:47.488742+0000","last_active":"2024-04-25T01:23:47.488742+0000","last_peered":"2024-04-25T01:23:47.488742+0000","last_clean":"2024-04-25T01:23:47.488742+0000","last_became_active":"2024-04-25T01:23:47.488193+0000","last_became_peered":"2024-04-25T01:23:47.488193+0000","last_unstale":"2024-04-25T01:23:47.488742+0000","last_undegraded":"2024-04-25T01:23:47.488742+0000","last_fullsized":"2024-04-25T01:23:47.488742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:02.658395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000673+0000","last_change":"2024-04-25T01:22:57.851138+0000","last_active":"2024-04-25T01:23:47.000673+0000","last_peered":"2024-04-25T01:23:47.000673+0000","last_clean":"2024-04-25T01:23:47.000673+0000","last_became_active":"2024-04-25T01:22:57.849107+0000","last_became_peered":"2024-04-25T01:22:57.849107+0000","last_unstale":"2024-04-25T01:23:47.000673+0000","last_undegraded":"2024-04-25T01:23:47.000673+0000","last_fullsized":"2024-04-25T01:23:47.000673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:47.665356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488162+0000","last_change":"2024-04-25T01:23:47.488162+0000","last_active":"2024-04-25T01:23:47.488162+0000","last_peered":"2024-04-25T01:23:47.488162+0000","last_clean":"2024-04-25T01:23:47.488162+0000","last_became_active":"2024-04-25T01:23:47.487731+0000","last_became_peered":"2024-04-25T01:23:47.487731+0000","last_unstale":"2024-04-25T01:23:47.488162+0000","last_undegraded":"2024-04-25T01:23:47.488162+0000","last_fullsized":"2024-04-25T01:23:47.488162+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:14.716313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487400+0000","last_change":"2024-04-25T01:22:57.851715+0000","last_active":"2024-04-25T01:23:47.487400+0000","last_peered":"2024-04-25T01:23:47.487400+0000","last_clean":"2024-04-25T01:23:47.487400+0000","last_became_active":"2024-04-25T01:22:57.847287+0000","last_became_peered":"2024-04-25T01:22:57.847287+0000","last_unstale":"2024-04-25T01:23:47.487400+0000","last_undegraded":"2024-04-25T01:23:47.487400+0000","last_fullsized":"2024-04-25T01:23:47.487400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.749931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004048+0000","last_change":"2024-04-25T01:22:57.846075+0000","last_active":"2024-04-25T01:23:47.004048+0000","last_peered":"2024-04-25T01:23:47.004048+0000","last_clean":"2024-04-25T01:23:47.004048+0000","last_became_active":"2024-04-25T01:22:57.845388+0000","last_became_peered":"2024-04-25T01:22:57.845388+0000","last_unstale":"2024-04-25T01:23:47.004048+0000","last_undegraded":"2024-04-25T01:23:47.004048+0000","last_fullsized":"2024-04-25T01:23:47.004048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:19.859806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.001611+0000","last_change":"2024-04-25T01:22:57.851446+0000","last_active":"2024-04-25T01:23:47.001611+0000","last_peered":"2024-04-25T01:23:47.001611+0000","last_clean":"2024-04-25T01:23:47.001611+0000","last_became_active":"2024-04-25T01:22:57.848729+0000","last_became_peered":"2024-04-25T01:22:57.848729+0000","last_unstale":"2024-04-25T01:23:47.001611+0000","last_undegraded":"2024-04-25T01:23:47.001611+0000","last_fullsized":"2024-04-25T01:23:47.001611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:49.721024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147722+0000","last_change":"2024-04-25T01:22:57.847598+0000","last_active":"2024-04-25T01:23:47.147722+0000","last_peered":"2024-04-25T01:23:47.147722+0000","last_clean":"2024-04-25T01:23:47.147722+0000","last_became_active":"2024-04-25T01:22:57.847232+0000","last_became_peered":"2024-04-25T01:22:57.847232+0000","last_unstale":"2024-04-25T01:23:47.147722+0000","last_undegraded":"2024-04-25T01:23:47.147722+0000","last_fullsized":"2024-04-25T01:23:47.147722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.227700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036924+0000","last_change":"2024-04-25T01:22:57.842881+0000","last_active":"2024-04-25T01:23:47.036924+0000","last_peered":"2024-04-25T01:23:47.036924+0000","last_clean":"2024-04-25T01:23:47.036924+0000","last_became_active":"2024-04-25T01:22:57.842108+0000","last_became_peered":"2024-04-25T01:22:57.842108+0000","last_unstale":"2024-04-25T01:23:47.036924+0000","last_undegraded":"2024-04-25T01:23:47.036924+0000","last_fullsized":"2024-04-25T01:23:47.036924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:53.208188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147038+0000","last_change":"2024-04-25T01:22:57.860047+0000","last_active":"2024-04-25T01:23:47.147038+0000","last_peered":"2024-04-25T01:23:47.147038+0000","last_clean":"2024-04-25T01:23:47.147038+0000","last_became_active":"2024-04-25T01:22:57.859702+0000","last_became_peered":"2024-04-25T01:22:57.859702+0000","last_unstale":"2024-04-25T01:23:47.147038+0000","last_undegraded":"2024-04-25T01:23:47.147038+0000","last_fullsized":"2024-04-25T01:23:47.147038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:00.684055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487044+0000","last_change":"2024-04-25T01:22:57.847996+0000","last_active":"2024-04-25T01:23:47.487044+0000","last_peered":"2024-04-25T01:23:47.487044+0000","last_clean":"2024-04-25T01:23:47.487044+0000","last_became_active":"2024-04-25T01:22:57.847561+0000","last_became_peered":"2024-04-25T01:22:57.847561+0000","last_unstale":"2024-04-25T01:23:47.487044+0000","last_undegraded":"2024-04-25T01:23:47.487044+0000","last_fullsized":"2024-04-25T01:23:47.487044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:16.327558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487311+0000","last_change":"2024-04-25T01:22:57.845216+0000","last_active":"2024-04-25T01:23:47.487311+0000","last_peered":"2024-04-25T01:23:47.487311+0000","last_clean":"2024-04-25T01:23:47.487311+0000","last_became_active":"2024-04-25T01:22:57.844883+0000","last_became_peered":"2024-04-25T01:22:57.844883+0000","last_unstale":"2024-04-25T01:23:47.487311+0000","last_undegraded":"2024-04-25T01:23:47.487311+0000","last_fullsized":"2024-04-25T01:23:47.487311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:33.782048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005024+0000","last_change":"2024-04-25T01:22:57.853251+0000","last_active":"2024-04-25T01:23:47.005024+0000","last_peered":"2024-04-25T01:23:47.005024+0000","last_clean":"2024-04-25T01:23:47.005024+0000","last_became_active":"2024-04-25T01:22:57.852878+0000","last_became_peered":"2024-04-25T01:22:57.852878+0000","last_unstale":"2024-04-25T01:23:47.005024+0000","last_undegraded":"2024-04-25T01:23:47.005024+0000","last_fullsized":"2024-04-25T01:23:47.005024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:42.487837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281206,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67357,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":448,"num_read_kb":294,"num_write":161,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":202071,"data_compressed":4881,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":500815,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1138688,"data_stored":1151439,"data_compressed":58064,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":505147,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":500971,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":1096880,"data_compressed":56437,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":496227,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":559493,"data_compressed":22299,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":497615,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":560652,"data_compressed":22299,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496421,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":501459,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1069056,"data_stored":1092468,"data_compressed":56437,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64515,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64934,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:24:39.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:24:39.461 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:39.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:39 smithi027 ceph-mon[29197]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:39.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:39 smithi153 ceph-mon[32621]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:39.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:39 smithi139 ceph-mon[32760]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:40.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:40 smithi027 ceph-mon[29197]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:40.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:40 smithi153 ceph-mon[32621]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:40.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:40 smithi139 ceph-mon[32760]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:24:41.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 -- 172.21.15.27:0/1023531230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4101120 msgr2=0x7f48b41015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1023531230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4101120 0x7f48b41015a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009990 tx=0x7f48a8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 -- 172.21.15.27:0/1023531230 shutdown_connections 2024-04-25T01:24:41.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1023531230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b4101ae0 0x7f48b410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1023531230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b4107190 0x7f48b4107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1023531230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4101120 0x7f48b41015a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 -- 172.21.15.27:0/1023531230 >> 172.21.15.27:0/1023531230 conn(0x7f48b40780a0 msgr2=0x7f48b40784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:41.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 -- 172.21.15.27:0/1023531230 shutdown_connections 2024-04-25T01:24:41.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.362+0000 7f48bb03c700 1 -- 172.21.15.27:0/1023531230 wait complete. 2024-04-25T01:24:41.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48bb03c700 1 Processor -- start 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48bb03c700 1 -- start start 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48bb03c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b4101120 0x7f48b419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48bb03c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b4101ae0 0x7f48b419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48bb03c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4107190 0x7f48b41987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48bb03c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f48b4111480 con 0x7f48b4107190 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48bb03c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f48b4111300 con 0x7f48b4101120 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48bb03c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48b4111600 con 0x7f48b4101ae0 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48b95d9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4107190 0x7f48b41987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48b95d9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4107190 0x7f48b41987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49666/0 (socket says 172.21.15.27:49666) 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.363+0000 7f48b95d9700 1 -- 172.21.15.27:0/1829326035 learned_addr learned my addr 172.21.15.27:0/1829326035 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b3fff700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b4101ae0 0x7f48b419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b8dd8700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b4101120 0x7f48b419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:41.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b95d9700 1 -- 172.21.15.27:0/1829326035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b4101ae0 msgr2=0x7f48b419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b95d9700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b4101ae0 0x7f48b419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b95d9700 1 -- 172.21.15.27:0/1829326035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b4101120 msgr2=0x7f48b419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b95d9700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b4101120 0x7f48b419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b95d9700 1 -- 172.21.15.27:0/1829326035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b4198fc0 con 0x7f48b4107190 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b3fff700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b4101ae0 0x7f48b419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b95d9700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4107190 0x7f48b41987e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f48ac007ee0 tx=0x7f48ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b1ffb700 1 -- 172.21.15.27:0/1829326035 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac00dbd0 con 0x7f48b4107190 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b1ffb700 1 -- 172.21.15.27:0/1829326035 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48ac00dd70 con 0x7f48b4107190 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b41992b0 con 0x7f48b4107190 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48b1ffb700 1 -- 172.21.15.27:0/1829326035 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac0133f0 con 0x7f48b4107190 2024-04-25T01:24:41.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.364+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b41a54c0 con 0x7f48b4107190 2024-04-25T01:24:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.365+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48b4102840 con 0x7f48b4107190 2024-04-25T01:24:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.367+0000 7f48b1ffb700 1 -- 172.21.15.27:0/1829326035 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f48ac01a020 con 0x7f48b4107190 2024-04-25T01:24:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.367+0000 7f48b1ffb700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f489c0a2d30 0x7f489c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.367+0000 7f48b1ffb700 1 -- 172.21.15.27:0/1829326035 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f48ac0cf350 con 0x7f48b4107190 2024-04-25T01:24:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.367+0000 7f48b8dd8700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f489c0a2d30 0x7f489c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:41.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.368+0000 7f48b8dd8700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f489c0a2d30 0x7f489c0a51f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f48a4006fd0 tx=0x7f48a4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:41.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.371+0000 7f48b1ffb700 1 -- 172.21.15.27:0/1829326035 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f48ac010070 con 0x7f48b4107190 2024-04-25T01:24:41.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.572+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f48b4104f40 con 0x7f489c0a2d30 2024-04-25T01:24:41.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.582+0000 7f48b1ffb700 1 -- 172.21.15.27:0/1829326035 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340668 (secure 0 0 0) 0x7f48b4104f40 con 0x7f489c0a2d30 2024-04-25T01:24:41.567 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:41.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f489c0a2d30 msgr2=0x7f489c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f489c0a2d30 0x7f489c0a51f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f48a4006fd0 tx=0x7f48a4008040 comp rx=0 tx=0).stop 2024-04-25T01:24:41.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4107190 msgr2=0x7f48b41987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4107190 0x7f48b41987e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f48ac007ee0 tx=0x7f48ac00d3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 shutdown_connections 2024-04-25T01:24:41.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f489c0a2d30 0x7f489c0a51f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b4107190 0x7f48b41987e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b4101120 0x7f48b419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 --2- 172.21.15.27:0/1829326035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b4101ae0 0x7f48b419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.585+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 >> 172.21.15.27:0/1829326035 conn(0x7f48b40780a0 msgr2=0x7f48b4100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:41.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.586+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 shutdown_connections 2024-04-25T01:24:41.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:41.586+0000 7f48bb03c700 1 -- 172.21.15.27:0/1829326035 wait complete. 2024-04-25T01:24:41.570 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T01:24:42.399 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-25T01:24:41.400922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657959,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":609,"num_read_kb":428,"num_write":344,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361356,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580468,"statfs":{"total":1151940427776,"available":1151570399232,"internally_reserved":0,"allocated":12443648,"data_stored":7959587,"data_compressed":360240,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003472"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005231+0000","last_change":"2024-04-25T01:22:57.847700+0000","last_active":"2024-04-25T01:23:47.005231+0000","last_peered":"2024-04-25T01:23:47.005231+0000","last_clean":"2024-04-25T01:23:47.005231+0000","last_became_active":"2024-04-25T01:22:57.846877+0000","last_became_peered":"2024-04-25T01:22:57.846877+0000","last_unstale":"2024-04-25T01:23:47.005231+0000","last_undegraded":"2024-04-25T01:23:47.005231+0000","last_fullsized":"2024-04-25T01:23:47.005231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:05.918261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002513+0000","last_change":"2024-04-25T01:22:57.848396+0000","last_active":"2024-04-25T01:23:47.002513+0000","last_peered":"2024-04-25T01:23:47.002513+0000","last_clean":"2024-04-25T01:23:47.002513+0000","last_became_active":"2024-04-25T01:22:57.847971+0000","last_became_peered":"2024-04-25T01:22:57.847971+0000","last_unstale":"2024-04-25T01:23:47.002513+0000","last_undegraded":"2024-04-25T01:23:47.002513+0000","last_fullsized":"2024-04-25T01:23:47.002513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:13.143828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002268+0000","last_change":"2024-04-25T01:22:57.856908+0000","last_active":"2024-04-25T01:23:47.002268+0000","last_peered":"2024-04-25T01:23:47.002268+0000","last_clean":"2024-04-25T01:23:47.002268+0000","last_became_active":"2024-04-25T01:22:57.856218+0000","last_became_peered":"2024-04-25T01:22:57.856218+0000","last_unstale":"2024-04-25T01:23:47.002268+0000","last_undegraded":"2024-04-25T01:23:47.002268+0000","last_fullsized":"2024-04-25T01:23:47.002268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:51.971222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002971+0000","last_change":"2024-04-25T01:22:57.849022+0000","last_active":"2024-04-25T01:23:47.002971+0000","last_peered":"2024-04-25T01:23:47.002971+0000","last_clean":"2024-04-25T01:23:47.002971+0000","last_became_active":"2024-04-25T01:22:57.848303+0000","last_became_peered":"2024-04-25T01:22:57.848303+0000","last_unstale":"2024-04-25T01:23:47.002971+0000","last_undegraded":"2024-04-25T01:23:47.002971+0000","last_fullsized":"2024-04-25T01:23:47.002971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:52.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488312+0000","last_change":"2024-04-25T01:23:47.488312+0000","last_active":"2024-04-25T01:23:47.488312+0000","last_peered":"2024-04-25T01:23:47.488312+0000","last_clean":"2024-04-25T01:23:47.488312+0000","last_became_active":"2024-04-25T01:23:47.487923+0000","last_became_peered":"2024-04-25T01:23:47.487923+0000","last_unstale":"2024-04-25T01:23:47.488312+0000","last_undegraded":"2024-04-25T01:23:47.488312+0000","last_fullsized":"2024-04-25T01:23:47.488312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:30.276853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002739+0000","last_change":"2024-04-25T01:22:57.855733+0000","last_active":"2024-04-25T01:23:47.002739+0000","last_peered":"2024-04-25T01:23:47.002739+0000","last_clean":"2024-04-25T01:23:47.002739+0000","last_became_active":"2024-04-25T01:22:57.855075+0000","last_became_peered":"2024-04-25T01:22:57.855075+0000","last_unstale":"2024-04-25T01:23:47.002739+0000","last_undegraded":"2024-04-25T01:23:47.002739+0000","last_fullsized":"2024-04-25T01:23:47.002739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:24.563461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487370+0000","last_change":"2024-04-25T01:22:57.849693+0000","last_active":"2024-04-25T01:23:47.487370+0000","last_peered":"2024-04-25T01:23:47.487370+0000","last_clean":"2024-04-25T01:23:47.487370+0000","last_became_active":"2024-04-25T01:22:57.849324+0000","last_became_peered":"2024-04-25T01:22:57.849324+0000","last_unstale":"2024-04-25T01:23:47.487370+0000","last_undegraded":"2024-04-25T01:23:47.487370+0000","last_fullsized":"2024-04-25T01:23:47.487370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:09.936496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147048+0000","last_change":"2024-04-25T01:22:57.847962+0000","last_active":"2024-04-25T01:23:47.147048+0000","last_peered":"2024-04-25T01:23:47.147048+0000","last_clean":"2024-04-25T01:23:47.147048+0000","last_became_active":"2024-04-25T01:22:57.847592+0000","last_became_peered":"2024-04-25T01:22:57.847592+0000","last_unstale":"2024-04-25T01:23:47.147048+0000","last_undegraded":"2024-04-25T01:23:47.147048+0000","last_fullsized":"2024-04-25T01:23:47.147048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:45.594118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487589+0000","last_change":"2024-04-25T01:22:57.860711+0000","last_active":"2024-04-25T01:23:47.487589+0000","last_peered":"2024-04-25T01:23:47.487589+0000","last_clean":"2024-04-25T01:23:47.487589+0000","last_became_active":"2024-04-25T01:22:57.860393+0000","last_became_peered":"2024-04-25T01:22:57.860393+0000","last_unstale":"2024-04-25T01:23:47.487589+0000","last_undegraded":"2024-04-25T01:23:47.487589+0000","last_fullsized":"2024-04-25T01:23:47.487589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:31.068095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004059+0000","last_change":"2024-04-25T01:22:57.853741+0000","last_active":"2024-04-25T01:23:47.004059+0000","last_peered":"2024-04-25T01:23:47.004059+0000","last_clean":"2024-04-25T01:23:47.004059+0000","last_became_active":"2024-04-25T01:22:57.851015+0000","last_became_peered":"2024-04-25T01:22:57.851015+0000","last_unstale":"2024-04-25T01:23:47.004059+0000","last_undegraded":"2024-04-25T01:23:47.004059+0000","last_fullsized":"2024-04-25T01:23:47.004059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:45.831335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488219+0000","last_change":"2024-04-25T01:22:57.853354+0000","last_active":"2024-04-25T01:23:47.488219+0000","last_peered":"2024-04-25T01:23:47.488219+0000","last_clean":"2024-04-25T01:23:47.488219+0000","last_became_active":"2024-04-25T01:22:57.853043+0000","last_became_peered":"2024-04-25T01:22:57.853043+0000","last_unstale":"2024-04-25T01:23:47.488219+0000","last_undegraded":"2024-04-25T01:23:47.488219+0000","last_fullsized":"2024-04-25T01:23:47.488219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:54.378751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003089+0000","last_change":"2024-04-25T01:22:57.855542+0000","last_active":"2024-04-25T01:23:47.003089+0000","last_peered":"2024-04-25T01:23:47.003089+0000","last_clean":"2024-04-25T01:23:47.003089+0000","last_became_active":"2024-04-25T01:22:57.855196+0000","last_became_peered":"2024-04-25T01:22:57.855196+0000","last_unstale":"2024-04-25T01:23:47.003089+0000","last_undegraded":"2024-04-25T01:23:47.003089+0000","last_fullsized":"2024-04-25T01:23:47.003089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:56.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487277+0000","last_change":"2024-04-25T01:22:57.829353+0000","last_active":"2024-04-25T01:23:47.487277+0000","last_peered":"2024-04-25T01:23:47.487277+0000","last_clean":"2024-04-25T01:23:47.487277+0000","last_became_active":"2024-04-25T01:22:57.828937+0000","last_became_peered":"2024-04-25T01:22:57.828937+0000","last_unstale":"2024-04-25T01:23:47.487277+0000","last_undegraded":"2024-04-25T01:23:47.487277+0000","last_fullsized":"2024-04-25T01:23:47.487277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.541556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002018+0000","last_change":"2024-04-25T01:22:57.854635+0000","last_active":"2024-04-25T01:23:47.002018+0000","last_peered":"2024-04-25T01:23:47.002018+0000","last_clean":"2024-04-25T01:23:47.002018+0000","last_became_active":"2024-04-25T01:22:57.854292+0000","last_became_peered":"2024-04-25T01:22:57.854292+0000","last_unstale":"2024-04-25T01:23:47.002018+0000","last_undegraded":"2024-04-25T01:23:47.002018+0000","last_fullsized":"2024-04-25T01:23:47.002018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:48.209059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002065+0000","last_change":"2024-04-25T01:22:57.852210+0000","last_active":"2024-04-25T01:23:47.002065+0000","last_peered":"2024-04-25T01:23:47.002065+0000","last_clean":"2024-04-25T01:23:47.002065+0000","last_became_active":"2024-04-25T01:22:57.851833+0000","last_became_peered":"2024-04-25T01:22:57.851833+0000","last_unstale":"2024-04-25T01:23:47.002065+0000","last_undegraded":"2024-04-25T01:23:47.002065+0000","last_fullsized":"2024-04-25T01:23:47.002065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:14.664785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.486765+0000","last_change":"2024-04-25T01:22:57.849357+0000","last_active":"2024-04-25T01:23:47.486765+0000","last_peered":"2024-04-25T01:23:47.486765+0000","last_clean":"2024-04-25T01:23:47.486765+0000","last_became_active":"2024-04-25T01:22:57.849050+0000","last_became_peered":"2024-04-25T01:22:57.849050+0000","last_unstale":"2024-04-25T01:23:47.486765+0000","last_undegraded":"2024-04-25T01:23:47.486765+0000","last_fullsized":"2024-04-25T01:23:47.486765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:37.669424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005156+0000","last_change":"2024-04-25T01:22:57.844790+0000","last_active":"2024-04-25T01:23:47.005156+0000","last_peered":"2024-04-25T01:23:47.005156+0000","last_clean":"2024-04-25T01:23:47.005156+0000","last_became_active":"2024-04-25T01:22:57.844442+0000","last_became_peered":"2024-04-25T01:22:57.844442+0000","last_unstale":"2024-04-25T01:23:47.005156+0000","last_undegraded":"2024-04-25T01:23:47.005156+0000","last_fullsized":"2024-04-25T01:23:47.005156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:33.766034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999905+0000","last_change":"2024-04-25T01:22:57.844430+0000","last_active":"2024-04-25T01:23:46.999905+0000","last_peered":"2024-04-25T01:23:46.999905+0000","last_clean":"2024-04-25T01:23:46.999905+0000","last_became_active":"2024-04-25T01:22:57.844110+0000","last_became_peered":"2024-04-25T01:22:57.844110+0000","last_unstale":"2024-04-25T01:23:46.999905+0000","last_undegraded":"2024-04-25T01:23:46.999905+0000","last_fullsized":"2024-04-25T01:23:46.999905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:50.348249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002121+0000","last_change":"2024-04-25T01:22:57.852705+0000","last_active":"2024-04-25T01:23:47.002121+0000","last_peered":"2024-04-25T01:23:47.002121+0000","last_clean":"2024-04-25T01:23:47.002121+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:23:47.002121+0000","last_undegraded":"2024-04-25T01:23:47.002121+0000","last_fullsized":"2024-04-25T01:23:47.002121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:10.320459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.001099+0000","last_change":"2024-04-25T01:23:01.879341+0000","last_active":"2024-04-25T01:23:47.001099+0000","last_peered":"2024-04-25T01:23:47.001099+0000","last_clean":"2024-04-25T01:23:47.001099+0000","last_became_active":"2024-04-25T01:23:01.878881+0000","last_became_peered":"2024-04-25T01:23:01.878881+0000","last_unstale":"2024-04-25T01:23:47.001099+0000","last_undegraded":"2024-04-25T01:23:47.001099+0000","last_fullsized":"2024-04-25T01:23:47.001099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:46.036868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999778+0000","last_change":"2024-04-25T01:22:56.839867+0000","last_active":"2024-04-25T01:23:46.999778+0000","last_peered":"2024-04-25T01:23:46.999778+0000","last_clean":"2024-04-25T01:23:46.999778+0000","last_became_active":"2024-04-25T01:22:56.839521+0000","last_became_peered":"2024-04-25T01:22:56.839521+0000","last_unstale":"2024-04-25T01:23:46.999778+0000","last_undegraded":"2024-04-25T01:23:46.999778+0000","last_fullsized":"2024-04-25T01:23:46.999778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:00.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147533+0000","last_change":"2024-04-25T01:22:57.855922+0000","last_active":"2024-04-25T01:23:47.147533+0000","last_peered":"2024-04-25T01:23:47.147533+0000","last_clean":"2024-04-25T01:23:47.147533+0000","last_became_active":"2024-04-25T01:22:57.855584+0000","last_became_peered":"2024-04-25T01:22:57.855584+0000","last_unstale":"2024-04-25T01:23:47.147533+0000","last_undegraded":"2024-04-25T01:23:47.147533+0000","last_fullsized":"2024-04-25T01:23:47.147533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:02.791269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002057+0000","last_change":"2024-04-25T01:23:01.872340+0000","last_active":"2024-04-25T01:23:47.002057+0000","last_peered":"2024-04-25T01:23:47.002057+0000","last_clean":"2024-04-25T01:23:47.002057+0000","last_became_active":"2024-04-25T01:23:01.871900+0000","last_became_peered":"2024-04-25T01:23:01.871900+0000","last_unstale":"2024-04-25T01:23:47.002057+0000","last_undegraded":"2024-04-25T01:23:47.002057+0000","last_fullsized":"2024-04-25T01:23:47.002057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:11.918022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147541+0000","last_change":"2024-04-25T01:22:56.843851+0000","last_active":"2024-04-25T01:23:47.147541+0000","last_peered":"2024-04-25T01:23:47.147541+0000","last_clean":"2024-04-25T01:23:47.147541+0000","last_became_active":"2024-04-25T01:22:56.843232+0000","last_became_peered":"2024-04-25T01:22:56.843232+0000","last_unstale":"2024-04-25T01:23:47.147541+0000","last_undegraded":"2024-04-25T01:23:47.147541+0000","last_fullsized":"2024-04-25T01:23:47.147541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:13.758328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002472+0000","last_change":"2024-04-25T01:22:57.857686+0000","last_active":"2024-04-25T01:23:47.002472+0000","last_peered":"2024-04-25T01:23:47.002472+0000","last_clean":"2024-04-25T01:23:47.002472+0000","last_became_active":"2024-04-25T01:22:57.857175+0000","last_became_peered":"2024-04-25T01:22:57.857175+0000","last_unstale":"2024-04-25T01:23:47.002472+0000","last_undegraded":"2024-04-25T01:23:47.002472+0000","last_fullsized":"2024-04-25T01:23:47.002472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:24.777861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002310+0000","last_change":"2024-04-25T01:23:01.867914+0000","last_active":"2024-04-25T01:23:47.002310+0000","last_peered":"2024-04-25T01:23:47.002310+0000","last_clean":"2024-04-25T01:23:47.002310+0000","last_became_active":"2024-04-25T01:23:01.867103+0000","last_became_peered":"2024-04-25T01:23:01.867103+0000","last_unstale":"2024-04-25T01:23:47.002310+0000","last_undegraded":"2024-04-25T01:23:47.002310+0000","last_fullsized":"2024-04-25T01:23:47.002310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:26.425433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004565+0000","last_change":"2024-04-25T01:22:56.841310+0000","last_active":"2024-04-25T01:23:47.004565+0000","last_peered":"2024-04-25T01:23:47.004565+0000","last_clean":"2024-04-25T01:23:47.004565+0000","last_became_active":"2024-04-25T01:22:56.840882+0000","last_became_peered":"2024-04-25T01:22:56.840882+0000","last_unstale":"2024-04-25T01:23:47.004565+0000","last_undegraded":"2024-04-25T01:23:47.004565+0000","last_fullsized":"2024-04-25T01:23:47.004565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:29.971448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487298+0000","last_change":"2024-04-25T01:22:57.842810+0000","last_active":"2024-04-25T01:23:47.487298+0000","last_peered":"2024-04-25T01:23:47.487298+0000","last_clean":"2024-04-25T01:23:47.487298+0000","last_became_active":"2024-04-25T01:22:57.842457+0000","last_became_peered":"2024-04-25T01:22:57.842457+0000","last_unstale":"2024-04-25T01:23:47.487298+0000","last_undegraded":"2024-04-25T01:23:47.487298+0000","last_fullsized":"2024-04-25T01:23:47.487298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:45.512037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002379+0000","last_change":"2024-04-25T01:23:01.877298+0000","last_active":"2024-04-25T01:23:47.002379+0000","last_peered":"2024-04-25T01:23:47.002379+0000","last_clean":"2024-04-25T01:23:47.002379+0000","last_became_active":"2024-04-25T01:23:01.876874+0000","last_became_peered":"2024-04-25T01:23:01.876874+0000","last_unstale":"2024-04-25T01:23:47.002379+0000","last_undegraded":"2024-04-25T01:23:47.002379+0000","last_fullsized":"2024-04-25T01:23:47.002379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:35:08.221388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003630+0000","last_change":"2024-04-25T01:22:56.838914+0000","last_active":"2024-04-25T01:23:47.003630+0000","last_peered":"2024-04-25T01:23:47.003630+0000","last_clean":"2024-04-25T01:23:47.003630+0000","last_became_active":"2024-04-25T01:22:56.838616+0000","last_became_peered":"2024-04-25T01:22:56.838616+0000","last_unstale":"2024-04-25T01:23:47.003630+0000","last_undegraded":"2024-04-25T01:23:47.003630+0000","last_fullsized":"2024-04-25T01:23:47.003630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:20.761770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037545+0000","last_change":"2024-04-25T01:22:57.851210+0000","last_active":"2024-04-25T01:23:47.037545+0000","last_peered":"2024-04-25T01:23:47.037545+0000","last_clean":"2024-04-25T01:23:47.037545+0000","last_became_active":"2024-04-25T01:22:57.850867+0000","last_became_peered":"2024-04-25T01:22:57.850867+0000","last_unstale":"2024-04-25T01:23:47.037545+0000","last_undegraded":"2024-04-25T01:23:47.037545+0000","last_fullsized":"2024-04-25T01:23:47.037545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:15.998433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.489730+0000","last_change":"2024-04-25T01:23:47.489730+0000","last_active":"2024-04-25T01:23:47.489730+0000","last_peered":"2024-04-25T01:23:47.489730+0000","last_clean":"2024-04-25T01:23:47.489730+0000","last_became_active":"2024-04-25T01:23:47.489190+0000","last_became_peered":"2024-04-25T01:23:47.489190+0000","last_unstale":"2024-04-25T01:23:47.489730+0000","last_undegraded":"2024-04-25T01:23:47.489730+0000","last_fullsized":"2024-04-25T01:23:47.489730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.101073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:35.046632+0000","last_change":"2024-04-25T01:22:56.834741+0000","last_active":"2024-04-25T01:24:35.046632+0000","last_peered":"2024-04-25T01:24:35.046632+0000","last_clean":"2024-04-25T01:24:35.046632+0000","last_became_active":"2024-04-25T01:22:56.834296+0000","last_became_peered":"2024-04-25T01:22:56.834296+0000","last_unstale":"2024-04-25T01:24:35.046632+0000","last_undegraded":"2024-04-25T01:24:35.046632+0000","last_fullsized":"2024-04-25T01:24:35.046632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:38.940603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002414+0000","last_change":"2024-04-25T01:22:57.854646+0000","last_active":"2024-04-25T01:23:47.002414+0000","last_peered":"2024-04-25T01:23:47.002414+0000","last_clean":"2024-04-25T01:23:47.002414+0000","last_became_active":"2024-04-25T01:22:57.853839+0000","last_became_peered":"2024-04-25T01:22:57.853839+0000","last_unstale":"2024-04-25T01:23:47.002414+0000","last_undegraded":"2024-04-25T01:23:47.002414+0000","last_fullsized":"2024-04-25T01:23:47.002414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:36.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000919+0000","last_change":"2024-04-25T01:23:01.875478+0000","last_active":"2024-04-25T01:23:47.000919+0000","last_peered":"2024-04-25T01:23:47.000919+0000","last_clean":"2024-04-25T01:23:47.000919+0000","last_became_active":"2024-04-25T01:23:01.874643+0000","last_became_peered":"2024-04-25T01:23:01.874643+0000","last_unstale":"2024-04-25T01:23:47.000919+0000","last_undegraded":"2024-04-25T01:23:47.000919+0000","last_fullsized":"2024-04-25T01:23:47.000919+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.953507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002703+0000","last_change":"2024-04-25T01:22:56.843118+0000","last_active":"2024-04-25T01:23:47.002703+0000","last_peered":"2024-04-25T01:23:47.002703+0000","last_clean":"2024-04-25T01:23:47.002703+0000","last_became_active":"2024-04-25T01:22:56.842798+0000","last_became_peered":"2024-04-25T01:22:56.842798+0000","last_unstale":"2024-04-25T01:23:47.002703+0000","last_undegraded":"2024-04-25T01:23:47.002703+0000","last_fullsized":"2024-04-25T01:23:47.002703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:00.054913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037229+0000","last_change":"2024-04-25T01:22:57.850265+0000","last_active":"2024-04-25T01:23:47.037229+0000","last_peered":"2024-04-25T01:23:47.037229+0000","last_clean":"2024-04-25T01:23:47.037229+0000","last_became_active":"2024-04-25T01:22:57.849480+0000","last_became_peered":"2024-04-25T01:22:57.849480+0000","last_unstale":"2024-04-25T01:23:47.037229+0000","last_undegraded":"2024-04-25T01:23:47.037229+0000","last_fullsized":"2024-04-25T01:23:47.037229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:48.577252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005121+0000","last_change":"2024-04-25T01:23:01.865369+0000","last_active":"2024-04-25T01:23:47.005121+0000","last_peered":"2024-04-25T01:23:47.005121+0000","last_clean":"2024-04-25T01:23:47.005121+0000","last_became_active":"2024-04-25T01:23:01.864872+0000","last_became_peered":"2024-04-25T01:23:01.864872+0000","last_unstale":"2024-04-25T01:23:47.005121+0000","last_undegraded":"2024-04-25T01:23:47.005121+0000","last_fullsized":"2024-04-25T01:23:47.005121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:56.266514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487008+0000","last_change":"2024-04-25T01:22:56.842576+0000","last_active":"2024-04-25T01:23:47.487008+0000","last_peered":"2024-04-25T01:23:47.487008+0000","last_clean":"2024-04-25T01:23:47.487008+0000","last_became_active":"2024-04-25T01:22:56.841081+0000","last_became_peered":"2024-04-25T01:22:56.841081+0000","last_unstale":"2024-04-25T01:23:47.487008+0000","last_undegraded":"2024-04-25T01:23:47.487008+0000","last_fullsized":"2024-04-25T01:23:47.487008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:08.174426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.146949+0000","last_change":"2024-04-25T01:22:57.842812+0000","last_active":"2024-04-25T01:23:47.146949+0000","last_peered":"2024-04-25T01:23:47.146949+0000","last_clean":"2024-04-25T01:23:47.146949+0000","last_became_active":"2024-04-25T01:22:57.842462+0000","last_became_peered":"2024-04-25T01:22:57.842462+0000","last_unstale":"2024-04-25T01:23:47.146949+0000","last_undegraded":"2024-04-25T01:23:47.146949+0000","last_fullsized":"2024-04-25T01:23:47.146949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:14:13.444850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002382+0000","last_change":"2024-04-25T01:23:01.870401+0000","last_active":"2024-04-25T01:23:47.002382+0000","last_peered":"2024-04-25T01:23:47.002382+0000","last_clean":"2024-04-25T01:23:47.002382+0000","last_became_active":"2024-04-25T01:23:01.869932+0000","last_became_peered":"2024-04-25T01:23:01.869932+0000","last_unstale":"2024-04-25T01:23:47.002382+0000","last_undegraded":"2024-04-25T01:23:47.002382+0000","last_fullsized":"2024-04-25T01:23:47.002382+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:13.323209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036874+0000","last_change":"2024-04-25T01:22:56.835946+0000","last_active":"2024-04-25T01:23:47.036874+0000","last_peered":"2024-04-25T01:23:47.036874+0000","last_clean":"2024-04-25T01:23:47.036874+0000","last_became_active":"2024-04-25T01:22:56.835630+0000","last_became_peered":"2024-04-25T01:22:56.835630+0000","last_unstale":"2024-04-25T01:23:47.036874+0000","last_undegraded":"2024-04-25T01:23:47.036874+0000","last_fullsized":"2024-04-25T01:23:47.036874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:02.128275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999518+0000","last_change":"2024-04-25T01:22:57.844462+0000","last_active":"2024-04-25T01:23:46.999518+0000","last_peered":"2024-04-25T01:23:46.999518+0000","last_clean":"2024-04-25T01:23:46.999518+0000","last_became_active":"2024-04-25T01:22:57.843426+0000","last_became_peered":"2024-04-25T01:22:57.843426+0000","last_unstale":"2024-04-25T01:23:46.999518+0000","last_undegraded":"2024-04-25T01:23:46.999518+0000","last_fullsized":"2024-04-25T01:23:46.999518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:42.896622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147275+0000","last_change":"2024-04-25T01:23:01.871387+0000","last_active":"2024-04-25T01:23:47.147275+0000","last_peered":"2024-04-25T01:23:47.147275+0000","last_clean":"2024-04-25T01:23:47.147275+0000","last_became_active":"2024-04-25T01:23:01.870901+0000","last_became_peered":"2024-04-25T01:23:01.870901+0000","last_unstale":"2024-04-25T01:23:47.147275+0000","last_undegraded":"2024-04-25T01:23:47.147275+0000","last_fullsized":"2024-04-25T01:23:47.147275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:49.582515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:37.048654+0000","last_change":"2024-04-25T01:22:56.834440+0000","last_active":"2024-04-25T01:24:37.048654+0000","last_peered":"2024-04-25T01:24:37.048654+0000","last_clean":"2024-04-25T01:24:37.048654+0000","last_became_active":"2024-04-25T01:22:56.834116+0000","last_became_peered":"2024-04-25T01:22:56.834116+0000","last_unstale":"2024-04-25T01:24:37.048654+0000","last_undegraded":"2024-04-25T01:24:37.048654+0000","last_fullsized":"2024-04-25T01:24:37.048654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:01.176601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487920+0000","last_change":"2024-04-25T01:22:57.853560+0000","last_active":"2024-04-25T01:23:47.487920+0000","last_peered":"2024-04-25T01:23:47.487920+0000","last_clean":"2024-04-25T01:23:47.487920+0000","last_became_active":"2024-04-25T01:22:57.853234+0000","last_became_peered":"2024-04-25T01:22:57.853234+0000","last_unstale":"2024-04-25T01:23:47.487920+0000","last_undegraded":"2024-04-25T01:23:47.487920+0000","last_fullsized":"2024-04-25T01:23:47.487920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:51.139371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999402+0000","last_change":"2024-04-25T01:23:01.873856+0000","last_active":"2024-04-25T01:23:46.999402+0000","last_peered":"2024-04-25T01:23:46.999402+0000","last_clean":"2024-04-25T01:23:46.999402+0000","last_became_active":"2024-04-25T01:23:01.873413+0000","last_became_peered":"2024-04-25T01:23:01.873413+0000","last_unstale":"2024-04-25T01:23:46.999402+0000","last_undegraded":"2024-04-25T01:23:46.999402+0000","last_fullsized":"2024-04-25T01:23:46.999402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:11.802479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487630+0000","last_change":"2024-04-25T01:22:56.831080+0000","last_active":"2024-04-25T01:23:47.487630+0000","last_peered":"2024-04-25T01:23:47.487630+0000","last_clean":"2024-04-25T01:23:47.487630+0000","last_became_active":"2024-04-25T01:22:56.830735+0000","last_became_peered":"2024-04-25T01:22:56.830735+0000","last_unstale":"2024-04-25T01:23:47.487630+0000","last_undegraded":"2024-04-25T01:23:47.487630+0000","last_fullsized":"2024-04-25T01:23:47.487630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:38.418439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002431+0000","last_change":"2024-04-25T01:22:57.853838+0000","last_active":"2024-04-25T01:23:47.002431+0000","last_peered":"2024-04-25T01:23:47.002431+0000","last_clean":"2024-04-25T01:23:47.002431+0000","last_became_active":"2024-04-25T01:22:57.853023+0000","last_became_peered":"2024-04-25T01:22:57.853023+0000","last_unstale":"2024-04-25T01:23:47.002431+0000","last_undegraded":"2024-04-25T01:23:47.002431+0000","last_fullsized":"2024-04-25T01:23:47.002431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:18.515831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003385+0000","last_change":"2024-04-25T01:23:01.865446+0000","last_active":"2024-04-25T01:23:47.003385+0000","last_peered":"2024-04-25T01:23:47.003385+0000","last_clean":"2024-04-25T01:23:47.003385+0000","last_became_active":"2024-04-25T01:23:01.864937+0000","last_became_peered":"2024-04-25T01:23:01.864937+0000","last_unstale":"2024-04-25T01:23:47.003385+0000","last_undegraded":"2024-04-25T01:23:47.003385+0000","last_fullsized":"2024-04-25T01:23:47.003385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:18.041430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037405+0000","last_change":"2024-04-25T01:22:56.836055+0000","last_active":"2024-04-25T01:23:47.037405+0000","last_peered":"2024-04-25T01:23:47.037405+0000","last_clean":"2024-04-25T01:23:47.037405+0000","last_became_active":"2024-04-25T01:22:56.835760+0000","last_became_peered":"2024-04-25T01:22:56.835760+0000","last_unstale":"2024-04-25T01:23:47.037405+0000","last_undegraded":"2024-04-25T01:23:47.037405+0000","last_fullsized":"2024-04-25T01:23:47.037405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:59.249996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487483+0000","last_change":"2024-04-25T01:22:57.852444+0000","last_active":"2024-04-25T01:23:47.487483+0000","last_peered":"2024-04-25T01:23:47.487483+0000","last_clean":"2024-04-25T01:23:47.487483+0000","last_became_active":"2024-04-25T01:22:57.852127+0000","last_became_peered":"2024-04-25T01:22:57.852127+0000","last_unstale":"2024-04-25T01:23:47.487483+0000","last_undegraded":"2024-04-25T01:23:47.487483+0000","last_fullsized":"2024-04-25T01:23:47.487483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:11.718512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.039488+0000","last_change":"2024-04-25T01:23:47.039057+0000","last_active":"2024-04-25T01:23:47.039488+0000","last_peered":"2024-04-25T01:23:47.039488+0000","last_clean":"2024-04-25T01:23:47.039488+0000","last_became_active":"2024-04-25T01:23:47.038568+0000","last_became_peered":"2024-04-25T01:23:47.038568+0000","last_unstale":"2024-04-25T01:23:47.039488+0000","last_undegraded":"2024-04-25T01:23:47.039488+0000","last_fullsized":"2024-04-25T01:23:47.039488+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:23.828975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'33","reported_seq":218,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:38.049776+0000","last_change":"2024-04-25T01:22:56.834125+0000","last_active":"2024-04-25T01:24:38.049776+0000","last_peered":"2024-04-25T01:24:38.049776+0000","last_clean":"2024-04-25T01:24:38.049776+0000","last_became_active":"2024-04-25T01:22:56.833817+0000","last_became_peered":"2024-04-25T01:22:56.833817+0000","last_unstale":"2024-04-25T01:24:38.049776+0000","last_undegraded":"2024-04-25T01:24:38.049776+0000","last_fullsized":"2024-04-25T01:24:38.049776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:58.451250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":158,"num_read_kb":141,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004854+0000","last_change":"2024-04-25T01:22:57.854554+0000","last_active":"2024-04-25T01:23:47.004854+0000","last_peered":"2024-04-25T01:23:47.004854+0000","last_clean":"2024-04-25T01:23:47.004854+0000","last_became_active":"2024-04-25T01:22:57.853328+0000","last_became_peered":"2024-04-25T01:22:57.853328+0000","last_unstale":"2024-04-25T01:23:47.004854+0000","last_undegraded":"2024-04-25T01:23:47.004854+0000","last_fullsized":"2024-04-25T01:23:47.004854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:21.774148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037782+0000","last_change":"2024-04-25T01:23:01.870571+0000","last_active":"2024-04-25T01:23:47.037782+0000","last_peered":"2024-04-25T01:23:47.037782+0000","last_clean":"2024-04-25T01:23:47.037782+0000","last_became_active":"2024-04-25T01:23:01.869982+0000","last_became_peered":"2024-04-25T01:23:01.869982+0000","last_unstale":"2024-04-25T01:23:47.037782+0000","last_undegraded":"2024-04-25T01:23:47.037782+0000","last_fullsized":"2024-04-25T01:23:47.037782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:17.294254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000939+0000","last_change":"2024-04-25T01:22:56.835134+0000","last_active":"2024-04-25T01:23:47.000939+0000","last_peered":"2024-04-25T01:23:47.000939+0000","last_clean":"2024-04-25T01:23:47.000939+0000","last_became_active":"2024-04-25T01:22:56.834207+0000","last_became_peered":"2024-04-25T01:22:56.834207+0000","last_unstale":"2024-04-25T01:23:47.000939+0000","last_undegraded":"2024-04-25T01:23:47.000939+0000","last_fullsized":"2024-04-25T01:23:47.000939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:07.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487006+0000","last_change":"2024-04-25T01:22:57.847693+0000","last_active":"2024-04-25T01:23:47.487006+0000","last_peered":"2024-04-25T01:23:47.487006+0000","last_clean":"2024-04-25T01:23:47.487006+0000","last_became_active":"2024-04-25T01:22:57.847211+0000","last_became_peered":"2024-04-25T01:22:57.847211+0000","last_unstale":"2024-04-25T01:23:47.487006+0000","last_undegraded":"2024-04-25T01:23:47.487006+0000","last_fullsized":"2024-04-25T01:23:47.487006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:48.295608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004806+0000","last_change":"2024-04-25T01:23:01.870340+0000","last_active":"2024-04-25T01:23:47.004806+0000","last_peered":"2024-04-25T01:23:47.004806+0000","last_clean":"2024-04-25T01:23:47.004806+0000","last_became_active":"2024-04-25T01:23:01.869662+0000","last_became_peered":"2024-04-25T01:23:01.869662+0000","last_unstale":"2024-04-25T01:23:47.004806+0000","last_undegraded":"2024-04-25T01:23:47.004806+0000","last_fullsized":"2024-04-25T01:23:47.004806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:05:57.313991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037246+0000","last_change":"2024-04-25T01:22:56.838146+0000","last_active":"2024-04-25T01:23:47.037246+0000","last_peered":"2024-04-25T01:23:47.037246+0000","last_clean":"2024-04-25T01:23:47.037246+0000","last_became_active":"2024-04-25T01:22:56.837815+0000","last_became_peered":"2024-04-25T01:22:56.837815+0000","last_unstale":"2024-04-25T01:23:47.037246+0000","last_undegraded":"2024-04-25T01:23:47.037246+0000","last_fullsized":"2024-04-25T01:23:47.037246+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:14.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488024+0000","last_change":"2024-04-25T01:22:57.850840+0000","last_active":"2024-04-25T01:23:47.488024+0000","last_peered":"2024-04-25T01:23:47.488024+0000","last_clean":"2024-04-25T01:23:47.488024+0000","last_became_active":"2024-04-25T01:22:57.850255+0000","last_became_peered":"2024-04-25T01:22:57.850255+0000","last_unstale":"2024-04-25T01:23:47.488024+0000","last_undegraded":"2024-04-25T01:23:47.488024+0000","last_fullsized":"2024-04-25T01:23:47.488024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:25.887235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999607+0000","last_change":"2024-04-25T01:23:01.869396+0000","last_active":"2024-04-25T01:23:46.999607+0000","last_peered":"2024-04-25T01:23:46.999607+0000","last_clean":"2024-04-25T01:23:46.999607+0000","last_became_active":"2024-04-25T01:23:01.868777+0000","last_became_peered":"2024-04-25T01:23:01.868777+0000","last_unstale":"2024-04-25T01:23:46.999607+0000","last_undegraded":"2024-04-25T01:23:46.999607+0000","last_fullsized":"2024-04-25T01:23:46.999607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:37.276122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:36.047256+0000","last_change":"2024-04-25T01:22:56.833289+0000","last_active":"2024-04-25T01:24:36.047256+0000","last_peered":"2024-04-25T01:24:36.047256+0000","last_clean":"2024-04-25T01:24:36.047256+0000","last_became_active":"2024-04-25T01:22:56.832353+0000","last_became_peered":"2024-04-25T01:22:56.832353+0000","last_unstale":"2024-04-25T01:24:36.047256+0000","last_undegraded":"2024-04-25T01:24:36.047256+0000","last_fullsized":"2024-04-25T01:24:36.047256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:18.251018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487480+0000","last_change":"2024-04-25T01:22:57.851974+0000","last_active":"2024-04-25T01:23:47.487480+0000","last_peered":"2024-04-25T01:23:47.487480+0000","last_clean":"2024-04-25T01:23:47.487480+0000","last_became_active":"2024-04-25T01:22:57.851573+0000","last_became_peered":"2024-04-25T01:22:57.851573+0000","last_unstale":"2024-04-25T01:23:47.487480+0000","last_undegraded":"2024-04-25T01:23:47.487480+0000","last_fullsized":"2024-04-25T01:23:47.487480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:27.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487521+0000","last_change":"2024-04-25T01:23:01.876980+0000","last_active":"2024-04-25T01:23:47.487521+0000","last_peered":"2024-04-25T01:23:47.487521+0000","last_clean":"2024-04-25T01:23:47.487521+0000","last_became_active":"2024-04-25T01:23:01.876548+0000","last_became_peered":"2024-04-25T01:23:01.876548+0000","last_unstale":"2024-04-25T01:23:47.487521+0000","last_undegraded":"2024-04-25T01:23:47.487521+0000","last_fullsized":"2024-04-25T01:23:47.487521+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:13.470736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.001830+0000","last_change":"2024-04-25T01:22:56.839694+0000","last_active":"2024-04-25T01:23:47.001830+0000","last_peered":"2024-04-25T01:23:47.001830+0000","last_clean":"2024-04-25T01:23:47.001830+0000","last_became_active":"2024-04-25T01:22:56.839367+0000","last_became_peered":"2024-04-25T01:22:56.839367+0000","last_unstale":"2024-04-25T01:23:47.001830+0000","last_undegraded":"2024-04-25T01:23:47.001830+0000","last_fullsized":"2024-04-25T01:23:47.001830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:49.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036903+0000","last_change":"2024-04-25T01:22:57.841526+0000","last_active":"2024-04-25T01:23:47.036903+0000","last_peered":"2024-04-25T01:23:47.036903+0000","last_clean":"2024-04-25T01:23:47.036903+0000","last_became_active":"2024-04-25T01:22:57.841192+0000","last_became_peered":"2024-04-25T01:22:57.841192+0000","last_unstale":"2024-04-25T01:23:47.036903+0000","last_undegraded":"2024-04-25T01:23:47.036903+0000","last_fullsized":"2024-04-25T01:23:47.036903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:38.958901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036829+0000","last_change":"2024-04-25T01:23:01.865486+0000","last_active":"2024-04-25T01:23:47.036829+0000","last_peered":"2024-04-25T01:23:47.036829+0000","last_clean":"2024-04-25T01:23:47.036829+0000","last_became_active":"2024-04-25T01:23:01.865068+0000","last_became_peered":"2024-04-25T01:23:01.865068+0000","last_unstale":"2024-04-25T01:23:47.036829+0000","last_undegraded":"2024-04-25T01:23:47.036829+0000","last_fullsized":"2024-04-25T01:23:47.036829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:50.432724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005728+0000","last_change":"2024-04-25T01:22:56.841507+0000","last_active":"2024-04-25T01:23:47.005728+0000","last_peered":"2024-04-25T01:23:47.005728+0000","last_clean":"2024-04-25T01:23:47.005728+0000","last_became_active":"2024-04-25T01:22:56.839970+0000","last_became_peered":"2024-04-25T01:22:56.839970+0000","last_unstale":"2024-04-25T01:23:47.005728+0000","last_undegraded":"2024-04-25T01:23:47.005728+0000","last_fullsized":"2024-04-25T01:23:47.005728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:37.670052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003598+0000","last_change":"2024-04-25T01:22:57.850853+0000","last_active":"2024-04-25T01:23:47.003598+0000","last_peered":"2024-04-25T01:23:47.003598+0000","last_clean":"2024-04-25T01:23:47.003598+0000","last_became_active":"2024-04-25T01:22:57.850498+0000","last_became_peered":"2024-04-25T01:22:57.850498+0000","last_unstale":"2024-04-25T01:23:47.003598+0000","last_undegraded":"2024-04-25T01:23:47.003598+0000","last_fullsized":"2024-04-25T01:23:47.003598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:13.271048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002278+0000","last_change":"2024-04-25T01:23:01.874254+0000","last_active":"2024-04-25T01:23:47.002278+0000","last_peered":"2024-04-25T01:23:47.002278+0000","last_clean":"2024-04-25T01:23:47.002278+0000","last_became_active":"2024-04-25T01:23:01.873711+0000","last_became_peered":"2024-04-25T01:23:01.873711+0000","last_unstale":"2024-04-25T01:23:47.002278+0000","last_undegraded":"2024-04-25T01:23:47.002278+0000","last_fullsized":"2024-04-25T01:23:47.002278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:13.510061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999705+0000","last_change":"2024-04-25T01:22:56.827137+0000","last_active":"2024-04-25T01:23:46.999705+0000","last_peered":"2024-04-25T01:23:46.999705+0000","last_clean":"2024-04-25T01:23:46.999705+0000","last_became_active":"2024-04-25T01:22:56.826769+0000","last_became_peered":"2024-04-25T01:22:56.826769+0000","last_unstale":"2024-04-25T01:23:46.999705+0000","last_undegraded":"2024-04-25T01:23:46.999705+0000","last_fullsized":"2024-04-25T01:23:46.999705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:21.072643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003833+0000","last_change":"2024-04-25T01:22:57.843713+0000","last_active":"2024-04-25T01:23:47.003833+0000","last_peered":"2024-04-25T01:23:47.003833+0000","last_clean":"2024-04-25T01:23:47.003833+0000","last_became_active":"2024-04-25T01:22:57.843384+0000","last_became_peered":"2024-04-25T01:22:57.843384+0000","last_unstale":"2024-04-25T01:23:47.003833+0000","last_undegraded":"2024-04-25T01:23:47.003833+0000","last_fullsized":"2024-04-25T01:23:47.003833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:18.772379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487286+0000","last_change":"2024-04-25T01:23:01.874150+0000","last_active":"2024-04-25T01:23:47.487286+0000","last_peered":"2024-04-25T01:23:47.487286+0000","last_clean":"2024-04-25T01:23:47.487286+0000","last_became_active":"2024-04-25T01:23:01.873718+0000","last_became_peered":"2024-04-25T01:23:01.873718+0000","last_unstale":"2024-04-25T01:23:47.487286+0000","last_undegraded":"2024-04-25T01:23:47.487286+0000","last_fullsized":"2024-04-25T01:23:47.487286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:26.317703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487645+0000","last_change":"2024-04-25T01:22:56.843605+0000","last_active":"2024-04-25T01:23:47.487645+0000","last_peered":"2024-04-25T01:23:47.487645+0000","last_clean":"2024-04-25T01:23:47.487645+0000","last_became_active":"2024-04-25T01:22:56.843313+0000","last_became_peered":"2024-04-25T01:22:56.843313+0000","last_unstale":"2024-04-25T01:23:47.487645+0000","last_undegraded":"2024-04-25T01:23:47.487645+0000","last_fullsized":"2024-04-25T01:23:47.487645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:41.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005378+0000","last_change":"2024-04-25T01:23:47.005378+0000","last_active":"2024-04-25T01:23:47.005378+0000","last_peered":"2024-04-25T01:23:47.005378+0000","last_clean":"2024-04-25T01:23:47.005378+0000","last_became_active":"2024-04-25T01:23:47.004962+0000","last_became_peered":"2024-04-25T01:23:47.004962+0000","last_unstale":"2024-04-25T01:23:47.005378+0000","last_undegraded":"2024-04-25T01:23:47.005378+0000","last_fullsized":"2024-04-25T01:23:47.005378+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:28.548285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487520+0000","last_change":"2024-04-25T01:23:01.872792+0000","last_active":"2024-04-25T01:23:47.487520+0000","last_peered":"2024-04-25T01:23:47.487520+0000","last_clean":"2024-04-25T01:23:47.487520+0000","last_became_active":"2024-04-25T01:23:01.872391+0000","last_became_peered":"2024-04-25T01:23:01.872391+0000","last_unstale":"2024-04-25T01:23:47.487520+0000","last_undegraded":"2024-04-25T01:23:47.487520+0000","last_fullsized":"2024-04-25T01:23:47.487520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:13.088996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037123+0000","last_change":"2024-04-25T01:22:57.843131+0000","last_active":"2024-04-25T01:23:47.037123+0000","last_peered":"2024-04-25T01:23:47.037123+0000","last_clean":"2024-04-25T01:23:47.037123+0000","last_became_active":"2024-04-25T01:22:57.842540+0000","last_became_peered":"2024-04-25T01:22:57.842540+0000","last_unstale":"2024-04-25T01:23:47.037123+0000","last_undegraded":"2024-04-25T01:23:47.037123+0000","last_fullsized":"2024-04-25T01:23:47.037123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:51.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:24:37.048225+0000","last_change":"2024-04-25T01:22:56.844610+0000","last_active":"2024-04-25T01:24:37.048225+0000","last_peered":"2024-04-25T01:24:37.048225+0000","last_clean":"2024-04-25T01:24:37.048225+0000","last_became_active":"2024-04-25T01:22:56.844237+0000","last_became_peered":"2024-04-25T01:22:56.844237+0000","last_unstale":"2024-04-25T01:24:37.048225+0000","last_undegraded":"2024-04-25T01:24:37.048225+0000","last_fullsized":"2024-04-25T01:24:37.048225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:09.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488111+0000","last_change":"2024-04-25T01:23:01.865711+0000","last_active":"2024-04-25T01:23:47.488111+0000","last_peered":"2024-04-25T01:23:47.488111+0000","last_clean":"2024-04-25T01:23:47.488111+0000","last_became_active":"2024-04-25T01:23:01.865317+0000","last_became_peered":"2024-04-25T01:23:01.865317+0000","last_unstale":"2024-04-25T01:23:47.488111+0000","last_undegraded":"2024-04-25T01:23:47.488111+0000","last_fullsized":"2024-04-25T01:23:47.488111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:08.995490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147119+0000","last_change":"2024-04-25T01:22:57.851623+0000","last_active":"2024-04-25T01:23:47.147119+0000","last_peered":"2024-04-25T01:23:47.147119+0000","last_clean":"2024-04-25T01:23:47.147119+0000","last_became_active":"2024-04-25T01:22:57.850616+0000","last_became_peered":"2024-04-25T01:22:57.850616+0000","last_unstale":"2024-04-25T01:23:47.147119+0000","last_undegraded":"2024-04-25T01:23:47.147119+0000","last_fullsized":"2024-04-25T01:23:47.147119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:27.447996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004047+0000","last_change":"2024-04-25T01:22:56.832094+0000","last_active":"2024-04-25T01:23:47.004047+0000","last_peered":"2024-04-25T01:23:47.004047+0000","last_clean":"2024-04-25T01:23:47.004047+0000","last_became_active":"2024-04-25T01:22:56.831743+0000","last_became_peered":"2024-04-25T01:22:56.831743+0000","last_unstale":"2024-04-25T01:23:47.004047+0000","last_undegraded":"2024-04-25T01:23:47.004047+0000","last_fullsized":"2024-04-25T01:23:47.004047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:06.116081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487298+0000","last_change":"2024-04-25T01:21:40.899517+0000","last_active":"2024-04-25T01:23:47.487298+0000","last_peered":"2024-04-25T01:23:47.487298+0000","last_clean":"2024-04-25T01:23:47.487298+0000","last_became_active":"2024-04-25T01:21:40.899043+0000","last_became_peered":"2024-04-25T01:21:40.899043+0000","last_unstale":"2024-04-25T01:23:47.487298+0000","last_undegraded":"2024-04-25T01:23:47.487298+0000","last_fullsized":"2024-04-25T01:23:47.487298+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_clean_scrub_stamp":"2024-04-25T01:18:12.053452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:41.234852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037363+0000","last_change":"2024-04-25T01:23:01.878334+0000","last_active":"2024-04-25T01:23:47.037363+0000","last_peered":"2024-04-25T01:23:47.037363+0000","last_clean":"2024-04-25T01:23:47.037363+0000","last_became_active":"2024-04-25T01:23:01.877955+0000","last_became_peered":"2024-04-25T01:23:01.877955+0000","last_unstale":"2024-04-25T01:23:47.037363+0000","last_undegraded":"2024-04-25T01:23:47.037363+0000","last_fullsized":"2024-04-25T01:23:47.037363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:33.339430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.486850+0000","last_change":"2024-04-25T01:22:57.843020+0000","last_active":"2024-04-25T01:23:47.486850+0000","last_peered":"2024-04-25T01:23:47.486850+0000","last_clean":"2024-04-25T01:23:47.486850+0000","last_became_active":"2024-04-25T01:22:57.842671+0000","last_became_peered":"2024-04-25T01:22:57.842671+0000","last_unstale":"2024-04-25T01:23:47.486850+0000","last_undegraded":"2024-04-25T01:23:47.486850+0000","last_fullsized":"2024-04-25T01:23:47.486850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:53.831432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487219+0000","last_change":"2024-04-25T01:22:56.836106+0000","last_active":"2024-04-25T01:23:47.487219+0000","last_peered":"2024-04-25T01:23:47.487219+0000","last_clean":"2024-04-25T01:23:47.487219+0000","last_became_active":"2024-04-25T01:22:56.835814+0000","last_became_peered":"2024-04-25T01:22:56.835814+0000","last_unstale":"2024-04-25T01:23:47.487219+0000","last_undegraded":"2024-04-25T01:23:47.487219+0000","last_fullsized":"2024-04-25T01:23:47.487219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:18.004312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004645+0000","last_change":"2024-04-25T01:23:01.870723+0000","last_active":"2024-04-25T01:23:47.004645+0000","last_peered":"2024-04-25T01:23:47.004645+0000","last_clean":"2024-04-25T01:23:47.004645+0000","last_became_active":"2024-04-25T01:23:01.870205+0000","last_became_peered":"2024-04-25T01:23:01.870205+0000","last_unstale":"2024-04-25T01:23:47.004645+0000","last_undegraded":"2024-04-25T01:23:47.004645+0000","last_fullsized":"2024-04-25T01:23:47.004645+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:47.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004743+0000","last_change":"2024-04-25T01:22:56.839180+0000","last_active":"2024-04-25T01:23:47.004743+0000","last_peered":"2024-04-25T01:23:47.004743+0000","last_clean":"2024-04-25T01:23:47.004743+0000","last_became_active":"2024-04-25T01:22:56.838897+0000","last_became_peered":"2024-04-25T01:22:56.838897+0000","last_unstale":"2024-04-25T01:23:47.004743+0000","last_undegraded":"2024-04-25T01:23:47.004743+0000","last_fullsized":"2024-04-25T01:23:47.004743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:27.176333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487420+0000","last_change":"2024-04-25T01:22:57.841727+0000","last_active":"2024-04-25T01:23:47.487420+0000","last_peered":"2024-04-25T01:23:47.487420+0000","last_clean":"2024-04-25T01:23:47.487420+0000","last_became_active":"2024-04-25T01:22:57.841366+0000","last_became_peered":"2024-04-25T01:22:57.841366+0000","last_unstale":"2024-04-25T01:23:47.487420+0000","last_undegraded":"2024-04-25T01:23:47.487420+0000","last_fullsized":"2024-04-25T01:23:47.487420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:20.585916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.146941+0000","last_change":"2024-04-25T01:23:01.869814+0000","last_active":"2024-04-25T01:23:47.146941+0000","last_peered":"2024-04-25T01:23:47.146941+0000","last_clean":"2024-04-25T01:23:47.146941+0000","last_became_active":"2024-04-25T01:23:01.869387+0000","last_became_peered":"2024-04-25T01:23:01.869387+0000","last_unstale":"2024-04-25T01:23:47.146941+0000","last_undegraded":"2024-04-25T01:23:47.146941+0000","last_fullsized":"2024-04-25T01:23:47.146941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:18:45.098295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:46.999944+0000","last_change":"2024-04-25T01:22:56.838506+0000","last_active":"2024-04-25T01:23:46.999944+0000","last_peered":"2024-04-25T01:23:46.999944+0000","last_clean":"2024-04-25T01:23:46.999944+0000","last_became_active":"2024-04-25T01:22:56.838185+0000","last_became_peered":"2024-04-25T01:22:56.838185+0000","last_unstale":"2024-04-25T01:23:46.999944+0000","last_undegraded":"2024-04-25T01:23:46.999944+0000","last_fullsized":"2024-04-25T01:23:46.999944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:35.306231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147665+0000","last_change":"2024-04-25T01:22:57.846544+0000","last_active":"2024-04-25T01:23:47.147665+0000","last_peered":"2024-04-25T01:23:47.147665+0000","last_clean":"2024-04-25T01:23:47.147665+0000","last_became_active":"2024-04-25T01:22:57.843507+0000","last_became_peered":"2024-04-25T01:22:57.843507+0000","last_unstale":"2024-04-25T01:23:47.147665+0000","last_undegraded":"2024-04-25T01:23:47.147665+0000","last_fullsized":"2024-04-25T01:23:47.147665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:30.600317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000574+0000","last_change":"2024-04-25T01:23:01.863986+0000","last_active":"2024-04-25T01:23:47.000574+0000","last_peered":"2024-04-25T01:23:47.000574+0000","last_clean":"2024-04-25T01:23:47.000574+0000","last_became_active":"2024-04-25T01:23:01.863397+0000","last_became_peered":"2024-04-25T01:23:01.863397+0000","last_unstale":"2024-04-25T01:23:47.000574+0000","last_undegraded":"2024-04-25T01:23:47.000574+0000","last_fullsized":"2024-04-25T01:23:47.000574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:45:56.229349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002538+0000","last_change":"2024-04-25T01:22:56.838201+0000","last_active":"2024-04-25T01:23:47.002538+0000","last_peered":"2024-04-25T01:23:47.002538+0000","last_clean":"2024-04-25T01:23:47.002538+0000","last_became_active":"2024-04-25T01:22:56.837890+0000","last_became_peered":"2024-04-25T01:22:56.837890+0000","last_unstale":"2024-04-25T01:23:47.002538+0000","last_undegraded":"2024-04-25T01:23:47.002538+0000","last_fullsized":"2024-04-25T01:23:47.002538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:17.418112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002385+0000","last_change":"2024-04-25T01:22:57.853923+0000","last_active":"2024-04-25T01:23:47.002385+0000","last_peered":"2024-04-25T01:23:47.002385+0000","last_clean":"2024-04-25T01:23:47.002385+0000","last_became_active":"2024-04-25T01:22:57.853566+0000","last_became_peered":"2024-04-25T01:22:57.853566+0000","last_unstale":"2024-04-25T01:23:47.002385+0000","last_undegraded":"2024-04-25T01:23:47.002385+0000","last_fullsized":"2024-04-25T01:23:47.002385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:36.509650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.038657+0000","last_change":"2024-04-25T01:23:47.038657+0000","last_active":"2024-04-25T01:23:47.038657+0000","last_peered":"2024-04-25T01:23:47.038657+0000","last_clean":"2024-04-25T01:23:47.038657+0000","last_became_active":"2024-04-25T01:23:47.038168+0000","last_became_peered":"2024-04-25T01:23:47.038168+0000","last_unstale":"2024-04-25T01:23:47.038657+0000","last_undegraded":"2024-04-25T01:23:47.038657+0000","last_fullsized":"2024-04-25T01:23:47.038657+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:59.844333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.486808+0000","last_change":"2024-04-25T01:22:56.828651+0000","last_active":"2024-04-25T01:23:47.486808+0000","last_peered":"2024-04-25T01:23:47.486808+0000","last_clean":"2024-04-25T01:23:47.486808+0000","last_became_active":"2024-04-25T01:22:56.828186+0000","last_became_peered":"2024-04-25T01:22:56.828186+0000","last_unstale":"2024-04-25T01:23:47.486808+0000","last_undegraded":"2024-04-25T01:23:47.486808+0000","last_fullsized":"2024-04-25T01:23:47.486808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:56.020822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.001971+0000","last_change":"2024-04-25T01:22:57.849768+0000","last_active":"2024-04-25T01:23:47.001971+0000","last_peered":"2024-04-25T01:23:47.001971+0000","last_clean":"2024-04-25T01:23:47.001971+0000","last_became_active":"2024-04-25T01:22:57.849418+0000","last_became_peered":"2024-04-25T01:22:57.849418+0000","last_unstale":"2024-04-25T01:23:47.001971+0000","last_undegraded":"2024-04-25T01:23:47.001971+0000","last_fullsized":"2024-04-25T01:23:47.001971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:41.292915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.149351+0000","last_change":"2024-04-25T01:23:47.148790+0000","last_active":"2024-04-25T01:23:47.149351+0000","last_peered":"2024-04-25T01:23:47.149351+0000","last_clean":"2024-04-25T01:23:47.149351+0000","last_became_active":"2024-04-25T01:23:47.148180+0000","last_became_peered":"2024-04-25T01:23:47.148180+0000","last_unstale":"2024-04-25T01:23:47.149351+0000","last_undegraded":"2024-04-25T01:23:47.149351+0000","last_fullsized":"2024-04-25T01:23:47.149351+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:08.296696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147048+0000","last_change":"2024-04-25T01:22:56.844672+0000","last_active":"2024-04-25T01:23:47.147048+0000","last_peered":"2024-04-25T01:23:47.147048+0000","last_clean":"2024-04-25T01:23:47.147048+0000","last_became_active":"2024-04-25T01:22:56.844141+0000","last_became_peered":"2024-04-25T01:22:56.844141+0000","last_unstale":"2024-04-25T01:23:47.147048+0000","last_undegraded":"2024-04-25T01:23:47.147048+0000","last_fullsized":"2024-04-25T01:23:47.147048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:14.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004864+0000","last_change":"2024-04-25T01:22:57.845006+0000","last_active":"2024-04-25T01:23:47.004864+0000","last_peered":"2024-04-25T01:23:47.004864+0000","last_clean":"2024-04-25T01:23:47.004864+0000","last_became_active":"2024-04-25T01:22:57.844672+0000","last_became_peered":"2024-04-25T01:22:57.844672+0000","last_unstale":"2024-04-25T01:23:47.004864+0000","last_undegraded":"2024-04-25T01:23:47.004864+0000","last_fullsized":"2024-04-25T01:23:47.004864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:18.238512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.003702+0000","last_change":"2024-04-25T01:23:01.869438+0000","last_active":"2024-04-25T01:23:47.003702+0000","last_peered":"2024-04-25T01:23:47.003702+0000","last_clean":"2024-04-25T01:23:47.003702+0000","last_became_active":"2024-04-25T01:23:01.868985+0000","last_became_peered":"2024-04-25T01:23:01.868985+0000","last_unstale":"2024-04-25T01:23:47.003702+0000","last_undegraded":"2024-04-25T01:23:47.003702+0000","last_fullsized":"2024-04-25T01:23:47.003702+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:35.509098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000742+0000","last_change":"2024-04-25T01:22:56.834475+0000","last_active":"2024-04-25T01:23:47.000742+0000","last_peered":"2024-04-25T01:23:47.000742+0000","last_clean":"2024-04-25T01:23:47.000742+0000","last_became_active":"2024-04-25T01:22:56.834076+0000","last_became_peered":"2024-04-25T01:22:56.834076+0000","last_unstale":"2024-04-25T01:23:47.000742+0000","last_undegraded":"2024-04-25T01:23:47.000742+0000","last_fullsized":"2024-04-25T01:23:47.000742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:55:47.464009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147329+0000","last_change":"2024-04-25T01:22:57.860549+0000","last_active":"2024-04-25T01:23:47.147329+0000","last_peered":"2024-04-25T01:23:47.147329+0000","last_clean":"2024-04-25T01:23:47.147329+0000","last_became_active":"2024-04-25T01:22:57.860226+0000","last_became_peered":"2024-04-25T01:22:57.860226+0000","last_unstale":"2024-04-25T01:23:47.147329+0000","last_undegraded":"2024-04-25T01:23:47.147329+0000","last_fullsized":"2024-04-25T01:23:47.147329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:10.663508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002194+0000","last_change":"2024-04-25T01:23:01.867602+0000","last_active":"2024-04-25T01:23:47.002194+0000","last_peered":"2024-04-25T01:23:47.002194+0000","last_clean":"2024-04-25T01:23:47.002194+0000","last_became_active":"2024-04-25T01:23:01.867150+0000","last_became_peered":"2024-04-25T01:23:01.867150+0000","last_unstale":"2024-04-25T01:23:47.002194+0000","last_undegraded":"2024-04-25T01:23:47.002194+0000","last_fullsized":"2024-04-25T01:23:47.002194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:34.571031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002827+0000","last_change":"2024-04-25T01:22:56.841082+0000","last_active":"2024-04-25T01:23:47.002827+0000","last_peered":"2024-04-25T01:23:47.002827+0000","last_clean":"2024-04-25T01:23:47.002827+0000","last_became_active":"2024-04-25T01:22:56.839856+0000","last_became_peered":"2024-04-25T01:22:56.839856+0000","last_unstale":"2024-04-25T01:23:47.002827+0000","last_undegraded":"2024-04-25T01:23:47.002827+0000","last_fullsized":"2024-04-25T01:23:47.002827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:14.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487775+0000","last_change":"2024-04-25T01:22:57.852242+0000","last_active":"2024-04-25T01:23:47.487775+0000","last_peered":"2024-04-25T01:23:47.487775+0000","last_clean":"2024-04-25T01:23:47.487775+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:23:47.487775+0000","last_undegraded":"2024-04-25T01:23:47.487775+0000","last_fullsized":"2024-04-25T01:23:47.487775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:38:02.810025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.486894+0000","last_change":"2024-04-25T01:23:01.871074+0000","last_active":"2024-04-25T01:23:47.486894+0000","last_peered":"2024-04-25T01:23:47.486894+0000","last_clean":"2024-04-25T01:23:47.486894+0000","last_became_active":"2024-04-25T01:23:01.870570+0000","last_became_peered":"2024-04-25T01:23:01.870570+0000","last_unstale":"2024-04-25T01:23:47.486894+0000","last_undegraded":"2024-04-25T01:23:47.486894+0000","last_fullsized":"2024-04-25T01:23:47.486894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:14.476400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.011645+0000","last_change":"2024-04-25T01:23:47.011202+0000","last_active":"2024-04-25T01:23:47.011645+0000","last_peered":"2024-04-25T01:23:47.011645+0000","last_clean":"2024-04-25T01:23:47.011645+0000","last_became_active":"2024-04-25T01:23:47.006425+0000","last_became_peered":"2024-04-25T01:23:47.006425+0000","last_unstale":"2024-04-25T01:23:47.011645+0000","last_undegraded":"2024-04-25T01:23:47.011645+0000","last_fullsized":"2024-04-25T01:23:47.011645+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:51:46.791184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488241+0000","last_change":"2024-04-25T01:22:57.843173+0000","last_active":"2024-04-25T01:23:47.488241+0000","last_peered":"2024-04-25T01:23:47.488241+0000","last_clean":"2024-04-25T01:23:47.488241+0000","last_became_active":"2024-04-25T01:22:57.842734+0000","last_became_peered":"2024-04-25T01:22:57.842734+0000","last_unstale":"2024-04-25T01:23:47.488241+0000","last_undegraded":"2024-04-25T01:23:47.488241+0000","last_fullsized":"2024-04-25T01:23:47.488241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:33.197493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.002039+0000","last_change":"2024-04-25T01:23:01.864068+0000","last_active":"2024-04-25T01:23:47.002039+0000","last_peered":"2024-04-25T01:23:47.002039+0000","last_clean":"2024-04-25T01:23:47.002039+0000","last_became_active":"2024-04-25T01:23:01.863640+0000","last_became_peered":"2024-04-25T01:23:01.863640+0000","last_unstale":"2024-04-25T01:23:47.002039+0000","last_undegraded":"2024-04-25T01:23:47.002039+0000","last_fullsized":"2024-04-25T01:23:47.002039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:26.925012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.037647+0000","last_change":"2024-04-25T01:22:56.837969+0000","last_active":"2024-04-25T01:23:47.037647+0000","last_peered":"2024-04-25T01:23:47.037647+0000","last_clean":"2024-04-25T01:23:47.037647+0000","last_became_active":"2024-04-25T01:22:56.837641+0000","last_became_peered":"2024-04-25T01:22:56.837641+0000","last_unstale":"2024-04-25T01:23:47.037647+0000","last_undegraded":"2024-04-25T01:23:47.037647+0000","last_fullsized":"2024-04-25T01:23:47.037647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:04.376038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487604+0000","last_change":"2024-04-25T01:22:57.851057+0000","last_active":"2024-04-25T01:23:47.487604+0000","last_peered":"2024-04-25T01:23:47.487604+0000","last_clean":"2024-04-25T01:23:47.487604+0000","last_became_active":"2024-04-25T01:22:57.850278+0000","last_became_peered":"2024-04-25T01:22:57.850278+0000","last_unstale":"2024-04-25T01:23:47.487604+0000","last_undegraded":"2024-04-25T01:23:47.487604+0000","last_fullsized":"2024-04-25T01:23:47.487604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:15.332193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036445+0000","last_change":"2024-04-25T01:23:01.866673+0000","last_active":"2024-04-25T01:23:47.036445+0000","last_peered":"2024-04-25T01:23:47.036445+0000","last_clean":"2024-04-25T01:23:47.036445+0000","last_became_active":"2024-04-25T01:23:01.866210+0000","last_became_peered":"2024-04-25T01:23:01.866210+0000","last_unstale":"2024-04-25T01:23:47.036445+0000","last_undegraded":"2024-04-25T01:23:47.036445+0000","last_fullsized":"2024-04-25T01:23:47.036445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:20.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487558+0000","last_change":"2024-04-25T01:22:56.844590+0000","last_active":"2024-04-25T01:23:47.487558+0000","last_peered":"2024-04-25T01:23:47.487558+0000","last_clean":"2024-04-25T01:23:47.487558+0000","last_became_active":"2024-04-25T01:22:56.844279+0000","last_became_peered":"2024-04-25T01:22:56.844279+0000","last_unstale":"2024-04-25T01:23:47.487558+0000","last_undegraded":"2024-04-25T01:23:47.487558+0000","last_fullsized":"2024-04-25T01:23:47.487558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:47.155832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487881+0000","last_change":"2024-04-25T01:22:57.858850+0000","last_active":"2024-04-25T01:23:47.487881+0000","last_peered":"2024-04-25T01:23:47.487881+0000","last_clean":"2024-04-25T01:23:47.487881+0000","last_became_active":"2024-04-25T01:22:57.858498+0000","last_became_peered":"2024-04-25T01:22:57.858498+0000","last_unstale":"2024-04-25T01:23:47.487881+0000","last_undegraded":"2024-04-25T01:23:47.487881+0000","last_fullsized":"2024-04-25T01:23:47.487881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:04.836062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.007111+0000","last_change":"2024-04-25T01:23:47.007111+0000","last_active":"2024-04-25T01:23:47.007111+0000","last_peered":"2024-04-25T01:23:47.007111+0000","last_clean":"2024-04-25T01:23:47.007111+0000","last_became_active":"2024-04-25T01:23:47.006699+0000","last_became_peered":"2024-04-25T01:23:47.006699+0000","last_unstale":"2024-04-25T01:23:47.007111+0000","last_undegraded":"2024-04-25T01:23:47.007111+0000","last_fullsized":"2024-04-25T01:23:47.007111+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:57:43.608183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488742+0000","last_change":"2024-04-25T01:23:47.488742+0000","last_active":"2024-04-25T01:23:47.488742+0000","last_peered":"2024-04-25T01:23:47.488742+0000","last_clean":"2024-04-25T01:23:47.488742+0000","last_became_active":"2024-04-25T01:23:47.488193+0000","last_became_peered":"2024-04-25T01:23:47.488193+0000","last_unstale":"2024-04-25T01:23:47.488742+0000","last_undegraded":"2024-04-25T01:23:47.488742+0000","last_fullsized":"2024-04-25T01:23:47.488742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:02.658395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.000673+0000","last_change":"2024-04-25T01:22:57.851138+0000","last_active":"2024-04-25T01:23:47.000673+0000","last_peered":"2024-04-25T01:23:47.000673+0000","last_clean":"2024-04-25T01:23:47.000673+0000","last_became_active":"2024-04-25T01:22:57.849107+0000","last_became_peered":"2024-04-25T01:22:57.849107+0000","last_unstale":"2024-04-25T01:23:47.000673+0000","last_undegraded":"2024-04-25T01:23:47.000673+0000","last_fullsized":"2024-04-25T01:23:47.000673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:47.665356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.488162+0000","last_change":"2024-04-25T01:23:47.488162+0000","last_active":"2024-04-25T01:23:47.488162+0000","last_peered":"2024-04-25T01:23:47.488162+0000","last_clean":"2024-04-25T01:23:47.488162+0000","last_became_active":"2024-04-25T01:23:47.487731+0000","last_became_peered":"2024-04-25T01:23:47.487731+0000","last_unstale":"2024-04-25T01:23:47.488162+0000","last_undegraded":"2024-04-25T01:23:47.488162+0000","last_fullsized":"2024-04-25T01:23:47.488162+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:14.716313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487400+0000","last_change":"2024-04-25T01:22:57.851715+0000","last_active":"2024-04-25T01:23:47.487400+0000","last_peered":"2024-04-25T01:23:47.487400+0000","last_clean":"2024-04-25T01:23:47.487400+0000","last_became_active":"2024-04-25T01:22:57.847287+0000","last_became_peered":"2024-04-25T01:22:57.847287+0000","last_unstale":"2024-04-25T01:23:47.487400+0000","last_undegraded":"2024-04-25T01:23:47.487400+0000","last_fullsized":"2024-04-25T01:23:47.487400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.749931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.004048+0000","last_change":"2024-04-25T01:22:57.846075+0000","last_active":"2024-04-25T01:23:47.004048+0000","last_peered":"2024-04-25T01:23:47.004048+0000","last_clean":"2024-04-25T01:23:47.004048+0000","last_became_active":"2024-04-25T01:22:57.845388+0000","last_became_peered":"2024-04-25T01:22:57.845388+0000","last_unstale":"2024-04-25T01:23:47.004048+0000","last_undegraded":"2024-04-25T01:23:47.004048+0000","last_fullsized":"2024-04-25T01:23:47.004048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:19.859806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.001611+0000","last_change":"2024-04-25T01:22:57.851446+0000","last_active":"2024-04-25T01:23:47.001611+0000","last_peered":"2024-04-25T01:23:47.001611+0000","last_clean":"2024-04-25T01:23:47.001611+0000","last_became_active":"2024-04-25T01:22:57.848729+0000","last_became_peered":"2024-04-25T01:22:57.848729+0000","last_unstale":"2024-04-25T01:23:47.001611+0000","last_undegraded":"2024-04-25T01:23:47.001611+0000","last_fullsized":"2024-04-25T01:23:47.001611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:49.721024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147722+0000","last_change":"2024-04-25T01:22:57.847598+0000","last_active":"2024-04-25T01:23:47.147722+0000","last_peered":"2024-04-25T01:23:47.147722+0000","last_clean":"2024-04-25T01:23:47.147722+0000","last_became_active":"2024-04-25T01:22:57.847232+0000","last_became_peered":"2024-04-25T01:22:57.847232+0000","last_unstale":"2024-04-25T01:23:47.147722+0000","last_undegraded":"2024-04-25T01:23:47.147722+0000","last_fullsized":"2024-04-25T01:23:47.147722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.227700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.036924+0000","last_change":"2024-04-25T01:22:57.842881+0000","last_active":"2024-04-25T01:23:47.036924+0000","last_peered":"2024-04-25T01:23:47.036924+0000","last_clean":"2024-04-25T01:23:47.036924+0000","last_became_active":"2024-04-25T01:22:57.842108+0000","last_became_peered":"2024-04-25T01:22:57.842108+0000","last_unstale":"2024-04-25T01:23:47.036924+0000","last_undegraded":"2024-04-25T01:23:47.036924+0000","last_fullsized":"2024-04-25T01:23:47.036924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:53.208188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.147038+0000","last_change":"2024-04-25T01:22:57.860047+0000","last_active":"2024-04-25T01:23:47.147038+0000","last_peered":"2024-04-25T01:23:47.147038+0000","last_clean":"2024-04-25T01:23:47.147038+0000","last_became_active":"2024-04-25T01:22:57.859702+0000","last_became_peered":"2024-04-25T01:22:57.859702+0000","last_unstale":"2024-04-25T01:23:47.147038+0000","last_undegraded":"2024-04-25T01:23:47.147038+0000","last_fullsized":"2024-04-25T01:23:47.147038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:00.684055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487044+0000","last_change":"2024-04-25T01:22:57.847996+0000","last_active":"2024-04-25T01:23:47.487044+0000","last_peered":"2024-04-25T01:23:47.487044+0000","last_clean":"2024-04-25T01:23:47.487044+0000","last_became_active":"2024-04-25T01:22:57.847561+0000","last_became_peered":"2024-04-25T01:22:57.847561+0000","last_unstale":"2024-04-25T01:23:47.487044+0000","last_undegraded":"2024-04-25T01:23:47.487044+0000","last_fullsized":"2024-04-25T01:23:47.487044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:16.327558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.487311+0000","last_change":"2024-04-25T01:22:57.845216+0000","last_active":"2024-04-25T01:23:47.487311+0000","last_peered":"2024-04-25T01:23:47.487311+0000","last_clean":"2024-04-25T01:23:47.487311+0000","last_became_active":"2024-04-25T01:22:57.844883+0000","last_became_peered":"2024-04-25T01:22:57.844883+0000","last_unstale":"2024-04-25T01:23:47.487311+0000","last_undegraded":"2024-04-25T01:23:47.487311+0000","last_fullsized":"2024-04-25T01:23:47.487311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:33.782048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:23:47.005024+0000","last_change":"2024-04-25T01:22:57.853251+0000","last_active":"2024-04-25T01:23:47.005024+0000","last_peered":"2024-04-25T01:23:47.005024+0000","last_clean":"2024-04-25T01:23:47.005024+0000","last_became_active":"2024-04-25T01:22:57.852878+0000","last_became_peered":"2024-04-25T01:22:57.852878+0000","last_unstale":"2024-04-25T01:23:47.005024+0000","last_undegraded":"2024-04-25T01:23:47.005024+0000","last_fullsized":"2024-04-25T01:23:47.005024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:42.487837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281206,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67357,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":473,"num_read_kb":309,"num_write":161,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":202071,"data_compressed":4881,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383715,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":500815,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645352,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1138688,"data_stored":1151439,"data_compressed":58064,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":505147,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":500971,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":1096880,"data_compressed":56437,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":496227,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":559493,"data_compressed":22299,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":497615,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":560652,"data_compressed":22299,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496421,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":501459,"data_compressed":20672,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1069056,"data_stored":1092468,"data_compressed":56437,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64515,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64934,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:24:42.404 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T01:24:42.404 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T01:24:42.404 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T01:24:42.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T01:24:42.657 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:42.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:42 smithi027 ceph-mon[29197]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:42.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:42 smithi027 ceph-mon[29197]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:42.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:42 smithi153 ceph-mon[32621]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:42.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:42 smithi153 ceph-mon[32621]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:42.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:42 smithi139 ceph-mon[32760]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:24:42.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:42 smithi139 ceph-mon[32760]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 -- 172.21.15.27:0/1645795761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0098101130 msgr2=0x7f00981015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 --2- 172.21.15.27:0/1645795761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0098101130 0x7f00981015b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0080009990 tx=0x7f0080009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 -- 172.21.15.27:0/1645795761 shutdown_connections 2024-04-25T01:24:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 --2- 172.21.15.27:0/1645795761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0098101af0 0x7f009810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 --2- 172.21.15.27:0/1645795761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f00981071a0 0x7f0098107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 --2- 172.21.15.27:0/1645795761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0098101130 0x7f00981015b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 -- 172.21.15.27:0/1645795761 >> 172.21.15.27:0/1645795761 conn(0x7f00980780e0 msgr2=0x7f00980784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:44.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:44 smithi027 ceph-mon[29197]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:44.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 -- 172.21.15.27:0/1645795761 shutdown_connections 2024-04-25T01:24:44.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.509+0000 7f009e80f700 1 -- 172.21.15.27:0/1645795761 wait complete. 2024-04-25T01:24:44.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009e80f700 1 Processor -- start 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009e80f700 1 -- start start 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009e80f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0098101130 0x7f009819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009e80f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0098101af0 0x7f009819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009e80f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00981071a0 0x7f00981987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009e80f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0098111440 con 0x7f00981071a0 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009e80f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f00981112c0 con 0x7f0098101af0 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009e80f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00981115c0 con 0x7f0098101130 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.510+0000 7f009cdac700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00981071a0 0x7f00981987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f0097fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0098101130 0x7f009819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009cdac700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00981071a0 0x7f00981987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49674/0 (socket says 172.21.15.27:49674) 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009cdac700 1 -- 172.21.15.27:0/4275229645 learned_addr learned my addr 172.21.15.27:0/4275229645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f0097fff700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0098101130 0x7f009819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:44580/0 (socket says 172.21.15.27:44580) 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f00977fe700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0098101af0 0x7f009819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009cdac700 1 -- 172.21.15.27:0/4275229645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0098101130 msgr2=0x7f009819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009cdac700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0098101130 0x7f009819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009cdac700 1 -- 172.21.15.27:0/4275229645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0098101af0 msgr2=0x7f009819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009cdac700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0098101af0 0x7f009819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009cdac700 1 -- 172.21.15.27:0/4275229645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0098198f80 con 0x7f00981071a0 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009cdac700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00981071a0 0x7f00981987a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0090007ee0 tx=0x7f009000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f00957fa700 1 -- 172.21.15.27:0/4275229645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090004bb0 con 0x7f00981071a0 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f00957fa700 1 -- 172.21.15.27:0/4275229645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0090004d50 con 0x7f00981071a0 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0098199270 con 0x7f00981071a0 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f00957fa700 1 -- 172.21.15.27:0/4275229645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090005960 con 0x7f00981071a0 2024-04-25T01:24:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.511+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00981a5480 con 0x7f00981071a0 2024-04-25T01:24:44.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.512+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0098102850 con 0x7f00981071a0 2024-04-25T01:24:44.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.514+0000 7f00957fa700 1 -- 172.21.15.27:0/4275229645 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f009001a020 con 0x7f00981071a0 2024-04-25T01:24:44.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.514+0000 7f00957fa700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f00840a2d30 0x7f00840a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:44.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.514+0000 7f0097fff700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f00840a2d30 0x7f00840a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:44.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.514+0000 7f00957fa700 1 -- 172.21.15.27:0/4275229645 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f00900cebc0 con 0x7f00981071a0 2024-04-25T01:24:44.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.515+0000 7f0097fff700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f00840a2d30 0x7f00840a51f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0088009be0 tx=0x7f0088009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:44.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.519+0000 7f00957fa700 1 -- 172.21.15.27:0/4275229645 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0090010070 con 0x7f00981071a0 2024-04-25T01:24:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.769+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f00981a5700 con 0x7f00981071a0 2024-04-25T01:24:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.769+0000 7f00957fa700 1 -- 172.21.15.27:0/4275229645 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f009009b660 con 0x7f00981071a0 2024-04-25T01:24:44.750 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:44.750 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T01:24:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f00840a2d30 msgr2=0x7f00840a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f00840a2d30 0x7f00840a51f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0088009be0 tx=0x7f0088009260 comp rx=0 tx=0).stop 2024-04-25T01:24:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00981071a0 msgr2=0x7f00981987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00981071a0 0x7f00981987a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0090007ee0 tx=0x7f009000d3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 shutdown_connections 2024-04-25T01:24:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f00840a2d30 0x7f00840a51f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00981071a0 0x7f00981987a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0098101130 0x7f009819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 --2- 172.21.15.27:0/4275229645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0098101af0 0x7f009819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.772+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 >> 172.21.15.27:0/4275229645 conn(0x7f00980780e0 msgr2=0x7f00981003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.773+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 shutdown_connections 2024-04-25T01:24:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:44.773+0000 7f009e80f700 1 -- 172.21.15.27:0/4275229645 wait complete. 2024-04-25T01:24:44.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:44 smithi153 ceph-mon[32621]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:44.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:44 smithi139 ceph-mon[32760]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:24:45.565 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T01:24:45.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:45.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.689+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/2205211056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 msgr2=0x7f5c281013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.689+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/2205211056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 0x7f5c281013f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400aa10 tx=0x7f5c2402b720 comp rx=0 tx=0).stop 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/2205211056 shutdown_connections 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/2205211056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c28101930 0x7f5c28108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/2205211056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 0x7f5c281013f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/2205211056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c280ffe20 0x7f5c28100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/2205211056 >> 172.21.15.27:0/2205211056 conn(0x7f5c280fb3b0 msgr2=0x7f5c280fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/2205211056 shutdown_connections 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/2205211056 wait complete. 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 Processor -- start 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 -- start start 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c280ffe20 0x7f5c281958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 0x7f5c28195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:45.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c28101930 0x7f5c2819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2810b240 con 0x7f5c28100f70 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2810b0c0 con 0x7f5c280ffe20 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.690+0000 7f5c2ffe9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2810b3c0 con 0x7f5c28101930 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 0x7f5c28195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 0x7f5c28195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49694/0 (socket says 172.21.15.27:49694) 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 -- 172.21.15.27:0/4076900967 learned_addr learned my addr 172.21.15.27:0/4076900967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2e586640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c28101930 0x7f5c2819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2dd85640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c280ffe20 0x7f5c281958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 -- 172.21.15.27:0/4076900967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c28101930 msgr2=0x7f5c2819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c28101930 0x7f5c2819ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 -- 172.21.15.27:0/4076900967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c280ffe20 msgr2=0x7f5c281958c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c280ffe20 0x7f5c281958c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 -- 172.21.15.27:0/4076900967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2819d4e0 con 0x7f5c28100f70 2024-04-25T01:24:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2e586640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c28101930 0x7f5c2819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:45.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2d584640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 0x7f5c28195e00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5c24030720 tx=0x7f5c240043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:45.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c1effd640 1 -- 172.21.15.27:0/4076900967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24040070 con 0x7f5c28100f70 2024-04-25T01:24:45.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2819d770 con 0x7f5c28100f70 2024-04-25T01:24:45.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c1effd640 1 -- 172.21.15.27:0/4076900967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c2403c460 con 0x7f5c28100f70 2024-04-25T01:24:45.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c1effd640 1 -- 172.21.15.27:0/4076900967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24036030 con 0x7f5c28100f70 2024-04-25T01:24:45.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.691+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c2819dc60 con 0x7f5c28100f70 2024-04-25T01:24:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.692+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c28107150 con 0x7f5c28100f70 2024-04-25T01:24:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.693+0000 7f5c1effd640 1 -- 172.21.15.27:0/4076900967 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5c24043070 con 0x7f5c28100f70 2024-04-25T01:24:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.694+0000 7f5c1effd640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5c140a2c20 0x7f5c140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.694+0000 7f5c2dd85640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5c140a2c20 0x7f5c140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.694+0000 7f5c1effd640 1 -- 172.21.15.27:0/4076900967 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5c240f0f40 con 0x7f5c28100f70 2024-04-25T01:24:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.694+0000 7f5c2dd85640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5c140a2c20 0x7f5c140a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5c18005d00 tx=0x7f5c1800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:45.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.699+0000 7f5c1effd640 1 -- 172.21.15.27:0/4076900967 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5c240bd390 con 0x7f5c28100f70 2024-04-25T01:24:45.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:45 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4275229645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:24:45.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:45 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4275229645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:24:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.887+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c2802d320 con 0x7f5c28100f70 2024-04-25T01:24:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.888+0000 7f5c1effd640 1 -- 172.21.15.27:0/4076900967 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24594 (secure 0 0 0) 0x7f5c240c1690 con 0x7f5c28100f70 2024-04-25T01:24:45.869 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:45.869 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:23:46.996369+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:45.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5c140a2c20 msgr2=0x7f5c140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:45.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5c140a2c20 0x7f5c140a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5c18005d00 tx=0x7f5c1800b410 comp rx=0 tx=0).stop 2024-04-25T01:24:45.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 msgr2=0x7f5c28195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:45.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 0x7f5c28195e00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5c24030720 tx=0x7f5c240043b0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 shutdown_connections 2024-04-25T01:24:45.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5c140a2c20 0x7f5c140a50e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c28101930 0x7f5c2819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c28100f70 0x7f5c28195e00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 --2- 172.21.15.27:0/4076900967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5c280ffe20 0x7f5c281958c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 >> 172.21.15.27:0/4076900967 conn(0x7f5c280fb3b0 msgr2=0x7f5c28104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 shutdown_connections 2024-04-25T01:24:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:45.890+0000 7f5c2ffe9640 1 -- 172.21.15.27:0/4076900967 wait complete. 2024-04-25T01:24:45.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:45 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4275229645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:24:45.882 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:24:45.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:24:45.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 -- 172.21.15.27:0/1864335173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d0102af0 msgr2=0x7fd8d0102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:45.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 --2- 172.21.15.27:0/1864335173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d0102af0 0x7fd8d0102f70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009930 tx=0x7fd8c402a420 comp rx=0 tx=0).stop 2024-04-25T01:24:45.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 -- 172.21.15.27:0/1864335173 shutdown_connections 2024-04-25T01:24:45.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 --2- 172.21.15.27:0/1864335173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8d01034b0 0x7fd8d0109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 --2- 172.21.15.27:0/1864335173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d0102af0 0x7fd8d0102f70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 --2- 172.21.15.27:0/1864335173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d00ffa80 0x7fd8d00ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 -- 172.21.15.27:0/1864335173 >> 172.21.15.27:0/1864335173 conn(0x7fd8d00fb370 msgr2=0x7fd8d00fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 -- 172.21.15.27:0/1864335173 shutdown_connections 2024-04-25T01:24:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.008+0000 7fd8d6858640 1 -- 172.21.15.27:0/1864335173 wait complete. 2024-04-25T01:24:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d6858640 1 Processor -- start 2024-04-25T01:24:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d6858640 1 -- start start 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d6858640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d00ffa80 0x7fd8d0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d6858640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d0102af0 0x7fd8d0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d6858640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8d01034b0 0x7fd8d019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d6858640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d010d410 con 0x7fd8d00ffa80 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d6858640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d010d290 con 0x7fd8d0102af0 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d6858640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d010d590 con 0x7fd8d01034b0 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d00ffa80 0x7fd8d0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d00ffa80 0x7fd8d0197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49712/0 (socket says 172.21.15.27:49712) 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8cffff640 1 -- 172.21.15.27:0/3260191116 learned_addr learned my addr 172.21.15.27:0/3260191116 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8d4df5640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8d01034b0 0x7fd8d019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.009+0000 7fd8cf7fe640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d0102af0 0x7fd8d0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cffff640 1 -- 172.21.15.27:0/3260191116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8d01034b0 msgr2=0x7fd8d019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cffff640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8d01034b0 0x7fd8d019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cffff640 1 -- 172.21.15.27:0/3260191116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d0102af0 msgr2=0x7fd8d0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cffff640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d0102af0 0x7fd8d0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cffff640 1 -- 172.21.15.27:0/3260191116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d019f730 con 0x7fd8d00ffa80 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cffff640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d00ffa80 0x7fd8d0197b10 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000d950 tx=0x7fd8c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cd7fa640 1 -- 172.21.15.27:0/3260191116 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c0017020 con 0x7fd8d00ffa80 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cd7fa640 1 -- 172.21.15.27:0/3260191116 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8c0009c30 con 0x7fd8d00ffa80 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d019fa20 con 0x7fd8d00ffa80 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8cd7fa640 1 -- 172.21.15.27:0/3260191116 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c000f740 con 0x7fd8d00ffa80 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.010+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d019ff40 con 0x7fd8d00ffa80 2024-04-25T01:24:45.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.011+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d00689b0 con 0x7fd8d00ffa80 2024-04-25T01:24:45.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.012+0000 7fd8cd7fa640 1 -- 172.21.15.27:0/3260191116 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd8c001a020 con 0x7fd8d00ffa80 2024-04-25T01:24:45.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.013+0000 7fd8cd7fa640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd8b80a2c20 0x7fd8b80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:45.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.013+0000 7fd8cf7fe640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd8b80a2c20 0x7fd8b80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:45.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.013+0000 7fd8cd7fa640 1 -- 172.21.15.27:0/3260191116 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd8c00cf1f0 con 0x7fd8d00ffa80 2024-04-25T01:24:45.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.013+0000 7fd8cf7fe640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd8b80a2c20 0x7fd8b80a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4005ab0 tx=0x7fd8c4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.017+0000 7fd8cd7fa640 1 -- 172.21.15.27:0/3260191116 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd8c0021070 con 0x7fd8d00ffa80 2024-04-25T01:24:46.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.199+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd8d0108c70 con 0x7fd8d00ffa80 2024-04-25T01:24:46.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.200+0000 7fd8cd7fa640 1 -- 172.21.15.27:0/3260191116 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fd8d0108c70 con 0x7fd8d00ffa80 2024-04-25T01:24:46.180 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:24:46.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd8b80a2c20 msgr2=0x7fd8b80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd8b80a2c20 0x7fd8b80a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4005ab0 tx=0x7fd8c4000bc0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d00ffa80 msgr2=0x7fd8d0197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d00ffa80 0x7fd8d0197b10 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000d950 tx=0x7fd8c000de10 comp rx=0 tx=0).stop 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 shutdown_connections 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd8b80a2c20 0x7fd8b80a50e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8d01034b0 0x7fd8d019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d0102af0 0x7fd8d0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 --2- 172.21.15.27:0/3260191116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d00ffa80 0x7fd8d0197b10 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 >> 172.21.15.27:0/3260191116 conn(0x7fd8d00fb370 msgr2=0x7fd8d0105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 shutdown_connections 2024-04-25T01:24:46.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.202+0000 7fd8d6858640 1 -- 172.21.15.27:0/3260191116 wait complete. 2024-04-25T01:24:46.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 -- 172.21.15.27:0/867454149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c0104bf0 msgr2=0x7fc5c0106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/867454149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c0104bf0 0x7fc5c0106fe0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009900 tx=0x7fc5b002a410 comp rx=0 tx=0).stop 2024-04-25T01:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 -- 172.21.15.27:0/867454149 shutdown_connections 2024-04-25T01:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/867454149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5c0107520 0x7fc5c0109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/867454149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c0104bf0 0x7fc5c0106fe0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/867454149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c01022c0 0x7fc5c01046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 -- 172.21.15.27:0/867454149 >> 172.21.15.27:0/867454149 conn(0x7fc5c00fbc30 msgr2=0x7fc5c00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 -- 172.21.15.27:0/867454149 shutdown_connections 2024-04-25T01:24:46.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 -- 172.21.15.27:0/867454149 wait complete. 2024-04-25T01:24:46.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.318+0000 7fc5c60a5640 1 Processor -- start 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5c60a5640 1 -- start start 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5c60a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c01022c0 0x7fc5c0197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5c60a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5c0104bf0 0x7fc5c01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5c60a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c0107520 0x7fc5c019f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5c60a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c010d4f0 con 0x7fc5c01022c0 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5c60a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c010d370 con 0x7fc5c0104bf0 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5c60a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c010d670 con 0x7fc5c0107520 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c01022c0 0x7fc5c0197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c01022c0 0x7fc5c0197cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49728/0 (socket says 172.21.15.27:49728) 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bf7fe640 1 -- 172.21.15.27:0/2027472523 learned_addr learned my addr 172.21.15.27:0/2027472523 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5beffd640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5c0104bf0 0x7fc5c01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bffff640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c0107520 0x7fc5c019f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bf7fe640 1 -- 172.21.15.27:0/2027472523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c0107520 msgr2=0x7fc5c019f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bf7fe640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c0107520 0x7fc5c019f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bf7fe640 1 -- 172.21.15.27:0/2027472523 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5c0104bf0 msgr2=0x7fc5c01981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bf7fe640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5c0104bf0 0x7fc5c01981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5bf7fe640 1 -- 172.21.15.27:0/2027472523 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c019f870 con 0x7fc5c01022c0 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.319+0000 7fc5beffd640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5c0104bf0 0x7fc5c01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.320+0000 7fc5bf7fe640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c01022c0 0x7fc5c0197cb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400b4a0 tx=0x7fc5b400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.320+0000 7fc5bcff9640 1 -- 172.21.15.27:0/2027472523 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4014070 con 0x7fc5c01022c0 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.320+0000 7fc5bcff9640 1 -- 172.21.15.27:0/2027472523 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b4010af0 con 0x7fc5c01022c0 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.320+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c019fb60 con 0x7fc5c01022c0 2024-04-25T01:24:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.320+0000 7fc5bcff9640 1 -- 172.21.15.27:0/2027472523 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4017d10 con 0x7fc5c01022c0 2024-04-25T01:24:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.320+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c01a00b0 con 0x7fc5c01022c0 2024-04-25T01:24:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.321+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c00fd720 con 0x7fc5c01022c0 2024-04-25T01:24:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.322+0000 7fc5bcff9640 1 -- 172.21.15.27:0/2027472523 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc5b4017070 con 0x7fc5c01022c0 2024-04-25T01:24:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.323+0000 7fc5bcff9640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5a80a2c20 0x7fc5a80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.323+0000 7fc5beffd640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5a80a2c20 0x7fc5a80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.323+0000 7fc5bcff9640 1 -- 172.21.15.27:0/2027472523 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc5b40ce8d0 con 0x7fc5c01022c0 2024-04-25T01:24:46.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.327+0000 7fc5beffd640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5a80a2c20 0x7fc5a80a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006010 tx=0x7fc5b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:46.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.331+0000 7fc5bcff9640 1 -- 172.21.15.27:0/2027472523 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc5b4010ca0 con 0x7fc5c01022c0 2024-04-25T01:24:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.337+0000 7fc5bcff9640 1 -- 172.21.15.27:0/2027472523 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc5b409b480 con 0x7fc5c01022c0 2024-04-25T01:24:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.525+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc5c002d400 con 0x7fc5c01022c0 2024-04-25T01:24:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.526+0000 7fc5bcff9640 1 -- 172.21.15.27:0/2027472523 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7fc5b4085a00 con 0x7fc5c01022c0 2024-04-25T01:24:46.507 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5a80a2c20 msgr2=0x7fc5a80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5a80a2c20 0x7fc5a80a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006010 tx=0x7fc5b0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c01022c0 msgr2=0x7fc5c0197cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c01022c0 0x7fc5c0197cb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400b4a0 tx=0x7fc5b400b960 comp rx=0 tx=0).stop 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 shutdown_connections 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5a80a2c20 0x7fc5a80a50e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5c0107520 0x7fc5c019f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5c0104bf0 0x7fc5c01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 --2- 172.21.15.27:0/2027472523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c01022c0 0x7fc5c0197cb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 >> 172.21.15.27:0/2027472523 conn(0x7fc5c00fbc30 msgr2=0x7fc5c0102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 shutdown_connections 2024-04-25T01:24:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.528+0000 7fc5c60a5640 1 -- 172.21.15.27:0/2027472523 wait complete. 2024-04-25T01:24:46.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/1902719109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28103350 msgr2=0x7f0a28109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/1902719109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28103350 0x7f0a28109be0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009ca0 tx=0x7f0a2002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/1902719109 shutdown_connections 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/1902719109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28103350 0x7f0a28109be0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/1902719109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a28102990 0x7f0a28102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/1902719109 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a281017a0 0x7f0a28101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/1902719109 >> 172.21.15.27:0/1902719109 conn(0x7f0a280fcd10 msgr2=0x7f0a280ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/1902719109 shutdown_connections 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/1902719109 wait complete. 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.644+0000 7f0a2e1dc640 1 Processor -- start 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a2e1dc640 1 -- start start 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a2e1dc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a281017a0 0x7f0a28197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a2e1dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28102990 0x7f0a28197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a2e1dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a28103350 0x7f0a2819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a2e1dc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2810d420 con 0x7f0a28102990 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a2e1dc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2810d2a0 con 0x7f0a281017a0 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a2e1dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2810d5a0 con 0x7f0a28103350 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28102990 0x7f0a28197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28102990 0x7f0a28197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49742/0 (socket says 172.21.15.27:49742) 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 -- 172.21.15.27:0/195277145 learned_addr learned my addr 172.21.15.27:0/195277145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a277fe640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a281017a0 0x7f0a28197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a27fff640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a28103350 0x7f0a2819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 -- 172.21.15.27:0/195277145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a28103350 msgr2=0x7f0a2819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a28103350 0x7f0a2819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 -- 172.21.15.27:0/195277145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a281017a0 msgr2=0x7f0a28197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a281017a0 0x7f0a28197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 -- 172.21.15.27:0/195277145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2819f660 con 0x7f0a28102990 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.645+0000 7f0a26ffd640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28102990 0x7f0a28197f80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000d770 tx=0x7f0a1000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.646+0000 7f0a24ff9640 1 -- 172.21.15.27:0/195277145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10017070 con 0x7f0a28102990 2024-04-25T01:24:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.646+0000 7f0a24ff9640 1 -- 172.21.15.27:0/195277145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a10005500 con 0x7f0a28102990 2024-04-25T01:24:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.646+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2819f950 con 0x7f0a28102990 2024-04-25T01:24:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.646+0000 7f0a24ff9640 1 -- 172.21.15.27:0/195277145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10004e00 con 0x7f0a28102990 2024-04-25T01:24:46.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.646+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a2819fe50 con 0x7f0a28102990 2024-04-25T01:24:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.647+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a28108ac0 con 0x7f0a28102990 2024-04-25T01:24:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.648+0000 7f0a24ff9640 1 -- 172.21.15.27:0/195277145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0a10004fa0 con 0x7f0a28102990 2024-04-25T01:24:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.648+0000 7f0a24ff9640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0a140a2b90 0x7f0a140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.648+0000 7f0a277fe640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0a140a2b90 0x7f0a140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.648+0000 7f0a24ff9640 1 -- 172.21.15.27:0/195277145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9179+0+0 (secure 0 0 0) 0x7f0a100cee70 con 0x7f0a28102990 2024-04-25T01:24:46.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.649+0000 7f0a277fe640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0a140a2b90 0x7f0a140a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0a281027f0 tx=0x7f0a1800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:46.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.653+0000 7f0a24ff9640 1 -- 172.21.15.27:0/195277145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0a1009ba20 con 0x7f0a28102990 2024-04-25T01:24:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:46 smithi153 ceph-mon[32621]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:46 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4076900967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T01:24:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:24:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:46 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-25T01:24:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:46 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3260191116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:46.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:46 smithi153 ceph-mon[32621]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:24:46.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:46 smithi139 ceph-mon[32760]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:46.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:46 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4076900967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:46.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T01:24:46.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:24:46.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:46 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-25T01:24:46.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:46 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3260191116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:46.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:46 smithi139 ceph-mon[32760]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.836+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a28062340 con 0x7f0a28102990 2024-04-25T01:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.837+0000 7f0a24ff9640 1 -- 172.21.15.27:0/195277145 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f0a1009b320 con 0x7f0a28102990 2024-04-25T01:24:46.817 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0a140a2b90 msgr2=0x7f0a140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0a140a2b90 0x7f0a140a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0a281027f0 tx=0x7f0a1800b390 comp rx=0 tx=0).stop 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28102990 msgr2=0x7f0a28197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28102990 0x7f0a28197f80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000d770 tx=0x7f0a1000dc30 comp rx=0 tx=0).stop 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 shutdown_connections 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f0a140a2b90 0x7f0a140a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a28103350 0x7f0a2819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a28102990 0x7f0a28197f80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 --2- 172.21.15.27:0/195277145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a281017a0 0x7f0a28197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 >> 172.21.15.27:0/195277145 conn(0x7f0a280fcd10 msgr2=0x7f0a28105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 shutdown_connections 2024-04-25T01:24:46.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.839+0000 7f0a2e1dc640 1 -- 172.21.15.27:0/195277145 wait complete. 2024-04-25T01:24:46.830 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:46 smithi027 ceph-mon[29197]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:24:46.830 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:46 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4076900967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:46.830 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T01:24:46.830 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:24:46.830 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:46 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-25T01:24:46.830 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:46 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3260191116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:46.830 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:46 smithi027 ceph-mon[29197]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:24:46.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:24:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 -- 172.21.15.27:0/4009701361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70701017a0 msgr2=0x7f7070101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 --2- 172.21.15.27:0/4009701361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70701017a0 0x7f7070101bc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7064009900 tx=0x7f706402a410 comp rx=0 tx=0).stop 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 -- 172.21.15.27:0/4009701361 shutdown_connections 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 --2- 172.21.15.27:0/4009701361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7070103350 0x7f7070109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 --2- 172.21.15.27:0/4009701361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070102990 0x7f7070102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 --2- 172.21.15.27:0/4009701361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70701017a0 0x7f7070101bc0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 -- 172.21.15.27:0/4009701361 >> 172.21.15.27:0/4009701361 conn(0x7f70700fcd10 msgr2=0x7f70700ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 -- 172.21.15.27:0/4009701361 shutdown_connections 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 -- 172.21.15.27:0/4009701361 wait complete. 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 Processor -- start 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 -- start start 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70701017a0 0x7f7070197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.957+0000 7f7076a88640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070102990 0x7f70701981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7076a88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7070103350 0x7f707019f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7076a88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f707010d540 con 0x7f7070103350 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7076a88640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f707010d3c0 con 0x7f70701017a0 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7076a88640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f707010d6c0 con 0x7f7070102990 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7070103350 0x7f707019f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7070103350 0x7f707019f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49760/0 (socket says 172.21.15.27:49760) 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 -- 172.21.15.27:0/1539755171 learned_addr learned my addr 172.21.15.27:0/1539755171 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f706ffff640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070102990 0x7f70701981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7074824640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70701017a0 0x7f7070197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 -- 172.21.15.27:0/1539755171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070102990 msgr2=0x7f70701981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070102990 0x7f70701981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 -- 172.21.15.27:0/1539755171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70701017a0 msgr2=0x7f7070197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70701017a0 0x7f7070197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 -- 172.21.15.27:0/1539755171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707019f870 con 0x7f7070103350 2024-04-25T01:24:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f706ffff640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070102990 0x7f70701981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7074824640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70701017a0 0x7f7070197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7075025640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7070103350 0x7f707019f1d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f706800b970 tx=0x7f706800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f706dffb640 1 -- 172.21.15.27:0/1539755171 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068019070 con 0x7f7070103350 2024-04-25T01:24:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f706dffb640 1 -- 172.21.15.27:0/1539755171 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f706800cb40 con 0x7f7070103350 2024-04-25T01:24:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f707019fb60 con 0x7f7070103350 2024-04-25T01:24:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.958+0000 7f706dffb640 1 -- 172.21.15.27:0/1539755171 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068007650 con 0x7f7070103350 2024-04-25T01:24:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.959+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70701a00b0 con 0x7f7070103350 2024-04-25T01:24:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.959+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7070108b10 con 0x7f7070103350 2024-04-25T01:24:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.960+0000 7f706dffb640 1 -- 172.21.15.27:0/1539755171 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f706801c070 con 0x7f7070103350 2024-04-25T01:24:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.961+0000 7f706dffb640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70600a2c10 0x7f70600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.961+0000 7f7074824640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70600a2c10 0x7f70600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.961+0000 7f706dffb640 1 -- 172.21.15.27:0/1539755171 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9179+0+0 (secure 0 0 0) 0x7f70680ceab0 con 0x7f7070103350 2024-04-25T01:24:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.962+0000 7f7074824640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70600a2c10 0x7f70600a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7064006010 tx=0x7f7064000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:46.966+0000 7f706dffb640 1 -- 172.21.15.27:0/1539755171 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f706809b660 con 0x7f7070103350 2024-04-25T01:24:47.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.149+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7070062340 con 0x7f7070103350 2024-04-25T01:24:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.149+0000 7f706dffb640 1 -- 172.21.15.27:0/1539755171 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f706809af60 con 0x7f7070103350 2024-04-25T01:24:47.130 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70600a2c10 msgr2=0x7f70600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70600a2c10 0x7f70600a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7064006010 tx=0x7f7064000bc0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7070103350 msgr2=0x7f707019f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7070103350 0x7f707019f1d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f706800b970 tx=0x7f706800be30 comp rx=0 tx=0).stop 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 shutdown_connections 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f70600a2c10 0x7f70600a50d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7070103350 0x7f707019f1d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070102990 0x7f70701981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 --2- 172.21.15.27:0/1539755171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70701017a0 0x7f7070197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 >> 172.21.15.27:0/1539755171 conn(0x7f70700fcd10 msgr2=0x7f7070105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 shutdown_connections 2024-04-25T01:24:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.151+0000 7f7076a88640 1 -- 172.21.15.27:0/1539755171 wait complete. 2024-04-25T01:24:47.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 -- 172.21.15.27:0/1250621315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf4101890 msgr2=0x7f9bf4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/1250621315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf4101890 0x7f9bf4101d10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009900 tx=0x7f9be802a410 comp rx=0 tx=0).stop 2024-04-25T01:24:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 -- 172.21.15.27:0/1250621315 shutdown_connections 2024-04-25T01:24:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/1250621315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bf4102250 0x7f9bf4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/1250621315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf4101890 0x7f9bf4101d10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/1250621315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf41006a0 0x7f9bf4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 -- 172.21.15.27:0/1250621315 >> 172.21.15.27:0/1250621315 conn(0x7f9bf40fbc30 msgr2=0x7f9bf40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 -- 172.21.15.27:0/1250621315 shutdown_connections 2024-04-25T01:24:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 -- 172.21.15.27:0/1250621315 wait complete. 2024-04-25T01:24:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.264+0000 7f9bf93ee640 1 Processor -- start 2024-04-25T01:24:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf93ee640 1 -- start start 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf93ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf41006a0 0x7f9bf4104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf93ee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bf4101890 0x7f9bf4102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf93ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf4102250 0x7f9bf41031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf93ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf410c320 con 0x7f9bf41006a0 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf93ee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf410c1a0 con 0x7f9bf4101890 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf93ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf410c4a0 con 0x7f9bf4102250 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf41006a0 0x7f9bf4104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf41006a0 0x7f9bf4104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49780/0 (socket says 172.21.15.27:49780) 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 -- 172.21.15.27:0/3961507855 learned_addr learned my addr 172.21.15.27:0/3961507855 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf27fc640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bf4101890 0x7f9bf4102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf37fe640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf4102250 0x7f9bf41031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 -- 172.21.15.27:0/3961507855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf4102250 msgr2=0x7f9bf41031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf4102250 0x7f9bf41031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 -- 172.21.15.27:0/3961507855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bf4101890 msgr2=0x7f9bf4102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bf4101890 0x7f9bf4102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 -- 172.21.15.27:0/3961507855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf4103a40 con 0x7f9bf41006a0 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.265+0000 7f9bf2ffd640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf41006a0 0x7f9bf4104630 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9be400d950 tx=0x7f9be400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.266+0000 7f9bdbfff640 1 -- 172.21.15.27:0/3961507855 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4017070 con 0x7f9bf41006a0 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.266+0000 7f9bdbfff640 1 -- 172.21.15.27:0/3961507855 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be4009c30 con 0x7f9bf41006a0 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.266+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf41a1c10 con 0x7f9bf41006a0 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.266+0000 7f9bdbfff640 1 -- 172.21.15.27:0/3961507855 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be400f600 con 0x7f9bf41006a0 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.266+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf41a20e0 con 0x7f9bf41006a0 2024-04-25T01:24:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.267+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf41a22b0 con 0x7f9bf41006a0 2024-04-25T01:24:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.268+0000 7f9bdbfff640 1 -- 172.21.15.27:0/3961507855 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9be401a070 con 0x7f9bf41006a0 2024-04-25T01:24:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.268+0000 7f9bdbfff640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9bdc0a2b90 0x7f9bdc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.268+0000 7f9bf27fc640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9bdc0a2b90 0x7f9bdc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.268+0000 7f9bdbfff640 1 -- 172.21.15.27:0/3961507855 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9179+0+0 (secure 0 0 0) 0x7f9be40cee90 con 0x7f9bf41006a0 2024-04-25T01:24:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.269+0000 7f9bf27fc640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9bdc0a2b90 0x7f9bdc0a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9be80314e0 tx=0x7f9be8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.273+0000 7f9bdbfff640 1 -- 172.21.15.27:0/3961507855 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9be4009de0 con 0x7f9bf41006a0 2024-04-25T01:24:47.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.501+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bf41a2640 con 0x7f9bf41006a0 2024-04-25T01:24:47.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.501+0000 7f9bdbfff640 1 -- 172.21.15.27:0/3961507855 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f9be409ba40 con 0x7f9bf41006a0 2024-04-25T01:24:47.482 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:47.482 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:23:23.940979+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:24:47.483 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9bdc0a2b90 msgr2=0x7f9bdc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9bdc0a2b90 0x7f9bdc0a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9be80314e0 tx=0x7f9be8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf41006a0 msgr2=0x7f9bf4104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf41006a0 0x7f9bf4104630 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9be400d950 tx=0x7f9be400de10 comp rx=0 tx=0).stop 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 shutdown_connections 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f9bdc0a2b90 0x7f9bdc0a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf4102250 0x7f9bf41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9bf4101890 0x7f9bf4102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 --2- 172.21.15.27:0/3961507855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf41006a0 0x7f9bf4104630 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 >> 172.21.15.27:0/3961507855 conn(0x7f9bf40fbc30 msgr2=0x7f9bf40fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:47.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 shutdown_connections 2024-04-25T01:24:47.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.504+0000 7f9bf93ee640 1 -- 172.21.15.27:0/3961507855 wait complete. 2024-04-25T01:24:47.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 -- 172.21.15.27:0/3019164494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3141006a0 msgr2=0x7fe314100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 --2- 172.21.15.27:0/3019164494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3141006a0 0x7fe314100ac0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe304009900 tx=0x7fe30402a410 comp rx=0 tx=0).stop 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 -- 172.21.15.27:0/3019164494 shutdown_connections 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 --2- 172.21.15.27:0/3019164494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe314102250 0x7fe314108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 --2- 172.21.15.27:0/3019164494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe314101890 0x7fe314101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 --2- 172.21.15.27:0/3019164494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3141006a0 0x7fe314100ac0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 -- 172.21.15.27:0/3019164494 >> 172.21.15.27:0/3019164494 conn(0x7fe3140fbc30 msgr2=0x7fe3140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 -- 172.21.15.27:0/3019164494 shutdown_connections 2024-04-25T01:24:47.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.618+0000 7fe3196fc640 1 -- 172.21.15.27:0/3019164494 wait complete. 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3196fc640 1 Processor -- start 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3196fc640 1 -- start start 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3196fc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3141006a0 0x7fe314197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3196fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe314101890 0x7fe314197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3196fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe314102250 0x7fe31419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3196fc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe31410d440 con 0x7fe314101890 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3196fc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe31410d2c0 con 0x7fe3141006a0 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3196fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe31410d5c0 con 0x7fe314102250 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe30bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe314101890 0x7fe314197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe30bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe314101890 0x7fe314197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49792/0 (socket says 172.21.15.27:49792) 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe30bfff640 1 -- 172.21.15.27:0/4159710025 learned_addr learned my addr 172.21.15.27:0/4159710025 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe3137fe640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe314102250 0x7fe31419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe312ffd640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3141006a0 0x7fe314197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe30bfff640 1 -- 172.21.15.27:0/4159710025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe314102250 msgr2=0x7fe31419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe30bfff640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe314102250 0x7fe31419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe30bfff640 1 -- 172.21.15.27:0/4159710025 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3141006a0 msgr2=0x7fe314197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.619+0000 7fe30bfff640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3141006a0 0x7fe314197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.620+0000 7fe30bfff640 1 -- 172.21.15.27:0/4159710025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31419f680 con 0x7fe314101890 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.620+0000 7fe3137fe640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe314102250 0x7fe31419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.620+0000 7fe30bfff640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe314101890 0x7fe314197fa0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00c920 tx=0x7fe2fc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.620+0000 7fe310ff9640 1 -- 172.21.15.27:0/4159710025 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc007c70 con 0x7fe314101890 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.620+0000 7fe310ff9640 1 -- 172.21.15.27:0/4159710025 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2fc007e10 con 0x7fe314101890 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.620+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31419f970 con 0x7fe314101890 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.620+0000 7fe310ff9640 1 -- 172.21.15.27:0/4159710025 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc018890 con 0x7fe314101890 2024-04-25T01:24:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.620+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31419ff10 con 0x7fe314101890 2024-04-25T01:24:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.621+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3141079c0 con 0x7fe314101890 2024-04-25T01:24:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.622+0000 7fe310ff9640 1 -- 172.21.15.27:0/4159710025 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe2fc018a70 con 0x7fe314101890 2024-04-25T01:24:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.623+0000 7fe310ff9640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe3000a2c10 0x7fe3000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.623+0000 7fe312ffd640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe3000a2c10 0x7fe3000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.623+0000 7fe310ff9640 1 -- 172.21.15.27:0/4159710025 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fe2fc0cf550 con 0x7fe314101890 2024-04-25T01:24:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.623+0000 7fe312ffd640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe3000a2c10 0x7fe3000a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe304006010 tx=0x7fe304000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.628+0000 7fe310ff9640 1 -- 172.21.15.27:0/4159710025 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe2fc014030 con 0x7fe314101890 2024-04-25T01:24:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T01:24:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:24:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:47 smithi153 ceph-mon[32621]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-25T01:24:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:47 smithi153 ceph-mon[32621]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T01:24:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:47 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2027472523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:47 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/195277145' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:47 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1539755171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T01:24:47.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:24:47.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:47 smithi139 ceph-mon[32760]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-25T01:24:47.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:47 smithi139 ceph-mon[32760]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T01:24:47.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:47 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2027472523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:47 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/195277145' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:47 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1539755171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.814+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe314062340 con 0x7fe314101890 2024-04-25T01:24:47.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.816+0000 7fe310ff9640 1 -- 172.21.15.27:0/4159710025 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24792 (secure 0 0 0) 0x7fe2fc09c100 con 0x7fe314101890 2024-04-25T01:24:47.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:47.797 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:24:47.477088+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe3000a2c10 msgr2=0x7fe3000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe3000a2c10 0x7fe3000a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe304006010 tx=0x7fe304000bc0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe314101890 msgr2=0x7fe314197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe314101890 0x7fe314197fa0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00c920 tx=0x7fe2fc00cde0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 shutdown_connections 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe3000a2c10 0x7fe3000a50d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe314102250 0x7fe31419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe314101890 0x7fe314197fa0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 --2- 172.21.15.27:0/4159710025 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3141006a0 0x7fe314197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 >> 172.21.15.27:0/4159710025 conn(0x7fe3140fbc30 msgr2=0x7fe314104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:47.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 shutdown_connections 2024-04-25T01:24:47.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.818+0000 7fe3196fc640 1 -- 172.21.15.27:0/4159710025 wait complete. 2024-04-25T01:24:47.809 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T01:24:47.809 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:24:47.809 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:47 smithi027 ceph-mon[29197]: from='mgr.14152 172.21.15.27:0/2951174866' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-25T01:24:47.809 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:47 smithi027 ceph-mon[29197]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T01:24:47.809 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:47 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2027472523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.809 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:47 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/195277145' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.809 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:47 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1539755171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:24:47.810 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:24:47.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:24:47.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 -- 172.21.15.27:0/2537422068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 msgr2=0x7fec50100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 --2- 172.21.15.27:0/2537422068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 0x7fec50100ac0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fec38009900 tx=0x7fec3802a410 comp rx=0 tx=0).stop 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 -- 172.21.15.27:0/2537422068 shutdown_connections 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 --2- 172.21.15.27:0/2537422068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec50102250 0x7fec50108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 --2- 172.21.15.27:0/2537422068 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec50101890 0x7fec50101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 --2- 172.21.15.27:0/2537422068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 0x7fec50100ac0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 -- 172.21.15.27:0/2537422068 >> 172.21.15.27:0/2537422068 conn(0x7fec500fbc30 msgr2=0x7fec500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 -- 172.21.15.27:0/2537422068 shutdown_connections 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.935+0000 7fec54c38640 1 -- 172.21.15.27:0/2537422068 wait complete. 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec54c38640 1 Processor -- start 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec54c38640 1 -- start start 2024-04-25T01:24:47.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec54c38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 0x7fec50196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec54c38640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec50101890 0x7fec50196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec54c38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec50102250 0x7fec5019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec54c38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010c370 con 0x7fec501006a0 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec54c38640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010c1f0 con 0x7fec50101890 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec54c38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010c4f0 con 0x7fec50102250 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 0x7fec50196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 0x7fec50196aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35134/0 (socket says 172.21.15.27:35134) 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 -- 172.21.15.27:0/1936703147 learned_addr learned my addr 172.21.15.27:0/1936703147 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4dd9b640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec50101890 0x7fec50196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4ed9d640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec50102250 0x7fec5019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 -- 172.21.15.27:0/1936703147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec50102250 msgr2=0x7fec5019e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec50102250 0x7fec5019e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 -- 172.21.15.27:0/1936703147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec50101890 msgr2=0x7fec50196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec50101890 0x7fec50196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 -- 172.21.15.27:0/1936703147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec50069d40 con 0x7fec501006a0 2024-04-25T01:24:47.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4ed9d640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec50102250 0x7fec5019e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.936+0000 7fec4e59c640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 0x7fec50196aa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fec38031610 tx=0x7fec380043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.937+0000 7fec477fe640 1 -- 172.21.15.27:0/1936703147 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3803f030 con 0x7fec501006a0 2024-04-25T01:24:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.937+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec50069fd0 con 0x7fec501006a0 2024-04-25T01:24:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.937+0000 7fec477fe640 1 -- 172.21.15.27:0/1936703147 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec38003ce0 con 0x7fec501006a0 2024-04-25T01:24:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.937+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec5006a510 con 0x7fec501006a0 2024-04-25T01:24:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.937+0000 7fec477fe640 1 -- 172.21.15.27:0/1936703147 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38049060 con 0x7fec501006a0 2024-04-25T01:24:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.938+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec50107a30 con 0x7fec501006a0 2024-04-25T01:24:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.939+0000 7fec477fe640 1 -- 172.21.15.27:0/1936703147 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fec38042030 con 0x7fec501006a0 2024-04-25T01:24:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.940+0000 7fec477fe640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fec3c0a2b90 0x7fec3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.940+0000 7fec4dd9b640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fec3c0a2b90 0x7fec3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.940+0000 7fec477fe640 1 -- 172.21.15.27:0/1936703147 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fec380eff60 con 0x7fec501006a0 2024-04-25T01:24:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.940+0000 7fec4dd9b640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fec3c0a2b90 0x7fec3c0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fec50197fb0 tx=0x7fec40009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:47.943+0000 7fec477fe640 1 -- 172.21.15.27:0/1936703147 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fec380bcb10 con 0x7fec501006a0 2024-04-25T01:24:48.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.129+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec5006a830 con 0x7fec501006a0 2024-04-25T01:24:48.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.129+0000 7fec477fe640 1 -- 172.21.15.27:0/1936703147 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fec380bc410 con 0x7fec501006a0 2024-04-25T01:24:48.110 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fec3c0a2b90 msgr2=0x7fec3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fec3c0a2b90 0x7fec3c0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fec50197fb0 tx=0x7fec40009110 comp rx=0 tx=0).stop 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 msgr2=0x7fec50196aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 0x7fec50196aa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fec38031610 tx=0x7fec380043b0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 shutdown_connections 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fec3c0a2b90 0x7fec3c0a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec50102250 0x7fec5019e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec50101890 0x7fec50196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 --2- 172.21.15.27:0/1936703147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec501006a0 0x7fec50196aa0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 >> 172.21.15.27:0/1936703147 conn(0x7fec500fbc30 msgr2=0x7fec50104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 shutdown_connections 2024-04-25T01:24:48.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.131+0000 7fec54c38640 1 -- 172.21.15.27:0/1936703147 wait complete. 2024-04-25T01:24:48.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 -- 172.21.15.27:0/1290675122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f4102250 msgr2=0x7fb4f4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 --2- 172.21.15.27:0/1290675122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f4102250 0x7fb4f4108ae0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009ca0 tx=0x7fb4ec02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 -- 172.21.15.27:0/1290675122 shutdown_connections 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 --2- 172.21.15.27:0/1290675122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f4102250 0x7fb4f4108ae0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 --2- 172.21.15.27:0/1290675122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101890 0x7fb4f4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 --2- 172.21.15.27:0/1290675122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4f41006a0 0x7fb4f4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 -- 172.21.15.27:0/1290675122 >> 172.21.15.27:0/1290675122 conn(0x7fb4f40fbc30 msgr2=0x7fb4f40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 -- 172.21.15.27:0/1290675122 shutdown_connections 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 -- 172.21.15.27:0/1290675122 wait complete. 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 Processor -- start 2024-04-25T01:24:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.247+0000 7fb4faf63640 1 -- start start 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4faf63640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f41006a0 0x7fb4f41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4faf63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101890 0x7fb4f4196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4faf63640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4f4102250 0x7fb4f419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4faf63640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410c350 con 0x7fb4f41006a0 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4faf63640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410c1d0 con 0x7fb4f4102250 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4faf63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410c4d0 con 0x7fb4f4101890 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f41006a0 0x7fb4f41968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f41006a0 0x7fb4f41968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35148/0 (socket says 172.21.15.27:35148) 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 -- 172.21.15.27:0/3548326966 learned_addr learned my addr 172.21.15.27:0/3548326966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f9500640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4f4102250 0x7fb4f419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f3fff640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101890 0x7fb4f4196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 -- 172.21.15.27:0/3548326966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101890 msgr2=0x7fb4f4196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101890 0x7fb4f4196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 -- 172.21.15.27:0/3548326966 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4f4102250 msgr2=0x7fb4f419de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4f4102250 0x7fb4f419de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 -- 172.21.15.27:0/3548326966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f419e4f0 con 0x7fb4f41006a0 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f9500640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4f4102250 0x7fb4f419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f8cff640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f41006a0 0x7fb4f41968d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8007b50 tx=0x7fb4e800e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f1ffb640 1 -- 172.21.15.27:0/3548326966 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8007c40 con 0x7fb4f41006a0 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f1ffb640 1 -- 172.21.15.27:0/3548326966 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4e8007de0 con 0x7fb4f41006a0 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f4069da0 con 0x7fb4f41006a0 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.248+0000 7fb4f1ffb640 1 -- 172.21.15.27:0/3548326966 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e800adb0 con 0x7fb4f41006a0 2024-04-25T01:24:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.249+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f406a2f0 con 0x7fb4f41006a0 2024-04-25T01:24:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.250+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f40689b0 con 0x7fb4f41006a0 2024-04-25T01:24:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.251+0000 7fb4f1ffb640 1 -- 172.21.15.27:0/3548326966 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb4e800a440 con 0x7fb4f41006a0 2024-04-25T01:24:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.251+0000 7fb4f1ffb640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4e40a2b90 0x7fb4e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.252+0000 7fb4f3fff640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4e40a2b90 0x7fb4e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.252+0000 7fb4f1ffb640 1 -- 172.21.15.27:0/3548326966 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fb4e80d0b80 con 0x7fb4f41006a0 2024-04-25T01:24:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.252+0000 7fb4f3fff640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4e40a2b90 0x7fb4e40a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0005fd0 tx=0x7fb4e0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:48.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.256+0000 7fb4f1ffb640 1 -- 172.21.15.27:0/3548326966 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb4e8023020 con 0x7fb4f41006a0 2024-04-25T01:24:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.440+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb4f41073d0 con 0x7fb4f41006a0 2024-04-25T01:24:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.440+0000 7fb4f1ffb640 1 -- 172.21.15.27:0/3548326966 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fb4f41073d0 con 0x7fb4f41006a0 2024-04-25T01:24:48.421 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:24:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4e40a2b90 msgr2=0x7fb4e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4e40a2b90 0x7fb4e40a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0005fd0 tx=0x7fb4e0005850 comp rx=0 tx=0).stop 2024-04-25T01:24:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f41006a0 msgr2=0x7fb4f41968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f41006a0 0x7fb4f41968d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8007b50 tx=0x7fb4e800e420 comp rx=0 tx=0).stop 2024-04-25T01:24:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 shutdown_connections 2024-04-25T01:24:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4e40a2b90 0x7fb4e40a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb4f4102250 0x7fb4f419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f4101890 0x7fb4f4196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 --2- 172.21.15.27:0/3548326966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4f41006a0 0x7fb4f41968d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.442+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 >> 172.21.15.27:0/3548326966 conn(0x7fb4f40fbc30 msgr2=0x7fb4f4104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.443+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 shutdown_connections 2024-04-25T01:24:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.443+0000 7fb4faf63640 1 -- 172.21.15.27:0/3548326966 wait complete. 2024-04-25T01:24:48.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:24:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 -- 172.21.15.27:0/3649433279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c102230 msgr2=0x7f264c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 --2- 172.21.15.27:0/3649433279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c102230 0x7f264c108ac0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2644009ca0 tx=0x7f264402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 -- 172.21.15.27:0/3649433279 shutdown_connections 2024-04-25T01:24:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 --2- 172.21.15.27:0/3649433279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c102230 0x7f264c108ac0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 --2- 172.21.15.27:0/3649433279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f264c101870 0x7f264c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 --2- 172.21.15.27:0/3649433279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f264c100680 0x7f264c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 -- 172.21.15.27:0/3649433279 >> 172.21.15.27:0/3649433279 conn(0x7f264c0fbbf0 msgr2=0x7f264c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 -- 172.21.15.27:0/3649433279 shutdown_connections 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 -- 172.21.15.27:0/3649433279 wait complete. 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 Processor -- start 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.563+0000 7f2651948640 1 -- start start 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2651948640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f264c100680 0x7f264c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2651948640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c101870 0x7f264c196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2651948640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f264c102230 0x7f264c19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2651948640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f264c10c300 con 0x7f264c101870 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2651948640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f264c10c180 con 0x7f264c102230 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2651948640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f264c10c480 con 0x7f264c100680 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c101870 0x7f264c196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c101870 0x7f264c196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35156/0 (socket says 172.21.15.27:35156) 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 -- 172.21.15.27:0/2536941916 learned_addr learned my addr 172.21.15.27:0/2536941916 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f264affd640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f264c100680 0x7f264c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f264b7fe640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f264c102230 0x7f264c19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 -- 172.21.15.27:0/2536941916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f264c100680 msgr2=0x7f264c196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f264c100680 0x7f264c196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 -- 172.21.15.27:0/2536941916 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f264c102230 msgr2=0x7f264c19dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f264c102230 0x7f264c19dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 -- 172.21.15.27:0/2536941916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f264c19e530 con 0x7f264c101870 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f264b7fe640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f264c102230 0x7f264c19dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2643fff640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c101870 0x7f264c196e50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f263400d770 tx=0x7f263400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2650946640 1 -- 172.21.15.27:0/2536941916 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634017070 con 0x7f264c101870 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f264c19e820 con 0x7f264c101870 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2650946640 1 -- 172.21.15.27:0/2536941916 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2634005500 con 0x7f264c101870 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.564+0000 7f2650946640 1 -- 172.21.15.27:0/2536941916 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634004e00 con 0x7f264c101870 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.565+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f264c19ed20 con 0x7f264c101870 2024-04-25T01:24:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.565+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f264c107940 con 0x7f264c101870 2024-04-25T01:24:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.567+0000 7f2650946640 1 -- 172.21.15.27:0/2536941916 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2634004fa0 con 0x7f264c101870 2024-04-25T01:24:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.567+0000 7f2650946640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f26380a2b90 0x7f26380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.567+0000 7f264affd640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f26380a2b90 0x7f26380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.567+0000 7f2650946640 1 -- 172.21.15.27:0/2536941916 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f26340cf2a0 con 0x7f264c101870 2024-04-25T01:24:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.568+0000 7f264affd640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f26380a2b90 0x7f26380a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f264c1016d0 tx=0x7f263c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.571+0000 7f2650946640 1 -- 172.21.15.27:0/2536941916 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f263409be50 con 0x7f264c101870 2024-04-25T01:24:48.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:48 smithi027 ceph-mon[29197]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:24:48.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:48 smithi027 ceph-mon[29197]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T01:24:48.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:48 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3961507855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:48.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:48 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4159710025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:48.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:48 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1936703147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:48.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:48 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3548326966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:24:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.754+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f264c02d300 con 0x7f264c101870 2024-04-25T01:24:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.755+0000 7f2650946640 1 -- 172.21.15.27:0/2536941916 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f263409b750 con 0x7f264c101870 2024-04-25T01:24:48.735 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:24:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f26380a2b90 msgr2=0x7f26380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f26380a2b90 0x7f26380a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f264c1016d0 tx=0x7f263c005f90 comp rx=0 tx=0).stop 2024-04-25T01:24:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c101870 msgr2=0x7f264c196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c101870 0x7f264c196e50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f263400d770 tx=0x7f263400dc30 comp rx=0 tx=0).stop 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 shutdown_connections 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f26380a2b90 0x7f26380a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f264c102230 0x7f264c19dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c101870 0x7f264c196e50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 --2- 172.21.15.27:0/2536941916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f264c100680 0x7f264c196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 >> 172.21.15.27:0/2536941916 conn(0x7f264c0fbbf0 msgr2=0x7f264c104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 shutdown_connections 2024-04-25T01:24:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.757+0000 7f2651948640 1 -- 172.21.15.27:0/2536941916 wait complete. 2024-04-25T01:24:48.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:24:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:48 smithi153 ceph-mon[32621]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:24:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:48 smithi153 ceph-mon[32621]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T01:24:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:48 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3961507855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:48 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4159710025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:48 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1936703147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:48.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:48 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3548326966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:24:48.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:48 smithi139 ceph-mon[32760]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:24:48.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:48 smithi139 ceph-mon[32760]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T01:24:48.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:48 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3961507855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:48.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:48 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4159710025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:48.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:48 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1936703147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:48.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:48 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3548326966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:24:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 -- 172.21.15.27:0/1064785140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c102200 msgr2=0x7f3e8c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1064785140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c102200 0x7f3e8c1045f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009ca0 tx=0x7f3e8402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 -- 172.21.15.27:0/1064785140 shutdown_connections 2024-04-25T01:24:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1064785140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e8c107460 0x7f3e8c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1064785140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8c104b30 0x7f3e8c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1064785140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c102200 0x7f3e8c1045f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 -- 172.21.15.27:0/1064785140 >> 172.21.15.27:0/1064785140 conn(0x7f3e8c0fbbf0 msgr2=0x7f3e8c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 -- 172.21.15.27:0/1064785140 shutdown_connections 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.873+0000 7f3e912f5640 1 -- 172.21.15.27:0/1064785140 wait complete. 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e912f5640 1 Processor -- start 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e912f5640 1 -- start start 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e912f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8c102200 0x7f3e8c101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e912f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c104b30 0x7f3e8c1001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e912f5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e8c107460 0x7f3e8c1006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e912f5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c10d430 con 0x7f3e8c104b30 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e912f5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c10d2b0 con 0x7f3e8c107460 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e912f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c10d5b0 con 0x7f3e8c102200 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e8a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c104b30 0x7f3e8c1001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e8a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c104b30 0x7f3e8c1001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35188/0 (socket says 172.21.15.27:35188) 2024-04-25T01:24:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.874+0000 7f3e8a7fc640 1 -- 172.21.15.27:0/1104784726 learned_addr learned my addr 172.21.15.27:0/1104784726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8b7fe640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e8c107460 0x7f3e8c1006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8affd640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8c102200 0x7f3e8c101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8a7fc640 1 -- 172.21.15.27:0/1104784726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8c102200 msgr2=0x7f3e8c101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8a7fc640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8c102200 0x7f3e8c101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8a7fc640 1 -- 172.21.15.27:0/1104784726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e8c107460 msgr2=0x7f3e8c1006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8a7fc640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e8c107460 0x7f3e8c1006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8a7fc640 1 -- 172.21.15.27:0/1104784726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c100f80 con 0x7f3e8c104b30 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8affd640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8c102200 0x7f3e8c101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e8a7fc640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c104b30 0x7f3e8c1001b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000c970 tx=0x7f3e8000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e73fff640 1 -- 172.21.15.27:0/1104784726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80007bb0 con 0x7f3e8c104b30 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e73fff640 1 -- 172.21.15.27:0/1104784726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e80007d50 con 0x7f3e8c104b30 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c1a1c60 con 0x7f3e8c104b30 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e73fff640 1 -- 172.21.15.27:0/1104784726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e800187a0 con 0x7f3e8c104b30 2024-04-25T01:24:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.875+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c1a2160 con 0x7f3e8c104b30 2024-04-25T01:24:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.876+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e8c0fd6e0 con 0x7f3e8c104b30 2024-04-25T01:24:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.877+0000 7f3e73fff640 1 -- 172.21.15.27:0/1104784726 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3e8000f450 con 0x7f3e8c104b30 2024-04-25T01:24:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.878+0000 7f3e73fff640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e740a2b90 0x7f3e740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.878+0000 7f3e8affd640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e740a2b90 0x7f3e740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.878+0000 7f3e73fff640 1 -- 172.21.15.27:0/1104784726 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f3e800cf490 con 0x7f3e8c104b30 2024-04-25T01:24:48.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.878+0000 7f3e8affd640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e740a2b90 0x7f3e740a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3e84031c10 tx=0x7f3e84009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:48.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:48.882+0000 7f3e73fff640 1 -- 172.21.15.27:0/1104784726 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3e80014030 con 0x7f3e8c104b30 2024-04-25T01:24:49.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.066+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3e8c062340 con 0x7f3e8c104b30 2024-04-25T01:24:49.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.066+0000 7f3e73fff640 1 -- 172.21.15.27:0/1104784726 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f3e8009c040 con 0x7f3e8c104b30 2024-04-25T01:24:49.047 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:24:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e740a2b90 msgr2=0x7f3e740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e740a2b90 0x7f3e740a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3e84031c10 tx=0x7f3e84009360 comp rx=0 tx=0).stop 2024-04-25T01:24:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c104b30 msgr2=0x7f3e8c1001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c104b30 0x7f3e8c1001b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000c970 tx=0x7f3e8000ce30 comp rx=0 tx=0).stop 2024-04-25T01:24:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 shutdown_connections 2024-04-25T01:24:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f3e740a2b90 0x7f3e740a5050 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e8c107460 0x7f3e8c1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e8c104b30 0x7f3e8c1001b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 --2- 172.21.15.27:0/1104784726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e8c102200 0x7f3e8c101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 >> 172.21.15.27:0/1104784726 conn(0x7f3e8c0fbbf0 msgr2=0x7f3e8c102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 shutdown_connections 2024-04-25T01:24:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.068+0000 7f3e912f5640 1 -- 172.21.15.27:0/1104784726 wait complete. 2024-04-25T01:24:49.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 -- 172.21.15.27:0/3283592211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb628103360 msgr2=0x7fb628109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/3283592211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb628103360 0x7fb628109bf0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb620009ca0 tx=0x7fb62002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 -- 172.21.15.27:0/3283592211 shutdown_connections 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/3283592211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb628103360 0x7fb628109bf0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/3283592211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6281029a0 0x7fb628102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/3283592211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb6281017b0 0x7fb628101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 -- 172.21.15.27:0/3283592211 >> 172.21.15.27:0/3283592211 conn(0x7fb6280fcd40 msgr2=0x7fb6280ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 -- 172.21.15.27:0/3283592211 shutdown_connections 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 -- 172.21.15.27:0/3283592211 wait complete. 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 Processor -- start 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.183+0000 7fb62dbd5640 1 -- start start 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb62dbd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6281017b0 0x7fb628197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb62dbd5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb6281029a0 0x7fb628197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb62dbd5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb628103360 0x7fb62819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb62dbd5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810d410 con 0x7fb6281017b0 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb62dbd5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810d290 con 0x7fb6281029a0 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb62dbd5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810d590 con 0x7fb628103360 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6281017b0 0x7fb628197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6281017b0 0x7fb628197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35202/0 (socket says 172.21.15.27:35202) 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 -- 172.21.15.27:0/607610237 learned_addr learned my addr 172.21.15.27:0/607610237 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb626ffd640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb6281029a0 0x7fb628197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb627fff640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb628103360 0x7fb62819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 -- 172.21.15.27:0/607610237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb628103360 msgr2=0x7fb62819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb628103360 0x7fb62819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 -- 172.21.15.27:0/607610237 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb6281029a0 msgr2=0x7fb628197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb6281029a0 0x7fb628197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 -- 172.21.15.27:0/607610237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62819f640 con 0x7fb6281017b0 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb6277fe640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6281017b0 0x7fb628197a20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb61800e9c0 tx=0x7fb61800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb624ff9640 1 -- 172.21.15.27:0/607610237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61800cd80 con 0x7fb6281017b0 2024-04-25T01:24:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.184+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62819f930 con 0x7fb6281017b0 2024-04-25T01:24:49.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.185+0000 7fb624ff9640 1 -- 172.21.15.27:0/607610237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6180035f0 con 0x7fb6281017b0 2024-04-25T01:24:49.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.185+0000 7fb624ff9640 1 -- 172.21.15.27:0/607610237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618010690 con 0x7fb6281017b0 2024-04-25T01:24:49.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.185+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb62819fed0 con 0x7fb6281017b0 2024-04-25T01:24:49.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.185+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6280689b0 con 0x7fb6281017b0 2024-04-25T01:24:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.187+0000 7fb624ff9640 1 -- 172.21.15.27:0/607610237 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb6180037a0 con 0x7fb6281017b0 2024-04-25T01:24:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.187+0000 7fb624ff9640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb6100a2b90 0x7fb6100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.188+0000 7fb626ffd640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb6100a2b90 0x7fb6100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.188+0000 7fb624ff9640 1 -- 172.21.15.27:0/607610237 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fb6180cf870 con 0x7fb6281017b0 2024-04-25T01:24:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.188+0000 7fb626ffd640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb6100a2b90 0x7fb6100a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb628198f30 tx=0x7fb61c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:49.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.192+0000 7fb624ff9640 1 -- 172.21.15.27:0/607610237 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb61809c420 con 0x7fb6281017b0 2024-04-25T01:24:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.417+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb628108a00 con 0x7fb6281017b0 2024-04-25T01:24:49.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.417+0000 7fb624ff9640 1 -- 172.21.15.27:0/607610237 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7fb61809bd20 con 0x7fb6281017b0 2024-04-25T01:24:49.398 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:49.398 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:23:23.940979+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:24:49.399 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-25T01:24:49.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.419+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb6100a2b90 msgr2=0x7fb6100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.419+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb6100a2b90 0x7fb6100a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb628198f30 tx=0x7fb61c00b410 comp rx=0 tx=0).stop 2024-04-25T01:24:49.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.419+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6281017b0 msgr2=0x7fb628197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.419+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6281017b0 0x7fb628197a20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb61800e9c0 tx=0x7fb61800ee80 comp rx=0 tx=0).stop 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.420+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 shutdown_connections 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.420+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb6100a2b90 0x7fb6100a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.420+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb628103360 0x7fb62819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.420+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb6281029a0 0x7fb628197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.420+0000 7fb62dbd5640 1 --2- 172.21.15.27:0/607610237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6281017b0 0x7fb628197a20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.420+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 >> 172.21.15.27:0/607610237 conn(0x7fb6280fcd40 msgr2=0x7fb628105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.420+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 shutdown_connections 2024-04-25T01:24:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.420+0000 7fb62dbd5640 1 -- 172.21.15.27:0/607610237 wait complete. 2024-04-25T01:24:49.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:49.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 -- 172.21.15.27:0/1026138386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4960102250 msgr2=0x7f4960108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 --2- 172.21.15.27:0/1026138386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4960102250 0x7f4960108ae0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4958009ca0 tx=0x7f495802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 -- 172.21.15.27:0/1026138386 shutdown_connections 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 --2- 172.21.15.27:0/1026138386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4960102250 0x7f4960108ae0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 --2- 172.21.15.27:0/1026138386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4960101890 0x7f4960101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 --2- 172.21.15.27:0/1026138386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49601006a0 0x7f4960100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 -- 172.21.15.27:0/1026138386 >> 172.21.15.27:0/1026138386 conn(0x7f49600fbc30 msgr2=0x7f49600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 -- 172.21.15.27:0/1026138386 shutdown_connections 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 -- 172.21.15.27:0/1026138386 wait complete. 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.534+0000 7f4965438640 1 Processor -- start 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f4965438640 1 -- start start 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f4965438640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49601006a0 0x7f4960197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f4965438640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4960101890 0x7f4960197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f4965438640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4960102250 0x7f496019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f4965438640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f496010c320 con 0x7f49601006a0 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f4965438640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f496010c1a0 con 0x7f4960102250 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f4965438640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f496010c4a0 con 0x7f4960101890 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49601006a0 0x7f4960197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49601006a0 0x7f4960197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35222/0 (socket says 172.21.15.27:35222) 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 -- 172.21.15.27:0/3549921074 learned_addr learned my addr 172.21.15.27:0/3549921074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495f7fe640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4960102250 0x7f496019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495e7fc640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4960101890 0x7f4960197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 -- 172.21.15.27:0/3549921074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4960101890 msgr2=0x7f4960197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4960101890 0x7f4960197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 -- 172.21.15.27:0/3549921074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4960102250 msgr2=0x7f496019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4960102250 0x7f496019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 -- 172.21.15.27:0/3549921074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496019f630 con 0x7f49601006a0 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495e7fc640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4960101890 0x7f4960197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.535+0000 7f495effd640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49601006a0 0x7f4960197a60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f495000d770 tx=0x7f495000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.536+0000 7f4957fff640 1 -- 172.21.15.27:0/3549921074 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950017070 con 0x7f49601006a0 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.536+0000 7f4957fff640 1 -- 172.21.15.27:0/3549921074 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4950009c30 con 0x7f49601006a0 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.536+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496019f920 con 0x7f49601006a0 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.536+0000 7f4957fff640 1 -- 172.21.15.27:0/3549921074 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950003dd0 con 0x7f49601006a0 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.536+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496019fe70 con 0x7f49601006a0 2024-04-25T01:24:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.537+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49600689b0 con 0x7f49601006a0 2024-04-25T01:24:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.538+0000 7f4957fff640 1 -- 172.21.15.27:0/3549921074 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4950003fb0 con 0x7f49601006a0 2024-04-25T01:24:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.538+0000 7f4957fff640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f494c0a2b90 0x7f494c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.539+0000 7f495e7fc640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f494c0a2b90 0x7f494c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.539+0000 7f4957fff640 1 -- 172.21.15.27:0/3549921074 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f49500ced80 con 0x7f49601006a0 2024-04-25T01:24:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.539+0000 7f495e7fc640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f494c0a2b90 0x7f494c0a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4960198f20 tx=0x7f4948005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.543+0000 7f4957fff640 1 -- 172.21.15.27:0/3549921074 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f495009b930 con 0x7f49601006a0 2024-04-25T01:24:49.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:49 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2536941916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:24:49.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:49 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1104784726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:24:49.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:49 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/607610237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:49.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.731+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49601a0870 con 0x7f49601006a0 2024-04-25T01:24:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.732+0000 7f4957fff640 1 -- 172.21.15.27:0/3549921074 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24792 (secure 0 0 0) 0x7f4950009de0 con 0x7f49601006a0 2024-04-25T01:24:49.713 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:49.713 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:24:47.477088+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f494c0a2b90 msgr2=0x7f494c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f494c0a2b90 0x7f494c0a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4960198f20 tx=0x7f4948005d30 comp rx=0 tx=0).stop 2024-04-25T01:24:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49601006a0 msgr2=0x7f4960197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49601006a0 0x7f4960197a60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f495000d770 tx=0x7f495000dc30 comp rx=0 tx=0).stop 2024-04-25T01:24:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 shutdown_connections 2024-04-25T01:24:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f494c0a2b90 0x7f494c0a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4960102250 0x7f496019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4960101890 0x7f4960197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 --2- 172.21.15.27:0/3549921074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49601006a0 0x7f4960197a60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 >> 172.21.15.27:0/3549921074 conn(0x7f49600fbc30 msgr2=0x7f49600fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 shutdown_connections 2024-04-25T01:24:49.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.734+0000 7f4965438640 1 -- 172.21.15.27:0/3549921074 wait complete. 2024-04-25T01:24:49.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:49 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2536941916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:24:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:49 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1104784726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:24:49.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:49 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/607610237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:49.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:49 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2536941916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:24:49.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:49 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1104784726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:24:49.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:49 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/607610237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 -- 172.21.15.27:0/3152997429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4107460 msgr2=0x7f54b41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/3152997429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4107460 0x7f54b41098b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009ca0 tx=0x7f54ac02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 -- 172.21.15.27:0/3152997429 shutdown_connections 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/3152997429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4107460 0x7f54b41098b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/3152997429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b4104b30 0x7f54b4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/3152997429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54b4102200 0x7f54b41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 -- 172.21.15.27:0/3152997429 >> 172.21.15.27:0/3152997429 conn(0x7f54b40fbbf0 msgr2=0x7f54b40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 -- 172.21.15.27:0/3152997429 shutdown_connections 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.849+0000 7f54ba1d5640 1 -- 172.21.15.27:0/3152997429 wait complete. 2024-04-25T01:24:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54ba1d5640 1 Processor -- start 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54ba1d5640 1 -- start start 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54ba1d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4102200 0x7f54b4197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54ba1d5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b4104b30 0x7f54b4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54ba1d5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54b4107460 0x7f54b419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54ba1d5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f54b410d420 con 0x7f54b4102200 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54ba1d5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f54b410d2a0 con 0x7f54b4104b30 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54ba1d5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54b410d5a0 con 0x7f54b4107460 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54b37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4102200 0x7f54b4197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54b37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4102200 0x7f54b4197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35242/0 (socket says 172.21.15.27:35242) 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54b37fe640 1 -- 172.21.15.27:0/573157434 learned_addr learned my addr 172.21.15.27:0/573157434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54b3fff640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54b4107460 0x7f54b419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.850+0000 7f54b2ffd640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b4104b30 0x7f54b4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b37fe640 1 -- 172.21.15.27:0/573157434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54b4107460 msgr2=0x7f54b419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b37fe640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54b4107460 0x7f54b419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b37fe640 1 -- 172.21.15.27:0/573157434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b4104b30 msgr2=0x7f54b4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b37fe640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b4104b30 0x7f54b4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b37fe640 1 -- 172.21.15.27:0/573157434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b419f690 con 0x7f54b4102200 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b2ffd640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b4104b30 0x7f54b4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b37fe640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4102200 0x7f54b4197a70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f54a400e9f0 tx=0x7f54a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b0ff9640 1 -- 172.21.15.27:0/573157434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a400cd80 con 0x7f54b4102200 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b0ff9640 1 -- 172.21.15.27:0/573157434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a4005500 con 0x7f54b4102200 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b419f980 con 0x7f54b4102200 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54b0ff9640 1 -- 172.21.15.27:0/573157434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4010690 con 0x7f54b4102200 2024-04-25T01:24:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.851+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b419fe80 con 0x7f54b4102200 2024-04-25T01:24:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.852+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b40fd6e0 con 0x7f54b4102200 2024-04-25T01:24:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.853+0000 7f54b0ff9640 1 -- 172.21.15.27:0/573157434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f54a40046c0 con 0x7f54b4102200 2024-04-25T01:24:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.854+0000 7f54b0ff9640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f549c0a2b90 0x7f549c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.854+0000 7f54b0ff9640 1 -- 172.21.15.27:0/573157434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f54a40cf8e0 con 0x7f54b4102200 2024-04-25T01:24:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.854+0000 7f54b2ffd640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f549c0a2b90 0x7f549c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.855+0000 7f54b2ffd640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f549c0a2b90 0x7f549c0a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f54b4198f80 tx=0x7f54a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:49.859+0000 7f54b0ff9640 1 -- 172.21.15.27:0/573157434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f54a409c430 con 0x7f54b4102200 2024-04-25T01:24:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.083+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54b402d170 con 0x7f54b4102200 2024-04-25T01:24:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.083+0000 7f54b0ff9640 1 -- 172.21.15.27:0/573157434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f54a409bd30 con 0x7f54b4102200 2024-04-25T01:24:50.064 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:24:50.064 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:23:23.940979+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:24:50.065 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-25T01:24:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f549c0a2b90 msgr2=0x7f549c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f549c0a2b90 0x7f549c0a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f54b4198f80 tx=0x7f54a8005f90 comp rx=0 tx=0).stop 2024-04-25T01:24:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4102200 msgr2=0x7f54b4197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4102200 0x7f54b4197a70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f54a400e9f0 tx=0x7f54a400edf0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 shutdown_connections 2024-04-25T01:24:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f549c0a2b90 0x7f549c0a5050 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54b4107460 0x7f54b419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54b4104b30 0x7f54b4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 --2- 172.21.15.27:0/573157434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54b4102200 0x7f54b4197a70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.085+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 >> 172.21.15.27:0/573157434 conn(0x7f54b40fbbf0 msgr2=0x7f54b4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.086+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 shutdown_connections 2024-04-25T01:24:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.086+0000 7f54ba1d5640 1 -- 172.21.15.27:0/573157434 wait complete. 2024-04-25T01:24:50.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:22:57.180706+0000', 'modified': '2024-04-25T01:23:23.940979+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24521, 'mds_1': 24517, 'mds_2': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24452': {'gid': 24452, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6835/418855472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 418855472}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 418855472}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/1981209233', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 1981209233}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 1981209233}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6837/1472553553', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 1472553553}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 1472553553}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.153:6837/529371680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 529371680}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 529371680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.153:6839/2846322026', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 2846322026}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 2846322026}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.139:6839/2167485871', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 2167485871}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 2167485871}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24521, 'qdb_cluster': [24505, 24517, 24521]} 2024-04-25T01:24:50.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T01:24:50.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 -- 172.21.15.27:0/3502662787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8102980 msgr2=0x7f5bf8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 --2- 172.21.15.27:0/3502662787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8102980 0x7f5bf8102e00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009900 tx=0x7f5bec02a410 comp rx=0 tx=0).stop 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 -- 172.21.15.27:0/3502662787 shutdown_connections 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 --2- 172.21.15.27:0/3502662787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5bf8103340 0x7f5bf8109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 --2- 172.21.15.27:0/3502662787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8102980 0x7f5bf8102e00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 --2- 172.21.15.27:0/3502662787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8101790 0x7f5bf8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 -- 172.21.15.27:0/3502662787 >> 172.21.15.27:0/3502662787 conn(0x7f5bf80fcd20 msgr2=0x7f5bf80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 -- 172.21.15.27:0/3502662787 shutdown_connections 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.201+0000 7f5bff57d640 1 -- 172.21.15.27:0/3502662787 wait complete. 2024-04-25T01:24:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bff57d640 1 Processor -- start 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bff57d640 1 -- start start 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bff57d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8101790 0x7f5bf8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bff57d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5bf8102980 0x7f5bf8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bff57d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8103340 0x7f5bf819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bff57d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf810d3f0 con 0x7f5bf8103340 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bff57d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf810d270 con 0x7f5bf8102980 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bff57d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf810d570 con 0x7f5bf8101790 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8103340 0x7f5bf819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8103340 0x7f5bf819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35268/0 (socket says 172.21.15.27:35268) 2024-04-25T01:24:50.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 -- 172.21.15.27:0/1432039734 learned_addr learned my addr 172.21.15.27:0/1432039734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5bf8102980 0x7f5bf8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfd319640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8101790 0x7f5bf8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 -- 172.21.15.27:0/1432039734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8101790 msgr2=0x7f5bf8197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8101790 0x7f5bf8197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 -- 172.21.15.27:0/1432039734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5bf8102980 msgr2=0x7f5bf8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5bf8102980 0x7f5bf8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 -- 172.21.15.27:0/1432039734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf819f6b0 con 0x7f5bf8103340 2024-04-25T01:24:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfd319640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8101790 0x7f5bf8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.202+0000 7f5bfdb1a640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8103340 0x7f5bf819f040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009fd0 tx=0x7f5be800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.203+0000 7f5bf67fc640 1 -- 172.21.15.27:0/1432039734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be8017070 con 0x7f5bf8103340 2024-04-25T01:24:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.203+0000 7f5bf67fc640 1 -- 172.21.15.27:0/1432039734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be8003e50 con 0x7f5bf8103340 2024-04-25T01:24:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.203+0000 7f5bf67fc640 1 -- 172.21.15.27:0/1432039734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be80106c0 con 0x7f5bf8103340 2024-04-25T01:24:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.203+0000 7f5bff57d640 1 -- 172.21.15.27:0/1432039734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf819f9a0 con 0x7f5bf8103340 2024-04-25T01:24:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.203+0000 7f5bff57d640 1 -- 172.21.15.27:0/1432039734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf819fe70 con 0x7f5bf8103340 2024-04-25T01:24:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5bf67fc640 1 -- 172.21.15.27:0/1432039734 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5be8003970 con 0x7f5bf8103340 2024-04-25T01:24:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5c02285740 1 Processor -- start 2024-04-25T01:24:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5c02285740 1 -- start start 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5c02285740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 0x555c4b781860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5c02285740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b786930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5c02285740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b789430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5c02285740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555c4b6e4f00 con 0x555c4b786fa0 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5c02285740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555c4b68d750 con 0x555c4b781440 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5c02285740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555c4b6e0ec0 con 0x555c4b781e30 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5bf67fc640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5be00a2b90 0x7f5be00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.205+0000 7f5bfcb18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b789430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b789430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35276/0 (socket says 172.21.15.27:35276) 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 -- 172.21.15.27:0/303469408 learned_addr learned my addr 172.21.15.27:0/303469408 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfd319640 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b786930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfdb1a640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5be00a2b90 0x7f5be00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 0x555c4b781860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bf67fc640 1 -- 172.21.15.27:0/1432039734 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f5be80cf1b0 con 0x7f5bf8103340 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 -- 172.21.15.27:0/303469408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 msgr2=0x555c4b786930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b786930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 -- 172.21.15.27:0/303469408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 msgr2=0x555c4b781860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 0x555c4b781860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 -- 172.21.15.27:0/303469408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c4b3f92d0 con 0x555c4b786fa0 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b789430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5bec000c00 tx=0x7f5bec005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa30a62537439bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bfdb1a640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5be00a2b90 0x7f5be00a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5be809b5f0 tx=0x7f5be8066780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bdf7fe640 1 -- 172.21.15.27:0/303469408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec035040 con 0x555c4b786fa0 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bdf7fe640 1 -- 172.21.15.27:0/303469408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bec045070 con 0x555c4b786fa0 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.206+0000 7f5bdf7fe640 1 -- 172.21.15.27:0/303469408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec004980 con 0x555c4b786fa0 2024-04-25T01:24:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 -- 172.21.15.27:0/303469408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 msgr2=0x555c4b789430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b789430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5bec000c00 tx=0x7f5bec005e10 comp rx=0 tx=0).stop 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 -- 172.21.15.27:0/303469408 shutdown_connections 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b789430 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b786930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 --2- 172.21.15.27:0/303469408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 0x555c4b781860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 -- 172.21.15.27:0/303469408 >> 172.21.15.27:0/303469408 conn(0x555c4b77cab0 msgr2=0x555c4b77ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 -- 172.21.15.27:0/303469408 shutdown_connections 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 -- 172.21.15.27:0/303469408 wait complete. 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 -- start start 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.207+0000 7f5c02285740 1 Processor -- start 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5c02285740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781440 0x555c4b78b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5c02285740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781e30 0x555c4b78bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5c02285740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 0x555c4b792d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5c02285740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555c4b789ef0 con 0x555c4b781e30 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5c02285740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555c4b6a0cf0 con 0x555c4b786fa0 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5c02285740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555c4b78a070 con 0x555c4b781440 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5c02285740 10 client.0 _ll_register_callbacks cb 0x555c4b6a06e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfd319640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781e30 0x555c4b78bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfd319640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781e30 0x555c4b78bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35284/0 (socket says 172.21.15.27:35284) 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfd319640 1 -- 172.21.15.27:0/2640763837 learned_addr learned my addr 172.21.15.27:0/2640763837 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781440 0x555c4b78b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 0x555c4b792d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfd319640 1 -- 172.21.15.27:0/2640763837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781440 msgr2=0x555c4b78b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfd319640 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781440 0x555c4b78b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfd319640 1 -- 172.21.15.27:0/2640763837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 msgr2=0x555c4b792d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfd319640 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 0x555c4b792d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.208+0000 7f5bfd319640 1 -- 172.21.15.27:0/2640763837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c4b4c36b0 con 0x555c4b781e30 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5bfd319640 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781e30 0x555c4b78bd50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00107f0 tx=0x7f5bf0010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0136e00b6f814c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5bdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5bdcff9640 1 -- 172.21.15.27:0/2640763837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0004440 con 0x555c4b781e30 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5bdcff9640 1 -- 172.21.15.27:0/2640763837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f5bf000e8b0 con 0x555c4b781e30 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 1 -- 172.21.15.27:0/2640763837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bec045070 con 0x555c4b781e30 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5bdcff9640 1 -- 172.21.15.27:0/2640763837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5bf000eab0 con 0x555c4b781e30 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 10 client.15240 fetch_fsmap learned FSMap version 24 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 1 -- 172.21.15.27:0/2640763837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555c4b795ec0 con 0x555c4b781e30 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5bdcff9640 1 -- 172.21.15.27:0/2640763837 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bec045070 con 0x555c4b781e30 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5bdcff9640 1 -- 172.21.15.27:0/2640763837 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5bf0017070 con 0x555c4b781e30 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 10 client.15240 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:24:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 10 client.15240 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 20 client.15240 populate_metadata read hostname 'smithi027' 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x555c4b7985f0 0x555c4b799a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 4 client.15240 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.209+0000 7f5c02285740 1 -- 172.21.15.27:0/2640763837 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c4b6f2440 con 0x555c4b7985f0 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.210+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x555c4b7985f0 0x555c4b799a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.210+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x555c4b7985f0 0x555c4b799a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3c8474bd463fa1b server_cookie=9da8e995825d72fa in_seq=0 out_seq=0 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.210+0000 7f5bdcff9640 10 client.15240 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5bdcff9640 1 -- 172.21.15.27:0/2640763837 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c4b6f2440 con 0x555c4b7985f0 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5bdcff9640 10 client.15240 handle_command_reply: tid=0 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 client.15240 shutdown 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 -- 172.21.15.27:0/2640763837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781e30 msgr2=0x555c4b78bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781e30 0x555c4b78bd50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00107f0 tx=0x7f5bf0010cb0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 -- 172.21.15.27:0/2640763837 shutdown_connections 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x555c4b7985f0 0x555c4b799a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 0x555c4b792d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781e30 0x555c4b78bd50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5bdcff9640 0 client.15240 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 --2- 172.21.15.27:0/2640763837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781440 0x555c4b78b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 -- 172.21.15.27:0/2640763837 >> 172.21.15.27:0/2640763837 conn(0x555c4b77cab0 msgr2=0x555c4b77bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 -- 172.21.15.27:0/2640763837 shutdown_connections 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 1 -- 172.21.15.27:0/2640763837 wait complete. 2024-04-25T01:24:50.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.211+0000 7f5c02285740 20 client.15240 trim_cache size 0 max 16384 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.229+0000 7f5c02285740 1 Processor -- start 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.229+0000 7f5c02285740 1 -- start start 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5c02285740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 0x555c4b782850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5c02285740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b792d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5c02285740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b777690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5c02285740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555c4b797da0 con 0x555c4b786fa0 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5c02285740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555c4b798470 con 0x555c4b781440 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5c02285740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555c4b798b40 con 0x555c4b781e30 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b777690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b777690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35296/0 (socket says 172.21.15.27:35296) 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 -- 172.21.15.27:0/3405304902 learned_addr learned my addr 172.21.15.27:0/3405304902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 0x555c4b782850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfd319640 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b792d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 -- 172.21.15.27:0/3405304902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 msgr2=0x555c4b792d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b792d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 -- 172.21.15.27:0/3405304902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 msgr2=0x555c4b782850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 0x555c4b782850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 -- 172.21.15.27:0/3405304902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c4b77afc0 con 0x555c4b786fa0 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b777690 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5bec031610 tx=0x7f5bec040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74fc15c2d02b167 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bdeffd640 1 -- 172.21.15.27:0/3405304902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec035040 con 0x555c4b786fa0 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bdeffd640 1 -- 172.21.15.27:0/3405304902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bec045070 con 0x555c4b786fa0 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.230+0000 7f5bdeffd640 1 -- 172.21.15.27:0/3405304902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec056520 con 0x555c4b786fa0 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 -- 172.21.15.27:0/3405304902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 msgr2=0x555c4b777690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b777690 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5bec031610 tx=0x7f5bec040040 comp rx=0 tx=0).stop 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 -- 172.21.15.27:0/3405304902 shutdown_connections 2024-04-25T01:24:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b786fa0 0x555c4b777690 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b792d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 --2- 172.21.15.27:0/3405304902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b781440 0x555c4b782850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 -- 172.21.15.27:0/3405304902 >> 172.21.15.27:0/3405304902 conn(0x555c4b77cab0 msgr2=0x555c4b77ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 -- 172.21.15.27:0/3405304902 shutdown_connections 2024-04-25T01:24:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 -- 172.21.15.27:0/3405304902 wait complete. 2024-04-25T01:24:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 -- start start 2024-04-25T01:24:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.231+0000 7f5c02285740 1 Processor -- start 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5c02285740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781440 0x555c4b78de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5c02285740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b78e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5c02285740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 0x555c4b791360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5c02285740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555c4b7998e0 con 0x555c4b781440 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5c02285740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555c4b799210 con 0x555c4b786fa0 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5c02285740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555c4b799fb0 con 0x555c4b781e30 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5c02285740 10 client.0 _ll_register_callbacks cb 0x7f5bf0010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781440 0x555c4b78de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781440 0x555c4b78de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35298/0 (socket says 172.21.15.27:35298) 2024-04-25T01:24:50.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 -- 172.21.15.27:0/3490981121 learned_addr learned my addr 172.21.15.27:0/3490981121 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 0x555c4b791360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfd319640 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b78e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 -- 172.21.15.27:0/3490981121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 msgr2=0x555c4b78e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b78e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 -- 172.21.15.27:0/3490981121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 msgr2=0x555c4b791360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 0x555c4b791360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 -- 172.21.15.27:0/3490981121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c4b791950 con 0x555c4b781440 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bfcb18640 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781440 0x555c4b78de70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5bec02a930 tx=0x7f5bec02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddd3b013725e7c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bdcff9640 1 -- 172.21.15.27:0/3490981121 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec035040 con 0x555c4b781440 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5bdcff9640 1 -- 172.21.15.27:0/3490981121 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f5bec056bd0 con 0x555c4b781440 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.232+0000 7f5c02285740 1 -- 172.21.15.27:0/3490981121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bec045070 con 0x555c4b781440 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5bdcff9640 1 -- 172.21.15.27:0/3490981121 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5bec056dd0 con 0x555c4b781440 2024-04-25T01:24:50.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5c02285740 10 client.15246 fetch_fsmap learned FSMap version 24 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5c02285740 1 -- 172.21.15.27:0/3490981121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555c4b792c50 con 0x555c4b781440 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5bdcff9640 1 -- 172.21.15.27:0/3490981121 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bec045070 con 0x555c4b781440 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5bdcff9640 1 -- 172.21.15.27:0/3490981121 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5bec04c420 con 0x555c4b781440 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5c02285740 10 client.15246 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5c02285740 10 client.15246 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5c02285740 20 client.15246 populate_metadata read hostname 'smithi027' 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5c02285740 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x555c4b802500 0x555c4b801210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5c02285740 4 client.15246 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5c02285740 1 -- 172.21.15.27:0/3490981121 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "status"}) -- 0x555c4b6ec790 con 0x555c4b802500 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.233+0000 7f5bfd319640 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x555c4b802500 0x555c4b801210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5bdcff9640 10 client.15246 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5bfd319640 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x555c4b802500 0x555c4b801210 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99f369c18f7cffe4 server_cookie=b4a0ba4375c9287b in_seq=0 out_seq=0 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5bdcff9640 1 -- 172.21.15.27:0/3490981121 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555c4b6ec790 con 0x555c4b802500 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5bdcff9640 10 client.15246 handle_command_reply: tid=0 2024-04-25T01:24:50.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 client.15246 shutdown 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 -- 172.21.15.27:0/3490981121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781440 msgr2=0x555c4b78de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781440 0x555c4b78de70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5bec02a930 tx=0x7f5bec02a410 comp rx=0 tx=0).stop 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 -- 172.21.15.27:0/3490981121 shutdown_connections 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x555c4b802500 0x555c4b801210 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555c4b786fa0 0x555c4b791360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c4b781e30 0x555c4b78e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5bdcff9640 0 client.15246 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 --2- 172.21.15.27:0/3490981121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555c4b781440 0x555c4b78de70 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 -- 172.21.15.27:0/3490981121 >> 172.21.15.27:0/3490981121 conn(0x555c4b77cab0 msgr2=0x555c4b777690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 -- 172.21.15.27:0/3490981121 shutdown_connections 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 1 -- 172.21.15.27:0/3490981121 wait complete. 2024-04-25T01:24:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.234+0000 7f5c02285740 20 client.15246 trim_cache size 0 max 16384 2024-04-25T01:24:50.222 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:24:50.222 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:24:50.222 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 2, 2024-04-25T01:24:50.222 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24505, 2024-04-25T01:24:50.222 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T01:24:50.222 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T01:24:50.223 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T01:24:50.223 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 87.302336038999997, 2024-04-25T01:24:50.223 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 24, 2024-04-25T01:24:50.223 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T01:24:50.223 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T01:24:50.223 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 132.58589148600001 2024-04-25T01:24:50.223 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:24:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 -- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5be00a2b90 msgr2=0x7f5be00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5be00a2b90 0x7f5be00a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5be809b5f0 tx=0x7f5be8066780 comp rx=0 tx=0).stop 2024-04-25T01:24:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 -- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8103340 msgr2=0x7f5bf819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8103340 0x7f5bf819f040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009fd0 tx=0x7f5be800b260 comp rx=0 tx=0).stop 2024-04-25T01:24:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 -- 172.21.15.27:0/1432039734 shutdown_connections 2024-04-25T01:24:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f5be00a2b90 0x7f5be00a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf8103340 0x7f5bf819f040 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5bf8102980 0x7f5bf8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 --2- 172.21.15.27:0/1432039734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bf8101790 0x7f5bf8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 -- 172.21.15.27:0/1432039734 >> 172.21.15.27:0/1432039734 conn(0x7f5bf80fcd20 msgr2=0x7f5bf8105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 -- 172.21.15.27:0/1432039734 shutdown_connections 2024-04-25T01:24:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.235+0000 7f5bff57d640 1 -- 172.21.15.27:0/1432039734 wait complete. 2024-04-25T01:24:50.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:24:50.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 -- 172.21.15.27:0/4106436823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc968107450 msgr2=0x7fc9681098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/4106436823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc968107450 0x7fc9681098a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc960009ca0 tx=0x7fc96002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 -- 172.21.15.27:0/4106436823 shutdown_connections 2024-04-25T01:24:50.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/4106436823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc968107450 0x7fc9681098a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/4106436823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc968104b20 0x7fc968106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/4106436823 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9681021f0 0x7fc9681045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 -- 172.21.15.27:0/4106436823 >> 172.21.15.27:0/4106436823 conn(0x7fc9680fbbe0 msgr2=0x7fc9680fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 -- 172.21.15.27:0/4106436823 shutdown_connections 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.350+0000 7fc96dcf5640 1 -- 172.21.15.27:0/4106436823 wait complete. 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc96dcf5640 1 Processor -- start 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc96dcf5640 1 -- start start 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc96dcf5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9681021f0 0x7fc968197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc96dcf5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc968104b20 0x7fc968197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc96dcf5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc968107450 0x7fc96819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc96dcf5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc96810d410 con 0x7fc9681021f0 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc96dcf5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc96810d290 con 0x7fc968107450 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc96dcf5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc96810d590 con 0x7fc968104b20 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9681021f0 0x7fc968197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9681021f0 0x7fc968197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35318/0 (socket says 172.21.15.27:35318) 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc9677fe640 1 -- 172.21.15.27:0/1442295245 learned_addr learned my addr 172.21.15.27:0/1442295245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.351+0000 7fc967fff640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc968107450 0x7fc96819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc9677fe640 1 -- 172.21.15.27:0/1442295245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc968104b20 msgr2=0x7fc968197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc966ffd640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc968104b20 0x7fc968197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc968104b20 0x7fc968197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc9677fe640 1 -- 172.21.15.27:0/1442295245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc968107450 msgr2=0x7fc96819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc968107450 0x7fc96819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc9677fe640 1 -- 172.21.15.27:0/1442295245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96819f6b0 con 0x7fc9681021f0 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc967fff640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc968107450 0x7fc96819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc966ffd640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc968104b20 0x7fc968197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9681021f0 0x7fc968197a90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00d950 tx=0x7fc95c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc964ff9640 1 -- 172.21.15.27:0/1442295245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c017070 con 0x7fc9681021f0 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc96dcf5640 1 -- 172.21.15.27:0/1442295245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96819f9a0 con 0x7fc9681021f0 2024-04-25T01:24:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc964ff9640 1 -- 172.21.15.27:0/1442295245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c009c30 con 0x7fc9681021f0 2024-04-25T01:24:50.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc964ff9640 1 -- 172.21.15.27:0/1442295245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c00f6b0 con 0x7fc9681021f0 2024-04-25T01:24:50.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.352+0000 7fc96dcf5640 1 -- 172.21.15.27:0/1442295245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc96819fef0 con 0x7fc9681021f0 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.354+0000 7fc964ff9640 1 -- 172.21.15.27:0/1442295245 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc95c0109d0 con 0x7fc9681021f0 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.354+0000 7fc970a02740 1 Processor -- start 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.354+0000 7fc970a02740 1 -- start start 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc970a02740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190440 0x55eb42190860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc970a02740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190e30 0x55eb42195930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc970a02740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb42198430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc970a02740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eb420f3f00 con 0x55eb42190e30 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc970a02740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eb4209c750 con 0x55eb42190440 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc970a02740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eb420efec0 con 0x55eb42195fa0 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190e30 0x55eb42195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190e30 0x55eb42195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35324/0 (socket says 172.21.15.27:35324) 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc9677fe640 1 -- 172.21.15.27:0/4186864815 learned_addr learned my addr 172.21.15.27:0/4186864815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc966ffd640 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb42198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc964ff9640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc9500a2b90 0x7fc9500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc967fff640 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190440 0x55eb42190860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc967fff640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc9500a2b90 0x7fc9500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc964ff9640 1 -- 172.21.15.27:0/1442295245 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc95c0cf130 con 0x7fc9681021f0 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc966ffd640 1 -- 172.21.15.27:0/4186864815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190440 msgr2=0x55eb42190860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc966ffd640 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190440 0x55eb42190860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc966ffd640 1 -- 172.21.15.27:0/4186864815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190e30 msgr2=0x55eb42195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc966ffd640 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190e30 0x55eb42195930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc966ffd640 1 -- 172.21.15.27:0/4186864815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb41e082d0 con 0x55eb42195fa0 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc9677fe640 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190e30 0x55eb42195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc966ffd640 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb42198430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc96806a8a0 tx=0x7fc958010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e343b739585fc94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.355+0000 7fc967fff640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc9500a2b90 0x7fc9500a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55eb42191690 tx=0x7fc9600095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc94dffb640 1 -- 172.21.15.27:0/4186864815 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958004030 con 0x55eb42195fa0 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc94dffb640 1 -- 172.21.15.27:0/4186864815 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9580041d0 con 0x55eb42195fa0 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc94dffb640 1 -- 172.21.15.27:0/4186864815 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9580126b0 con 0x55eb42195fa0 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 -- 172.21.15.27:0/4186864815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 msgr2=0x55eb42198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb42198430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc96806a8a0 tx=0x7fc958010740 comp rx=0 tx=0).stop 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 -- 172.21.15.27:0/4186864815 shutdown_connections 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb42198430 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190e30 0x55eb42195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 --2- 172.21.15.27:0/4186864815 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190440 0x55eb42190860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 -- 172.21.15.27:0/4186864815 >> 172.21.15.27:0/4186864815 conn(0x55eb4218bab0 msgr2=0x55eb4218beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 -- 172.21.15.27:0/4186864815 shutdown_connections 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.356+0000 7fc970a02740 1 -- 172.21.15.27:0/4186864815 wait complete. 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 1 -- start start 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 1 Processor -- start 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eb42198f50 con 0x55eb42190440 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eb420afcf0 con 0x55eb42190e30 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eb421990d0 con 0x55eb42195fa0 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc970a02740 10 client.0 _ll_register_callbacks cb 0x55eb420af6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35328/0 (socket says 172.21.15.27:35328) 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc9677fe640 1 -- 172.21.15.27:0/506561189 learned_addr learned my addr 172.21.15.27:0/506561189 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc966ffd640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc967fff640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc9677fe640 1 -- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 msgr2=0x55eb421a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc9677fe640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc9677fe640 1 -- 172.21.15.27:0/506561189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 msgr2=0x55eb4219ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc9677fe640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc9677fe640 1 -- 172.21.15.27:0/506561189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb41ed26b0 con 0x55eb42190440 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc967fff640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.357+0000 7fc966ffd640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc9677fe640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219a840 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0a3d20 tx=0x7fc95c0979a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4f7190cc14c70d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc946ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc946ffd640 1 -- 172.21.15.27:0/506561189 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c020070 con 0x55eb42190440 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc946ffd640 1 -- 172.21.15.27:0/506561189 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc95c097c50 con 0x55eb42190440 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 1 -- 172.21.15.27:0/506561189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9580041d0 con 0x55eb42190440 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc946ffd640 1 -- 172.21.15.27:0/506561189 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc95c09af40 con 0x55eb42190440 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 10 client.15258 fetch_fsmap learned FSMap version 24 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 1 -- 172.21.15.27:0/506561189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55eb421a4df0 con 0x55eb42190440 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc946ffd640 1 -- 172.21.15.27:0/506561189 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9580041d0 con 0x55eb42190440 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc946ffd640 1 -- 172.21.15.27:0/506561189 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc95c084b70 con 0x55eb42190440 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 10 client.15258 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:24:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 10 client.15258 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 20 client.15258 populate_metadata read hostname 'smithi027' 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55eb421a7410 0x55eb421a8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 4 client.15258 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.358+0000 7fc970a02740 1 -- 172.21.15.27:0/506561189 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb42101440 con 0x55eb421a7410 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.359+0000 7fc966ffd640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55eb421a7410 0x55eb421a8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.359+0000 7fc966ffd640 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55eb421a7410 0x55eb421a8880 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5569a566c02ee2fc server_cookie=7bfe50c77adb3799 in_seq=0 out_seq=0 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.359+0000 7fc946ffd640 10 client.15258 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc946ffd640 1 -- 172.21.15.27:0/506561189 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb42101440 con 0x55eb421a7410 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc946ffd640 10 client.15258 handle_command_reply: tid=0 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 client.15258 shutdown 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 -- 172.21.15.27:0/506561189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 msgr2=0x55eb4219a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219a840 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0a3d20 tx=0x7fc95c0979a0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 -- 172.21.15.27:0/506561189 shutdown_connections 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55eb421a7410 0x55eb421a8880 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc946ffd640 0 client.15258 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 --2- 172.21.15.27:0/506561189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219a840 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 -- 172.21.15.27:0/506561189 >> 172.21.15.27:0/506561189 conn(0x55eb4218bab0 msgr2=0x55eb4218aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 -- 172.21.15.27:0/506561189 shutdown_connections 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 1 -- 172.21.15.27:0/506561189 wait complete. 2024-04-25T01:24:50.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.360+0000 7fc970a02740 20 client.15258 trim_cache size 0 max 16384 2024-04-25T01:24:50.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.378+0000 7fc970a02740 1 Processor -- start 2024-04-25T01:24:50.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.378+0000 7fc970a02740 1 -- start start 2024-04-25T01:24:50.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc970a02740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc970a02740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42190e30 0x55eb421a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc970a02740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42195fa0 0x55eb42199be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc970a02740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eb421a6bc0 con 0x55eb42190440 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc970a02740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eb421a7290 con 0x55eb42195fa0 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc970a02740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eb421a7960 con 0x55eb42190e30 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35340/0 (socket says 172.21.15.27:35340) 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 -- 172.21.15.27:0/1343028173 learned_addr learned my addr 172.21.15.27:0/1343028173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc967fff640 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42195fa0 0x55eb42199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 -- 172.21.15.27:0/1343028173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42190e30 msgr2=0x55eb421a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42190e30 0x55eb421a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 -- 172.21.15.27:0/1343028173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42195fa0 msgr2=0x55eb42199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42195fa0 0x55eb42199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 -- 172.21.15.27:0/1343028173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb42191210 con 0x55eb42190440 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc967fff640 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42195fa0 0x55eb42199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219ace0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc95c099890 tx=0x7fc95c0d2b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2f9bfdd897c79d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc94d7fa640 1 -- 172.21.15.27:0/1343028173 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c0b4070 con 0x55eb42190440 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc94d7fa640 1 -- 172.21.15.27:0/1343028173 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c09bce0 con 0x55eb42190440 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.379+0000 7fc94d7fa640 1 -- 172.21.15.27:0/1343028173 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c020070 con 0x55eb42190440 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 -- 172.21.15.27:0/1343028173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 msgr2=0x55eb4219ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219ace0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc95c099890 tx=0x7fc95c0d2b90 comp rx=0 tx=0).stop 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 -- 172.21.15.27:0/1343028173 shutdown_connections 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42195fa0 0x55eb42199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42190e30 0x55eb421a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 --2- 172.21.15.27:0/1343028173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219ace0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 -- 172.21.15.27:0/1343028173 >> 172.21.15.27:0/1343028173 conn(0x55eb4218bab0 msgr2=0x55eb421994d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 -- 172.21.15.27:0/1343028173 shutdown_connections 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 -- 172.21.15.27:0/1343028173 wait complete. 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 -- start start 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.380+0000 7fc970a02740 1 Processor -- start 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eb421a8700 con 0x55eb42190440 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eb421a8030 con 0x55eb42190e30 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eb421a8dd0 con 0x55eb42195fa0 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 10 client.0 _ll_register_callbacks cb 0x7fc95c00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35344/0 (socket says 172.21.15.27:35344) 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 -- 172.21.15.27:0/1714115694 learned_addr learned my addr 172.21.15.27:0/1714115694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc966ffd640 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc967fff640 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 -- 172.21.15.27:0/1714115694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 msgr2=0x55eb421a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 -- 172.21.15.27:0/1714115694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 msgr2=0x55eb4219e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 -- 172.21.15.27:0/1714115694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb421a1600 con 0x55eb42190440 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc9677fe640 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219df40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc95c010fd0 tx=0x7fc95c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58574ed836cc0a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc946ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc946ffd640 1 -- 172.21.15.27:0/1714115694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c020070 con 0x55eb42190440 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc946ffd640 1 -- 172.21.15.27:0/1714115694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc95c09ab40 con 0x55eb42190440 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 1 -- 172.21.15.27:0/1714115694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc95c09bce0 con 0x55eb42190440 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc946ffd640 1 -- 172.21.15.27:0/1714115694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc95c0b4070 con 0x55eb42190440 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 10 client.15264 fetch_fsmap learned FSMap version 24 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.381+0000 7fc970a02740 1 -- 172.21.15.27:0/1714115694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55eb4220d990 con 0x55eb42190440 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc946ffd640 1 -- 172.21.15.27:0/1714115694 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c09bce0 con 0x55eb42190440 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc946ffd640 1 -- 172.21.15.27:0/1714115694 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc95c097330 con 0x55eb42190440 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc970a02740 10 client.15264 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:24:50.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc970a02740 10 client.15264 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc970a02740 20 client.15264 populate_metadata read hostname 'smithi027' 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc970a02740 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55eb42212250 0x55eb42210f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc970a02740 4 client.15264 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc970a02740 1 -- 172.21.15.27:0/1714115694 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "status"}) -- 0x55eb420fb790 con 0x55eb42212250 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.382+0000 7fc966ffd640 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55eb42212250 0x55eb42210f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc946ffd640 10 client.15264 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc966ffd640 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55eb42212250 0x55eb42210f60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac8b9ff8533b2583 server_cookie=56f0dcd6e2a33aed in_seq=0 out_seq=0 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc946ffd640 1 -- 172.21.15.27:0/1714115694 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55eb420fb790 con 0x55eb42212250 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc946ffd640 10 client.15264 handle_command_reply: tid=0 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 client.15264 shutdown 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 -- 172.21.15.27:0/1714115694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 msgr2=0x55eb4219df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219df40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc95c010fd0 tx=0x7fc95c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 -- 172.21.15.27:0/1714115694 shutdown_connections 2024-04-25T01:24:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55eb42212250 0x55eb42210f60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eb42195fa0 0x55eb421a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc946ffd640 0 client.15264 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eb42190e30 0x55eb4219e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 --2- 172.21.15.27:0/1714115694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eb42190440 0x55eb4219df40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 -- 172.21.15.27:0/1714115694 >> 172.21.15.27:0/1714115694 conn(0x55eb4218bab0 msgr2=0x55eb4218a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 -- 172.21.15.27:0/1714115694 shutdown_connections 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 1 -- 172.21.15.27:0/1714115694 wait complete. 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.383+0000 7fc970a02740 20 client.15264 trim_cache size 0 max 16384 2024-04-25T01:24:50.372 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 1, 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24517, 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 88.456093971000001, 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 24, 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T01:24:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T01:24:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 127.11395770199999 2024-04-25T01:24:50.374 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:24:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 -- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc9500a2b90 msgr2=0x7fc9500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc9500a2b90 0x7fc9500a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55eb42191690 tx=0x7fc9600095c0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 -- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9681021f0 msgr2=0x7fc968197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9681021f0 0x7fc968197a90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00d950 tx=0x7fc95c00de10 comp rx=0 tx=0).stop 2024-04-25T01:24:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 -- 172.21.15.27:0/1442295245 shutdown_connections 2024-04-25T01:24:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc9500a2b90 0x7fc9500a5050 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc968107450 0x7fc96819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc968104b20 0x7fc968197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 --2- 172.21.15.27:0/1442295245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9681021f0 0x7fc968197a90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 -- 172.21.15.27:0/1442295245 >> 172.21.15.27:0/1442295245 conn(0x7fc9680fbbe0 msgr2=0x7fc968102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 -- 172.21.15.27:0/1442295245 shutdown_connections 2024-04-25T01:24:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.384+0000 7fc96dcf5640 1 -- 172.21.15.27:0/1442295245 wait complete. 2024-04-25T01:24:50.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:24:50.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 -- 172.21.15.27:0/1980302833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 msgr2=0x7fbb6c109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 --2- 172.21.15.27:0/1980302833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 0x7fbb6c109910 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009ca0 tx=0x7fbb6402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 -- 172.21.15.27:0/1980302833 shutdown_connections 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 --2- 172.21.15.27:0/1980302833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 0x7fbb6c109910 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 --2- 172.21.15.27:0/1980302833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb6c104bf0 0x7fbb6c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 --2- 172.21.15.27:0/1980302833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbb6c1022c0 0x7fbb6c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 -- 172.21.15.27:0/1980302833 >> 172.21.15.27:0/1980302833 conn(0x7fbb6c0fbbf0 msgr2=0x7fbb6c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 -- 172.21.15.27:0/1980302833 shutdown_connections 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 -- 172.21.15.27:0/1980302833 wait complete. 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.500+0000 7fbb7154f640 1 Processor -- start 2024-04-25T01:24:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb7154f640 1 -- start start 2024-04-25T01:24:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb7154f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbb6c1022c0 0x7fbb6c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb7154f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb6c104bf0 0x7fbb6c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb7154f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 0x7fbb6c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb7154f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c10d430 con 0x7fbb6c107520 2024-04-25T01:24:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb7154f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c10d2b0 con 0x7fbb6c1022c0 2024-04-25T01:24:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb7154f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c10d5b0 con 0x7fbb6c104bf0 2024-04-25T01:24:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 0x7fbb6c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 0x7fbb6c19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35362/0 (socket says 172.21.15.27:35362) 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbb6c1022c0 0x7fbb6c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6b7fe640 1 -- 172.21.15.27:0/2946893418 learned_addr learned my addr 172.21.15.27:0/2946893418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb6c104bf0 0x7fbb6c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6b7fe640 1 -- 172.21.15.27:0/2946893418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb6c104bf0 msgr2=0x7fbb6c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6b7fe640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb6c104bf0 0x7fbb6c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6b7fe640 1 -- 172.21.15.27:0/2946893418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbb6c1022c0 msgr2=0x7fbb6c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6b7fe640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbb6c1022c0 0x7fbb6c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.501+0000 7fbb6b7fe640 1 -- 172.21.15.27:0/2946893418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6c19f6f0 con 0x7fbb6c107520 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.502+0000 7fbb6b7fe640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 0x7fbb6c19f080 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbb64031ae0 tx=0x7fbb64031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.502+0000 7fbb53fff640 1 -- 172.21.15.27:0/2946893418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64003b70 con 0x7fbb6c107520 2024-04-25T01:24:50.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.502+0000 7fbb53fff640 1 -- 172.21.15.27:0/2946893418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb64003d10 con 0x7fbb6c107520 2024-04-25T01:24:50.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.502+0000 7fbb7154f640 1 -- 172.21.15.27:0/2946893418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6c19f980 con 0x7fbb6c107520 2024-04-25T01:24:50.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.502+0000 7fbb53fff640 1 -- 172.21.15.27:0/2946893418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6402fc40 con 0x7fbb6c107520 2024-04-25T01:24:50.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.502+0000 7fbb7154f640 1 -- 172.21.15.27:0/2946893418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb6c19fed0 con 0x7fbb6c107520 2024-04-25T01:24:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb53fff640 1 -- 172.21.15.27:0/2946893418 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbb64004050 con 0x7fbb6c107520 2024-04-25T01:24:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb74256740 1 Processor -- start 2024-04-25T01:24:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb74256740 1 -- start start 2024-04-25T01:24:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb74256740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb74256740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ddd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb74256740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 0x55a4f17e0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb53fff640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbb540a2b90 0x7fbb540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb74256740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a4f1737f00 con 0x55a4f17d9240 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb74256740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a4f16e0750 con 0x55a4f17de3b0 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb74256740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a4f1733ec0 con 0x55a4f17d8850 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb6affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ddd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb6affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ddd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35374/0 (socket says 172.21.15.27:35374) 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb6affd640 1 -- 172.21.15.27:0/1899571360 learned_addr learned my addr 172.21.15.27:0/1899571360 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbb540a2b90 0x7fbb540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb53fff640 1 -- 172.21.15.27:0/2946893418 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fbb640f0640 con 0x7fbb6c107520 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.505+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6b7fe640 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 0x55a4f17e0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6affd640 1 -- 172.21.15.27:0/1899571360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 msgr2=0x55a4f17d8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6affd640 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17d8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6affd640 1 -- 172.21.15.27:0/1899571360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 msgr2=0x55a4f17e0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6affd640 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 0x55a4f17e0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6affd640 1 -- 172.21.15.27:0/1899571360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4f144c2d0 con 0x55a4f17d9240 2024-04-25T01:24:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17d8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6affd640 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ddd40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009cf0 tx=0x7fbb5c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b061be42809fc8d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbb540a2b90 0x7fbb540a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55a4f17d9aa0 tx=0x7fbb60009910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb517fa640 1 -- 172.21.15.27:0/1899571360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c004230 con 0x55a4f17d9240 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb517fa640 1 -- 172.21.15.27:0/1899571360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb5c0043d0 con 0x55a4f17d9240 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.506+0000 7fbb517fa640 1 -- 172.21.15.27:0/1899571360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0198c0 con 0x55a4f17d9240 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- 172.21.15.27:0/1899571360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 msgr2=0x55a4f17ddd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ddd40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009cf0 tx=0x7fbb5c011810 comp rx=0 tx=0).stop 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- 172.21.15.27:0/1899571360 shutdown_connections 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 0x55a4f17e0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ddd40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 --2- 172.21.15.27:0/1899571360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17d8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- 172.21.15.27:0/1899571360 >> 172.21.15.27:0/1899571360 conn(0x55a4f17cfab0 msgr2=0x55a4f17cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- 172.21.15.27:0/1899571360 shutdown_connections 2024-04-25T01:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- 172.21.15.27:0/1899571360 wait complete. 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- start start 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 Processor -- start 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17e2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 0x55a4f17ea1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a4f17e1360 con 0x55a4f17d9240 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a4f16f3cf0 con 0x55a4f17de3b0 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a4f17e14e0 con 0x55a4f17d8850 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb74256740 10 client.0 _ll_register_callbacks cb 0x55a4f16f36e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb6a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb6a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35380/0 (socket says 172.21.15.27:35380) 2024-04-25T01:24:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.507+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/367601145 learned_addr learned my addr 172.21.15.27:0/367601145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb6affd640 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17e2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb6b7fe640 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 0x55a4f17ea1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/367601145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 msgr2=0x55a4f17e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17e2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/367601145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 msgr2=0x55a4f17ea1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 0x55a4f17ea1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/367601145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4f15166b0 con 0x55a4f17d9240 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e3180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000f780 tx=0x7fbb60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195c4a358ea72fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb4eff5640 1 -- 172.21.15.27:0/367601145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60004050 con 0x55a4f17d9240 2024-04-25T01:24:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb4eff5640 1 -- 172.21.15.27:0/367601145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fbb600041f0 con 0x55a4f17d9240 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb74256740 1 -- 172.21.15.27:0/367601145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb5c0043d0 con 0x55a4f17d9240 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb4eff5640 1 -- 172.21.15.27:0/367601145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbb600043f0 con 0x55a4f17d9240 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb74256740 10 client.15276 fetch_fsmap learned FSMap version 24 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb74256740 1 -- 172.21.15.27:0/367601145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a4f17ed270 con 0x55a4f17d9240 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.508+0000 7fbb4eff5640 1 -- 172.21.15.27:0/367601145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb5c0043d0 con 0x55a4f17d9240 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb4eff5640 1 -- 172.21.15.27:0/367601145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbb60026070 con 0x55a4f17d9240 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb74256740 10 client.15276 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb74256740 10 client.15276 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb74256740 20 client.15276 populate_metadata read hostname 'smithi027' 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb74256740 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a4f17ef9a0 0x55a4f17f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb74256740 4 client.15276 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb74256740 1 -- 172.21.15.27:0/367601145 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4f1745440 con 0x55a4f17ef9a0 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb6affd640 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a4f17ef9a0 0x55a4f17f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb4eff5640 10 client.15276 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.509+0000 7fbb6affd640 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a4f17ef9a0 0x55a4f17f0e10 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=741df4a964dd34d9 server_cookie=36864311772212c9 in_seq=0 out_seq=0 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb4eff5640 1 -- 172.21.15.27:0/367601145 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4f1745440 con 0x55a4f17ef9a0 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb4eff5640 10 client.15276 handle_command_reply: tid=0 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 client.15276 shutdown 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 -- 172.21.15.27:0/367601145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 msgr2=0x55a4f17e3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e3180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000f780 tx=0x7fbb60004730 comp rx=0 tx=0).stop 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 -- 172.21.15.27:0/367601145 shutdown_connections 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a4f17ef9a0 0x55a4f17f0e10 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17de3b0 0x55a4f17ea1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb4eff5640 0 client.15276 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e3180 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 --2- 172.21.15.27:0/367601145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17d8850 0x55a4f17e2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.510+0000 7fbb74256740 1 -- 172.21.15.27:0/367601145 >> 172.21.15.27:0/367601145 conn(0x55a4f17cfab0 msgr2=0x55a4f17ceae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.511+0000 7fbb74256740 1 -- 172.21.15.27:0/367601145 shutdown_connections 2024-04-25T01:24:50.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.511+0000 7fbb74256740 1 -- 172.21.15.27:0/367601145 wait complete. 2024-04-25T01:24:50.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.511+0000 7fbb74256740 20 client.15276 trim_cache size 0 max 16384 2024-04-25T01:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb74256740 1 Processor -- start 2024-04-25T01:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb74256740 1 -- start start 2024-04-25T01:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb74256740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17ddc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb74256740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ea180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb74256740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 0x55a4f17e2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb74256740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a4f17ef150 con 0x55a4f17d9240 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb74256740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a4f17ef820 con 0x55a4f17d8850 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb74256740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a4f17efef0 con 0x55a4f17de3b0 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ea180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ea180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35384/0 (socket says 172.21.15.27:35384) 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/2780125503 learned_addr learned my addr 172.21.15.27:0/2780125503 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6affd640 1 --2- 172.21.15.27:0/2780125503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17ddc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/2780125503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 msgr2=0x55a4f17e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2780125503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 0x55a4f17e2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/2780125503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 msgr2=0x55a4f17ddc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2780125503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17ddc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/2780125503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4f17d9ad0 con 0x55a4f17d9240 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2780125503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ea180 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbb60028920 tx=0x7fbb60004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85d7f88234c8143 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.529+0000 7fbb50ff9640 1 -- 172.21.15.27:0/2780125503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60028440 con 0x55a4f17d9240 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb50ff9640 1 -- 172.21.15.27:0/2780125503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb600285e0 con 0x55a4f17d9240 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb50ff9640 1 -- 172.21.15.27:0/2780125503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60004a00 con 0x55a4f17d9240 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 -- 172.21.15.27:0/2780125503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 msgr2=0x55a4f17ea180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 --2- 172.21.15.27:0/2780125503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ea180 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbb60028920 tx=0x7fbb60004640 comp rx=0 tx=0).stop 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 -- 172.21.15.27:0/2780125503 shutdown_connections 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 --2- 172.21.15.27:0/2780125503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 0x55a4f17e2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 --2- 172.21.15.27:0/2780125503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17ea180 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 --2- 172.21.15.27:0/2780125503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17ddc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 -- 172.21.15.27:0/2780125503 >> 172.21.15.27:0/2780125503 conn(0x55a4f17cfab0 msgr2=0x55a4f17ca690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 -- 172.21.15.27:0/2780125503 shutdown_connections 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 -- 172.21.15.27:0/2780125503 wait complete. 2024-04-25T01:24:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 -- start start 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.530+0000 7fbb74256740 1 Processor -- start 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb74256740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17e8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb74256740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb74256740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 0x55a4f17e8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb74256740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a4f17f0c90 con 0x55a4f17d9240 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb74256740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a4f17f05c0 con 0x55a4f17d8850 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb74256740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a4f17f1360 con 0x55a4f17de3b0 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb74256740 10 client.0 _ll_register_callbacks cb 0x7fbb60011350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35388/0 (socket says 172.21.15.27:35388) 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/2464681079 learned_addr learned my addr 172.21.15.27:0/2464681079 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6affd640 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17e8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/2464681079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 msgr2=0x55a4f17e8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6b7fe640 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 0x55a4f17e8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 0x55a4f17e8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 msgr2=0x55a4f17e8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17e8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 -- 172.21.15.27:0/2464681079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4f17e95b0 con 0x55a4f17d9240 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6affd640 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17e8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb6a7fc640 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e8860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009a90 tx=0x7fbb60005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf2b9cec94214f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb4eff5640 1 -- 172.21.15.27:0/2464681079 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6003a020 con 0x55a4f17d9240 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb4eff5640 1 -- 172.21.15.27:0/2464681079 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fbb600048d0 con 0x55a4f17d9240 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.531+0000 7fbb74256740 1 -- 172.21.15.27:0/2464681079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb600285e0 con 0x55a4f17d9240 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb4eff5640 1 -- 172.21.15.27:0/2464681079 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbb60004ad0 con 0x55a4f17d9240 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb74256740 10 client.15282 fetch_fsmap learned FSMap version 24 2024-04-25T01:24:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb74256740 1 -- 172.21.15.27:0/2464681079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a4f1855b60 con 0x55a4f17d9240 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb4eff5640 1 -- 172.21.15.27:0/2464681079 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb600285e0 con 0x55a4f17d9240 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb4eff5640 1 -- 172.21.15.27:0/2464681079 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbb60026040 con 0x55a4f17d9240 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb74256740 10 client.15282 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb74256740 10 client.15282 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb74256740 20 client.15282 populate_metadata read hostname 'smithi027' 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb74256740 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a4f185a5c0 0x55a4f18592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb74256740 4 client.15282 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb74256740 1 -- 172.21.15.27:0/2464681079 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "status"}) -- 0x55a4f173f790 con 0x55a4f185a5c0 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.532+0000 7fbb6affd640 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a4f185a5c0 0x55a4f18592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb6affd640 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a4f185a5c0 0x55a4f18592d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc790ce80c9bd54a server_cookie=384bb8900555c6ab in_seq=0 out_seq=0 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb4eff5640 10 client.15282 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb4eff5640 1 -- 172.21.15.27:0/2464681079 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a4f173f790 con 0x55a4f185a5c0 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb4eff5640 10 client.15282 handle_command_reply: tid=0 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 client.15282 shutdown 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 -- 172.21.15.27:0/2464681079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 msgr2=0x55a4f17e8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e8860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009a90 tx=0x7fbb60005250 comp rx=0 tx=0).stop 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 -- 172.21.15.27:0/2464681079 shutdown_connections 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a4f185a5c0 0x55a4f18592d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a4f17de3b0 0x55a4f17e8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f17d9240 0x55a4f17e8860 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb4eff5640 0 client.15282 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 --2- 172.21.15.27:0/2464681079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a4f17d8850 0x55a4f17e8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 -- 172.21.15.27:0/2464681079 >> 172.21.15.27:0/2464681079 conn(0x55a4f17cfab0 msgr2=0x55a4f17dd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 -- 172.21.15.27:0/2464681079 shutdown_connections 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 1 -- 172.21.15.27:0/2464681079 wait complete. 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.533+0000 7fbb74256740 20 client.15282 trim_cache size 0 max 16384 2024-04-25T01:24:50.522 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24521, 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 112.71165400300001, 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 24, 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 118.109340614 2024-04-25T01:24:50.523 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 -- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbb540a2b90 msgr2=0x7fbb540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbb540a2b90 0x7fbb540a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55a4f17d9aa0 tx=0x7fbb60009910 comp rx=0 tx=0).stop 2024-04-25T01:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 -- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 msgr2=0x7fbb6c19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 0x7fbb6c19f080 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbb64031ae0 tx=0x7fbb64031ee0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 -- 172.21.15.27:0/2946893418 shutdown_connections 2024-04-25T01:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbb540a2b90 0x7fbb540a5050 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb6c107520 0x7fbb6c19f080 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb6c104bf0 0x7fbb6c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 --2- 172.21.15.27:0/2946893418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbb6c1022c0 0x7fbb6c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 -- 172.21.15.27:0/2946893418 >> 172.21.15.27:0/2946893418 conn(0x7fbb6c0fbbf0 msgr2=0x7fbb6c102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 -- 172.21.15.27:0/2946893418 shutdown_connections 2024-04-25T01:24:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:50.534+0000 7fbb7154f640 1 -- 172.21.15.27:0/2946893418 wait complete. 2024-04-25T01:24:50.526 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T01:24:50.527 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:24:50.537 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:24:50.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T01:24:50.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:50 smithi027 ceph-mon[29197]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 639 B/s wr, 7 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:50.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:50 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3549921074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:50.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:50 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/573157434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:50.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:50 smithi153 ceph-mon[32621]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 639 B/s wr, 7 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:50.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:50 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3549921074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:50.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:50 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/573157434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:50.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:50 smithi139 ceph-mon[32760]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 639 B/s wr, 7 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:50.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:50 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3549921074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:50.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:50 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/573157434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:52.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:52 smithi027 ceph-mon[29197]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 639 B/s wr, 6 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 -- 172.21.15.27:0/2609104760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 msgr2=0x7fb44c1051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 --2- 172.21.15.27:0/2609104760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 0x7fb44c1051d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb448009990 tx=0x7fb448009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 -- 172.21.15.27:0/2609104760 shutdown_connections 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 --2- 172.21.15.27:0/2609104760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb44c108040 0x7fb44c10a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 --2- 172.21.15.27:0/2609104760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 0x7fb44c1051d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 --2- 172.21.15.27:0/2609104760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb44c105710 0x7fb44c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 -- 172.21.15.27:0/2609104760 >> 172.21.15.27:0/2609104760 conn(0x7fb44c0fc730 msgr2=0x7fb44c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 -- 172.21.15.27:0/2609104760 shutdown_connections 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.661+0000 7fb453910700 1 -- 172.21.15.27:0/2609104760 wait complete. 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb453910700 1 Processor -- start 2024-04-25T01:24:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb453910700 1 -- start start 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb453910700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 0x7fb44c19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb453910700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb44c105710 0x7fb44c19abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb453910700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb44c108040 0x7fb44c1a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb453910700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c10de60 con 0x7fb44c102de0 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb453910700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c10dce0 con 0x7fb44c105710 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb453910700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c10dfe0 con 0x7fb44c108040 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb4516ac700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 0x7fb44c19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb4516ac700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 0x7fb44c19a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35408/0 (socket says 172.21.15.27:35408) 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb4516ac700 1 -- 172.21.15.27:0/1188801591 learned_addr learned my addr 172.21.15.27:0/1188801591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.662+0000 7fb451ead700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb44c108040 0x7fb44c1a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb450eab700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb44c105710 0x7fb44c19abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4516ac700 1 -- 172.21.15.27:0/1188801591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb44c108040 msgr2=0x7fb44c1a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4516ac700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb44c108040 0x7fb44c1a1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4516ac700 1 -- 172.21.15.27:0/1188801591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb44c105710 msgr2=0x7fb44c19abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4516ac700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb44c105710 0x7fb44c19abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4516ac700 1 -- 172.21.15.27:0/1188801591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44c1a22e0 con 0x7fb44c102de0 2024-04-25T01:24:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb451ead700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb44c108040 0x7fb44c1a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4516ac700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 0x7fb44c19a680 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb4480095b0 tx=0x7fb448009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4427fc700 1 -- 172.21.15.27:0/1188801591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb448027070 con 0x7fb44c102de0 2024-04-25T01:24:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4427fc700 1 -- 172.21.15.27:0/1188801591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb448004150 con 0x7fb44c102de0 2024-04-25T01:24:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb4427fc700 1 -- 172.21.15.27:0/1188801591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44800f460 con 0x7fb44c102de0 2024-04-25T01:24:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb44c1a2570 con 0x7fb44c102de0 2024-04-25T01:24:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.663+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb44c1a29c0 con 0x7fb44c102de0 2024-04-25T01:24:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.664+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb44c068f20 con 0x7fb44c102de0 2024-04-25T01:24:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.666+0000 7fb4427fc700 1 -- 172.21.15.27:0/1188801591 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb44802a070 con 0x7fb44c102de0 2024-04-25T01:24:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.666+0000 7fb4427fc700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4380a2c20 0x7fb4380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.667+0000 7fb450eab700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4380a2c20 0x7fb4380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.667+0000 7fb4427fc700 1 -- 172.21.15.27:0/1188801591 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fb4480d0ac0 con 0x7fb44c102de0 2024-04-25T01:24:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.667+0000 7fb450eab700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4380a2c20 0x7fb4380a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00ba60 tx=0x7fb43c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:52.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.670+0000 7fb4427fc700 1 -- 172.21.15.27:0/1188801591 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb448031020 con 0x7fb44c102de0 2024-04-25T01:24:52.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:52 smithi153 ceph-mon[32621]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 639 B/s wr, 6 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:52.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:52 smithi139 ceph-mon[32760]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 639 B/s wr, 6 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:52.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.892+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fb44c062670 con 0x7fb4380a2c20 2024-04-25T01:24:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.893+0000 7fb4427fc700 1 -- 172.21.15.27:0/1188801591 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fb44c062670 con 0x7fb4380a2c20 2024-04-25T01:24:52.874 INFO:teuthology.orchestra.run.smithi027.stdout:Backend: cephadm 2024-04-25T01:24:52.874 INFO:teuthology.orchestra.run.smithi027.stdout:Available: Yes 2024-04-25T01:24:52.874 INFO:teuthology.orchestra.run.smithi027.stdout:Paused: No 2024-04-25T01:24:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.895+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4380a2c20 msgr2=0x7fb4380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.895+0000 7fb453910700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4380a2c20 0x7fb4380a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00ba60 tx=0x7fb43c005d50 comp rx=0 tx=0).stop 2024-04-25T01:24:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.895+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 msgr2=0x7fb44c19a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.895+0000 7fb453910700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 0x7fb44c19a680 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb4480095b0 tx=0x7fb448009ee0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.895+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 shutdown_connections 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.895+0000 7fb453910700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fb4380a2c20 0x7fb4380a50e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.895+0000 7fb453910700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb44c108040 0x7fb44c1a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.896+0000 7fb453910700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb44c102de0 0x7fb44c19a680 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.896+0000 7fb453910700 1 --2- 172.21.15.27:0/1188801591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb44c105710 0x7fb44c19abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.896+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 >> 172.21.15.27:0/1188801591 conn(0x7fb44c0fc730 msgr2=0x7fb44c102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.896+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 shutdown_connections 2024-04-25T01:24:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:52.896+0000 7fb453910700 1 -- 172.21.15.27:0/1188801591 wait complete. 2024-04-25T01:24:53.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T01:24:53.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:53 smithi153 ceph-mon[32621]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:53.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:53 smithi139 ceph-mon[32760]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:53.913 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:53.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:53 smithi027 ceph-mon[29197]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:54.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:54 smithi027 ceph-mon[29197]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 639 B/s wr, 5 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:54.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:54 smithi153 ceph-mon[32621]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 639 B/s wr, 5 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:54.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:54 smithi139 ceph-mon[32760]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 639 B/s wr, 5 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-04-25T01:24:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 -- 172.21.15.27:0/2832280915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160102d20 msgr2=0x7f2160105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 --2- 172.21.15.27:0/2832280915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160102d20 0x7f2160105110 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2150009990 tx=0x7f2150009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 -- 172.21.15.27:0/2832280915 shutdown_connections 2024-04-25T01:24:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 --2- 172.21.15.27:0/2832280915 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2160107f80 0x7f216010a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 --2- 172.21.15.27:0/2832280915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160102d20 0x7f2160105110 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 --2- 172.21.15.27:0/2832280915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2160105650 0x7f2160107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 -- 172.21.15.27:0/2832280915 >> 172.21.15.27:0/2832280915 conn(0x7f21600fc730 msgr2=0x7f21600feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 -- 172.21.15.27:0/2832280915 shutdown_connections 2024-04-25T01:24:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.875+0000 7f2165e34700 1 -- 172.21.15.27:0/2832280915 wait complete. 2024-04-25T01:24:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f2165e34700 1 Processor -- start 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f2165e34700 1 -- start start 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f2165e34700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2160102d20 0x7f21601984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f2165e34700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2160105650 0x7f21601989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f2165e34700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160107f80 0x7f216019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f2165e34700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f216010dea0 con 0x7f2160107f80 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f2165e34700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f216010dd20 con 0x7f2160105650 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f2165e34700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f216010e020 con 0x7f2160102d20 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f215ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160107f80 0x7f216019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f215ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160107f80 0x7f216019fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35430/0 (socket says 172.21.15.27:35430) 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f215ffff700 1 -- 172.21.15.27:0/4276367276 learned_addr learned my addr 172.21.15.27:0/4276367276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f215f7fe700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2160102d20 0x7f21601984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.876+0000 7f215effd700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2160105650 0x7f21601989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215ffff700 1 -- 172.21.15.27:0/4276367276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2160102d20 msgr2=0x7f21601984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215ffff700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2160102d20 0x7f21601984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215ffff700 1 -- 172.21.15.27:0/4276367276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2160105650 msgr2=0x7f21601989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215ffff700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2160105650 0x7f21601989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215ffff700 1 -- 172.21.15.27:0/4276367276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21601a0100 con 0x7f2160107f80 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215ffff700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160107f80 0x7f216019fa60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2158007d70 tx=0x7f215800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215cff9700 1 -- 172.21.15.27:0/4276367276 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2158016020 con 0x7f2160107f80 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21601a03f0 con 0x7f2160107f80 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215cff9700 1 -- 172.21.15.27:0/4276367276 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215800ac50 con 0x7f2160107f80 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f215cff9700 1 -- 172.21.15.27:0/4276367276 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21580115e0 con 0x7f2160107f80 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.877+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21601a0930 con 0x7f2160107f80 2024-04-25T01:24:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.878+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2160068f20 con 0x7f2160107f80 2024-04-25T01:24:55.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.879+0000 7f215cff9700 1 -- 172.21.15.27:0/4276367276 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2158019020 con 0x7f2160107f80 2024-04-25T01:24:55.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.880+0000 7f215cff9700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21480a2c20 0x7f21480a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:55.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.880+0000 7f215f7fe700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21480a2c20 0x7f21480a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:55.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.880+0000 7f215cff9700 1 -- 172.21.15.27:0/4276367276 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f21580cf270 con 0x7f2160107f80 2024-04-25T01:24:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.880+0000 7f215f7fe700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21480a2c20 0x7f21480a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f215000b5c0 tx=0x7f2150003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:55.884+0000 7f215cff9700 1 -- 172.21.15.27:0/4276367276 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2158020020 con 0x7f2160107f80 2024-04-25T01:24:56.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.101+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2160062670 con 0x7f21480a2c20 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.115+0000 7f215cff9700 1 -- 172.21.15.27:0/4276367276 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f2160062670 con 0x7f21480a2c20 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.a smithi027 running (2m) 20s ago 2m 13.9M - 19.0.0-2445-gd2c45d97 f68e37b807ef ff3101829846 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.b smithi139 running (2m) 57s ago 2m 26.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef ffb82643ef8d 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.c smithi153 running (2m) 57s ago 2m 17.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef dd126713b2e7 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.d smithi027 running (2m) 20s ago 2m 15.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef b7ee065b9d57 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.e smithi139 running (2m) 57s ago 2m 17.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 9d7cc3edf7cf 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.f smithi153 running (2m) 57s ago 2m 16.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef 80297802d4b3 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.g smithi027 running (2m) 20s ago 2m 16.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 1b992ee59261 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.h smithi139 running (2m) 57s ago 2m 18.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef e5e6102d99a6 2024-04-25T01:24:56.096 INFO:teuthology.orchestra.run.smithi027.stdout:mds.i smithi153 running (2m) 57s ago 2m 19.8M - 19.0.0-2445-gd2c45d97 f68e37b807ef 4b77951ab54e 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mds.j smithi027 running (2m) 20s ago 2m 15.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 7a740179891c 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mds.k smithi139 running (2m) 57s ago 2m 15.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 7b6f4eeef614 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mds.l smithi153 running (2m) 57s ago 2m 14.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 5ec7d31dad34 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.x smithi027 *:9283,8765 running (9m) 20s ago 9m 514M - 19.0.0-2445-gd2c45d97 f68e37b807ef 63362b3c482c 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.y smithi139 *:8443,8765 running (7m) 57s ago 7m 452M - 19.0.0-2445-gd2c45d97 f68e37b807ef 12df09f15ded 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.z smithi153 *:8443,8765 running (8m) 57s ago 7m 454M - 19.0.0-2445-gd2c45d97 f68e37b807ef f231c531f574 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a smithi027 running (10m) 20s ago 10m 49.1M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef be8d93f92e99 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b smithi139 running (8m) 57s ago 8m 42.4M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 4dc38a28c2a8 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c smithi153 running (8m) 57s ago 8m 39.9M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 4c8282d0292e 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (7m) 20s ago 7m 46.1M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 1775e1bbea11 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi139 running (7m) 57s ago 7m 49.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef dd8934475080 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi153 running (6m) 57s ago 6m 59.8M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef bbf301f64991 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi027 running (6m) 20s ago 6m 46.7M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef a61c45ad8b69 2024-04-25T01:24:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi139 running (6m) 57s ago 6m 54.2M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 645b44028640 2024-04-25T01:24:56.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi153 running (5m) 57s ago 5m 56.8M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 6e3c7881334b 2024-04-25T01:24:56.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 smithi027 running (5m) 20s ago 5m 42.2M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef d8c06dd9b0ae 2024-04-25T01:24:56.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 smithi139 running (4m) 57s ago 4m 43.8M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef f44a9f64f6a3 2024-04-25T01:24:56.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 smithi153 running (4m) 57s ago 4m 45.2M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 2e1a75727449 2024-04-25T01:24:56.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 smithi027 running (3m) 20s ago 3m 44.0M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef bf7d2bc6bf80 2024-04-25T01:24:56.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 smithi139 running (3m) 57s ago 3m 54.9M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef c1f8fc219675 2024-04-25T01:24:56.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 smithi153 running (2m) 57s ago 2m 53.2M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 1a923186da80 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21480a2c20 msgr2=0x7f21480a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21480a2c20 0x7f21480a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f215000b5c0 tx=0x7f2150003960 comp rx=0 tx=0).stop 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160107f80 msgr2=0x7f216019fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160107f80 0x7f216019fa60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2158007d70 tx=0x7f215800a340 comp rx=0 tx=0).stop 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 shutdown_connections 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21480a2c20 0x7f21480a50e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2160107f80 0x7f216019fa60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2160102d20 0x7f21601984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 --2- 172.21.15.27:0/4276367276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2160105650 0x7f21601989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 >> 172.21.15.27:0/4276367276 conn(0x7f21600fc730 msgr2=0x7f2160103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 shutdown_connections 2024-04-25T01:24:56.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:56.118+0000 7f2165e34700 1 -- 172.21.15.27:0/4276367276 wait complete. 2024-04-25T01:24:56.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:56 smithi153 ceph-mon[32621]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 573 B/s wr, 7 op/s; 7.0 KiB/s, 0 objects/s recovering 2024-04-25T01:24:56.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:56 smithi153 ceph-mon[32621]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:56.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:56 smithi027 ceph-mon[29197]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 573 B/s wr, 7 op/s; 7.0 KiB/s, 0 objects/s recovering 2024-04-25T01:24:56.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:56 smithi027 ceph-mon[29197]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:56.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:56 smithi139 ceph-mon[32760]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 573 B/s wr, 7 op/s; 7.0 KiB/s, 0 objects/s recovering 2024-04-25T01:24:56.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:56 smithi139 ceph-mon[32760]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:56.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T01:24:57.061 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:58.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:58 smithi153 ceph-mon[32621]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 409 B/s wr, 5 op/s; 6.2 KiB/s, 0 objects/s recovering 2024-04-25T01:24:58.793 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:58 smithi139 ceph-mon[32760]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 409 B/s wr, 5 op/s; 6.2 KiB/s, 0 objects/s recovering 2024-04-25T01:24:58.831 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:58 smithi027 ceph-mon[29197]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 409 B/s wr, 5 op/s; 6.2 KiB/s, 0 objects/s recovering 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 -- 172.21.15.27:0/2388461864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c102d30 msgr2=0x7f407c1095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 --2- 172.21.15.27:0/2388461864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c102d30 0x7f407c1095e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f407400b0d0 tx=0x7f407400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 -- 172.21.15.27:0/2388461864 shutdown_connections 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 --2- 172.21.15.27:0/2388461864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c102d30 0x7f407c1095e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 --2- 172.21.15.27:0/2388461864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f407c101190 0x7f407c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 --2- 172.21.15.27:0/2388461864 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f407c102390 0x7f407c1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 -- 172.21.15.27:0/2388461864 >> 172.21.15.27:0/2388461864 conn(0x7f407c0fc730 msgr2=0x7f407c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 -- 172.21.15.27:0/2388461864 shutdown_connections 2024-04-25T01:24:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.858+0000 7f4080b27700 1 -- 172.21.15.27:0/2388461864 wait complete. 2024-04-25T01:24:58.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f4080b27700 1 Processor -- start 2024-04-25T01:24:58.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f4080b27700 1 -- start start 2024-04-25T01:24:58.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f4080b27700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c101190 0x7f407c06aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f4080b27700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f407c102390 0x7f407c19d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f4080b27700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f407c102d30 0x7f407c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f4080b27700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f407c10cdf0 con 0x7f407c101190 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f4080b27700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f407c10cc70 con 0x7f407c102390 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f4080b27700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f407c10cf70 con 0x7f407c102d30 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f407a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c101190 0x7f407c06aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f407a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c101190 0x7f407c06aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34232/0 (socket says 172.21.15.27:34232) 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.859+0000 7f407a59c700 1 -- 172.21.15.27:0/1545748269 learned_addr learned my addr 172.21.15.27:0/1545748269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f407ad9d700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f407c102d30 0x7f407c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f407a59c700 1 -- 172.21.15.27:0/1545748269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f407c102d30 msgr2=0x7f407c19f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f4079d9b700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f407c102390 0x7f407c19d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f407a59c700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f407c102d30 0x7f407c19f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f407a59c700 1 -- 172.21.15.27:0/1545748269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f407c102390 msgr2=0x7f407c19d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f407a59c700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f407c102390 0x7f407c19d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f407a59c700 1 -- 172.21.15.27:0/1545748269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407c1a0160 con 0x7f407c101190 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f407ad9d700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f407c102d30 0x7f407c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f4079d9b700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f407c102390 0x7f407c19d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f407a59c700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c101190 0x7f407c06aad0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f407000b500 tx=0x7f407000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f406b7fe700 1 -- 172.21.15.27:0/1545748269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070015020 con 0x7f407c101190 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f406b7fe700 1 -- 172.21.15.27:0/1545748269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4070011d10 con 0x7f407c101190 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f406b7fe700 1 -- 172.21.15.27:0/1545748269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407000f2e0 con 0x7f407c101190 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f407c1a0450 con 0x7f407c101190 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.860+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f407c1a0a10 con 0x7f407c101190 2024-04-25T01:24:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.861+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f407c068f20 con 0x7f407c101190 2024-04-25T01:24:58.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.862+0000 7f406b7fe700 1 -- 172.21.15.27:0/1545748269 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4070018020 con 0x7f407c101190 2024-04-25T01:24:58.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.863+0000 7f406b7fe700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f40640a2c20 0x7f40640a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:58.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.863+0000 7f4079d9b700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f40640a2c20 0x7f40640a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:58.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.863+0000 7f406b7fe700 1 -- 172.21.15.27:0/1545748269 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f40700cec00 con 0x7f407c101190 2024-04-25T01:24:58.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.863+0000 7f4079d9b700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f40640a2c20 0x7f40640a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f406c005fd0 tx=0x7f406c005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:58.867+0000 7f406b7fe700 1 -- 172.21.15.27:0/1545748269 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f40700d5080 con 0x7f407c101190 2024-04-25T01:24:59.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.085+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f407c062670 con 0x7f40640a2c20 2024-04-25T01:24:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.091+0000 7f406b7fe700 1 -- 172.21.15.27:0/1545748269 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f407c062670 con 0x7f40640a2c20 2024-04-25T01:24:59.071 INFO:teuthology.orchestra.run.smithi027.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T01:24:59.071 INFO:teuthology.orchestra.run.smithi027.stdout:mds.all 12/12 60s ago 2m smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi139=b;smithi139=e;smithi139=h;smithi139=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-25T01:24:59.071 INFO:teuthology.orchestra.run.smithi027.stdout:mgr 3/3 60s ago 8m smithi027=x;smithi139=y;smithi153=z;count:3 2024-04-25T01:24:59.071 INFO:teuthology.orchestra.run.smithi027.stdout:mon 3/3 60s ago 8m smithi027:172.21.15.27=a;smithi139:172.21.15.139=b;smithi153:172.21.15.153=c;count:3 2024-04-25T01:24:59.071 INFO:teuthology.orchestra.run.smithi027.stdout:osd 12 60s ago - 2024-04-25T01:24:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f40640a2c20 msgr2=0x7f40640a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f40640a2c20 0x7f40640a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f406c005fd0 tx=0x7f406c005dc0 comp rx=0 tx=0).stop 2024-04-25T01:24:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c101190 msgr2=0x7f407c06aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c101190 0x7f407c06aad0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f407000b500 tx=0x7f407000b8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 shutdown_connections 2024-04-25T01:24:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f40640a2c20 0x7f40640a50e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f407c102d30 0x7f407c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f407c101190 0x7f407c06aad0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:59.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 --2- 172.21.15.27:0/1545748269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f407c102390 0x7f407c19d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:59.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 >> 172.21.15.27:0/1545748269 conn(0x7f407c0fc730 msgr2=0x7f407c105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:59.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 shutdown_connections 2024-04-25T01:24:59.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:24:59.093+0000 7f4080b27700 1 -- 172.21.15.27:0/1545748269 wait complete. 2024-04-25T01:24:59.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:24:59 smithi027 ceph-mon[29197]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:59.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T01:24:59.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:24:59 smithi153 ceph-mon[32621]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:59.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:24:59 smithi139 ceph-mon[32760]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:00.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:00 smithi027 ceph-mon[29197]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 343 B/s wr, 7 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-25T01:25:00.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:00 smithi153 ceph-mon[32621]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 343 B/s wr, 7 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-25T01:25:00.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:00 smithi139 ceph-mon[32760]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 343 B/s wr, 7 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 -- 172.21.15.27:0/2217305685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4102d30 msgr2=0x7f35e41095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 --2- 172.21.15.27:0/2217305685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4102d30 0x7f35e41095e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00b0d0 tx=0x7f35dc00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 -- 172.21.15.27:0/2217305685 shutdown_connections 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 --2- 172.21.15.27:0/2217305685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4102d30 0x7f35e41095e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 --2- 172.21.15.27:0/2217305685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e4101190 0x7f35e41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 --2- 172.21.15.27:0/2217305685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e4102390 0x7f35e41027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 -- 172.21.15.27:0/2217305685 >> 172.21.15.27:0/2217305685 conn(0x7f35e40fc730 msgr2=0x7f35e40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 -- 172.21.15.27:0/2217305685 shutdown_connections 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.810+0000 7f35ec41e700 1 -- 172.21.15.27:0/2217305685 wait complete. 2024-04-25T01:25:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ec41e700 1 Processor -- start 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ec41e700 1 -- start start 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ec41e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4101190 0x7f35e41051f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ec41e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e4102390 0x7f35e4103840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ec41e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e4102d30 0x7f35e4103d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ec41e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410ce10 con 0x7f35e4101190 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ec41e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410cc90 con 0x7f35e4102390 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ec41e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410cf90 con 0x7f35e4102d30 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ea1ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4101190 0x7f35e41051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ea1ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4101190 0x7f35e41051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34258/0 (socket says 172.21.15.27:34258) 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35ea1ba700 1 -- 172.21.15.27:0/494064262 learned_addr learned my addr 172.21.15.27:0/494064262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.811+0000 7f35e99b9700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e4102390 0x7f35e4103840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ea9bb700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e4102d30 0x7f35e4103d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ea1ba700 1 -- 172.21.15.27:0/494064262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e4102d30 msgr2=0x7f35e4103d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ea1ba700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e4102d30 0x7f35e4103d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ea1ba700 1 -- 172.21.15.27:0/494064262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e4102390 msgr2=0x7f35e4103840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ea1ba700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e4102390 0x7f35e4103840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ea1ba700 1 -- 172.21.15.27:0/494064262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e4104630 con 0x7f35e4101190 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35e99b9700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e4102390 0x7f35e4103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ea1ba700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4101190 0x7f35e41051f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b500 tx=0x7f35d400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35db7fe700 1 -- 172.21.15.27:0/494064262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d4015020 con 0x7f35e4101190 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35db7fe700 1 -- 172.21.15.27:0/494064262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35d4011d10 con 0x7f35e4101190 2024-04-25T01:25:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e41a26b0 con 0x7f35e4101190 2024-04-25T01:25:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35db7fe700 1 -- 172.21.15.27:0/494064262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d400f2e0 con 0x7f35e4101190 2024-04-25T01:25:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.812+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e41a2c40 con 0x7f35e4101190 2024-04-25T01:25:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.813+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e4068f20 con 0x7f35e4101190 2024-04-25T01:25:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.814+0000 7f35db7fe700 1 -- 172.21.15.27:0/494064262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f35d4018020 con 0x7f35e4101190 2024-04-25T01:25:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.815+0000 7f35db7fe700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f35d00a2c20 0x7f35d00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.815+0000 7f35e99b9700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f35d00a2c20 0x7f35d00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.815+0000 7f35db7fe700 1 -- 172.21.15.27:0/494064262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f35d40cf070 con 0x7f35e4101190 2024-04-25T01:25:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.815+0000 7f35e99b9700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f35d00a2c20 0x7f35d00a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f35e000bcf0 tx=0x7f35e000b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:01.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:01.819+0000 7f35db7fe700 1 -- 172.21.15.27:0/494064262 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f35d401f020 con 0x7f35e4101190 2024-04-25T01:25:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.034+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f35e41a2f50 con 0x7f35d00a2c20 2024-04-25T01:25:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.035+0000 7f35db7fe700 1 -- 172.21.15.27:0/494064262 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f35e41a2f50 con 0x7f35d00a2c20 2024-04-25T01:25:02.015 INFO:teuthology.orchestra.run.smithi027.stdout:HOST ADDR LABELS STATUS 2024-04-25T01:25:02.015 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 172.21.15.27 2024-04-25T01:25:02.015 INFO:teuthology.orchestra.run.smithi027.stdout:smithi139 172.21.15.139 2024-04-25T01:25:02.015 INFO:teuthology.orchestra.run.smithi027.stdout:smithi153 172.21.15.153 2024-04-25T01:25:02.015 INFO:teuthology.orchestra.run.smithi027.stdout:3 hosts in cluster 2024-04-25T01:25:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f35d00a2c20 msgr2=0x7f35d00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f35d00a2c20 0x7f35d00a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f35e000bcf0 tx=0x7f35e000b2c0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4101190 msgr2=0x7f35e41051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4101190 0x7f35e41051f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b500 tx=0x7f35d400b8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 shutdown_connections 2024-04-25T01:25:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f35d00a2c20 0x7f35d00a50e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e4102d30 0x7f35e4103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e4101190 0x7f35e41051f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 --2- 172.21.15.27:0/494064262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e4102390 0x7f35e4103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 >> 172.21.15.27:0/494064262 conn(0x7f35e40fc730 msgr2=0x7f35e4069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 shutdown_connections 2024-04-25T01:25:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:02.037+0000 7f35ec41e700 1 -- 172.21.15.27:0/494064262 wait complete. 2024-04-25T01:25:02.687 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:02 smithi027 ceph-mon[29197]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-04-25T01:25:02.687 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:02 smithi027 ceph-mon[29197]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:02.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T01:25:02.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:02 smithi153 ceph-mon[32621]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-04-25T01:25:02.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:02 smithi153 ceph-mon[32621]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:02.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:02 smithi139 ceph-mon[32760]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-04-25T01:25:02.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:02 smithi139 ceph-mon[32760]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:02.974 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:04.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:04 smithi153 ceph-mon[32621]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:04.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:04 smithi139 ceph-mon[32760]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:04.829 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:04 smithi027 ceph-mon[29197]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 -- 172.21.15.27:0/845178053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 msgr2=0x7f17a410a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/845178053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 0x7f17a410a6f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f179c00b0d0 tx=0x7f179c00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 -- 172.21.15.27:0/845178053 shutdown_connections 2024-04-25T01:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/845178053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 0x7f17a410a6f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:04.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/845178053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a41022a0 0x7f17a41026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/845178053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a41034a0 0x7f17a4103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 -- 172.21.15.27:0/845178053 >> 172.21.15.27:0/845178053 conn(0x7f17a40fd840 msgr2=0x7f17a40ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 -- 172.21.15.27:0/845178053 shutdown_connections 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 -- 172.21.15.27:0/845178053 wait complete. 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 Processor -- start 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.850+0000 7f17ab1cf700 1 -- start start 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17ab1cf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a41022a0 0x7f17a419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17ab1cf700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a41034a0 0x7f17a419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17ab1cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 0x7f17a41a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17ab1cf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17a410de60 con 0x7f17a4103e40 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17ab1cf700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f17a410dce0 con 0x7f17a41034a0 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17ab1cf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17a410dfe0 con 0x7f17a41022a0 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 0x7f17a41a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 0x7f17a41a1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34268/0 (socket says 172.21.15.27:34268) 2024-04-25T01:25:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 -- 172.21.15.27:0/3158435973 learned_addr learned my addr 172.21.15.27:0/3158435973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a8f6b700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a41022a0 0x7f17a419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a3fff700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a41034a0 0x7f17a419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 -- 172.21.15.27:0/3158435973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a41022a0 msgr2=0x7f17a419a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a41022a0 0x7f17a419a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 -- 172.21.15.27:0/3158435973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a41034a0 msgr2=0x7f17a419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a41034a0 0x7f17a419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 -- 172.21.15.27:0/3158435973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a41a2250 con 0x7f17a4103e40 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.851+0000 7f17a976c700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 0x7f17a41a1bb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f179c00b770 tx=0x7f179c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.852+0000 7f17a1ffb700 1 -- 172.21.15.27:0/3158435973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c003e10 con 0x7f17a4103e40 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.852+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a41a24e0 con 0x7f17a4103e40 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.852+0000 7f17a1ffb700 1 -- 172.21.15.27:0/3158435973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f179c007cc0 con 0x7f17a4103e40 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.852+0000 7f17a1ffb700 1 -- 172.21.15.27:0/3158435973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c01ba90 con 0x7f17a4103e40 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.852+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a41a2990 con 0x7f17a4103e40 2024-04-25T01:25:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.853+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a4068f20 con 0x7f17a4103e40 2024-04-25T01:25:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.854+0000 7f17a1ffb700 1 -- 172.21.15.27:0/3158435973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f179c005020 con 0x7f17a4103e40 2024-04-25T01:25:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.855+0000 7f17a1ffb700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17940a2c20 0x7f17940a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.855+0000 7f17a8f6b700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17940a2c20 0x7f17940a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.855+0000 7f17a1ffb700 1 -- 172.21.15.27:0/3158435973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f179c00f020 con 0x7f17a4103e40 2024-04-25T01:25:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.855+0000 7f17a8f6b700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17940a2c20 0x7f17940a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f17a4103300 tx=0x7f1798007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:04.858+0000 7f17a1ffb700 1 -- 172.21.15.27:0/3158435973 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f179c09d730 con 0x7f17a4103e40 2024-04-25T01:25:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.075+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f17a4075b80 con 0x7f17940a2c20 2024-04-25T01:25:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.076+0000 7f17a1ffb700 1 -- 172.21.15.27:0/3158435973 <== mgr.14152 v2:172.21.15.27:6800/4052240654 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f17a4075b80 con 0x7f17940a2c20 2024-04-25T01:25:05.056 INFO:teuthology.orchestra.run.smithi027.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T01:25:05.056 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:25:05.056 INFO:teuthology.orchestra.run.smithi027.stdout:smithi139 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500N6400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:25:05.056 INFO:teuthology.orchestra.run.smithi027.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:25:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17940a2c20 msgr2=0x7f17940a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17940a2c20 0x7f17940a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f17a4103300 tx=0x7f1798007400 comp rx=0 tx=0).stop 2024-04-25T01:25:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 msgr2=0x7f17a41a1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 0x7f17a41a1bb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f179c00b770 tx=0x7f179c007a20 comp rx=0 tx=0).stop 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 shutdown_connections 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f17940a2c20 0x7f17940a50e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a4103e40 0x7f17a41a1bb0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17a41022a0 0x7f17a419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 --2- 172.21.15.27:0/3158435973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17a41034a0 0x7f17a419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.078+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 >> 172.21.15.27:0/3158435973 conn(0x7f17a40fd840 msgr2=0x7f17a4074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.079+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 shutdown_connections 2024-04-25T01:25:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:05.079+0000 7f17ab1cf700 1 -- 172.21.15.27:0/3158435973 wait complete. 2024-04-25T01:25:05.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:05 smithi027 ceph-mon[29197]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:05.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:05 smithi153 ceph-mon[32621]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:05.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:05 smithi139 ceph-mon[32760]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:05.815 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:25:05.827 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:25:05.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T01:25:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:06.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:06 smithi153 ceph-mon[32621]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:25:06.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:06 smithi139 ceph-mon[32760]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:25:06.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:06 smithi027 ceph-mon[29197]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:25:07.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 -- 172.21.15.27:0/187818367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 msgr2=0x7f21a81051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:07.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 --2- 172.21.15.27:0/187818367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 0x7f21a81051d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009990 tx=0x7f21a4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 -- 172.21.15.27:0/187818367 shutdown_connections 2024-04-25T01:25:07.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 --2- 172.21.15.27:0/187818367 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a8108040 0x7f21a810a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 --2- 172.21.15.27:0/187818367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 0x7f21a81051d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 --2- 172.21.15.27:0/187818367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a8105710 0x7f21a8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 -- 172.21.15.27:0/187818367 >> 172.21.15.27:0/187818367 conn(0x7f21a80fc730 msgr2=0x7f21a80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:07.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 -- 172.21.15.27:0/187818367 shutdown_connections 2024-04-25T01:25:07.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.892+0000 7f21afb19700 1 -- 172.21.15.27:0/187818367 wait complete. 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.893+0000 7f21afb19700 1 Processor -- start 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.893+0000 7f21afb19700 1 -- start start 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.893+0000 7f21afb19700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 0x7f21a8198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.893+0000 7f21afb19700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a8105710 0x7f21a81989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.893+0000 7f21afb19700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a8108040 0x7f21a819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.893+0000 7f21afb19700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f21a810de40 con 0x7f21a8102de0 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.893+0000 7f21afb19700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f21a810dcc0 con 0x7f21a8105710 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.893+0000 7f21afb19700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f21a810dfc0 con 0x7f21a8108040 2024-04-25T01:25:07.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 0x7f21a8198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 0x7f21a8198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43492/0 (socket says 172.21.15.27:43492) 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 -- 172.21.15.27:0/2528765220 learned_addr learned my addr 172.21.15.27:0/2528765220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ae0b6700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a8108040 0x7f21a819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad0b4700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a8105710 0x7f21a81989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 -- 172.21.15.27:0/2528765220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a8108040 msgr2=0x7f21a819fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a8108040 0x7f21a819fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 -- 172.21.15.27:0/2528765220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a8105710 msgr2=0x7f21a81989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a8105710 0x7f21a81989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 -- 172.21.15.27:0/2528765220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a81a00e0 con 0x7f21a8102de0 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f21ad8b5700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 0x7f21a8198480 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f21a40098e0 tx=0x7f21a4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f219effd700 1 -- 172.21.15.27:0/2528765220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4026070 con 0x7f21a8102de0 2024-04-25T01:25:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f219effd700 1 -- 172.21.15.27:0/2528765220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a4004150 con 0x7f21a8102de0 2024-04-25T01:25:07.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.894+0000 7f219effd700 1 -- 172.21.15.27:0/2528765220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4004b00 con 0x7f21a8102de0 2024-04-25T01:25:07.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.895+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a81a0370 con 0x7f21a8102de0 2024-04-25T01:25:07.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.895+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a81a0770 con 0x7f21a8102de0 2024-04-25T01:25:07.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.896+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21a8068f20 con 0x7f21a8102de0 2024-04-25T01:25:07.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.898+0000 7f219effd700 1 -- 172.21.15.27:0/2528765220 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f21a4029070 con 0x7f21a8102de0 2024-04-25T01:25:07.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.898+0000 7f219effd700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21940a2ca0 0x7f21940a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:07.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.898+0000 7f219effd700 1 -- 172.21.15.27:0/2528765220 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f21a40d01f0 con 0x7f21a8102de0 2024-04-25T01:25:07.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.898+0000 7f21ad0b4700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21940a2ca0 0x7f21940a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:07.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.899+0000 7f21ad0b4700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21940a2ca0 0x7f21940a5160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2198005950 tx=0x7f219800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:07.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:07.902+0000 7f219effd700 1 -- 172.21.15.27:0/2528765220 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f21a4030020 con 0x7f21a8102de0 2024-04-25T01:25:08.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.157+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f21a8062e40 con 0x7f21a8102de0 2024-04-25T01:25:08.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.157+0000 7f219effd700 1 -- 172.21.15.27:0/2528765220 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3058 (secure 0 0 0) 0x7f21a4013070 con 0x7f21a8102de0 2024-04-25T01:25:08.137 INFO:teuthology.orchestra.run.smithi027.stdout:e24 2024-04-25T01:25:08.137 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:25:08.137 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 24 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-25T01:22:57.180706+0000 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-25T01:23:23.940979+0000 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 300 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-25T01:25:08.138 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 0 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 3 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:in 0,1,2 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=24521,1=24517,2=24505} 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-25T01:25:08.139 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3,4] 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 0 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 24521 members: 24505,24517,24521 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b{0:24521} state up:active seq 3 addr [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.h{0:24488} state up:standby-replay seq 1 addr [v2:172.21.15.139:6836/1472553553,v1:172.21.15.139:6837/1472553553] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.i{1:24517} state up:active seq 11 addr [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.f{1:24487} state up:standby-replay seq 1 addr [v2:172.21.15.153:6834/1981209233,v1:172.21.15.153:6835/1981209233] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.140 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c{2:24505} state up:active seq 13 addr [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.e{2:24452} state up:standby-replay seq 1 addr [v2:172.21.15.139:6834/418855472,v1:172.21.15.139:6835/418855472] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.27:6834/2965551549,v1:172.21.15.27:6835/2965551549] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.27:6836/3543255516,v1:172.21.15.27:6837/3543255516] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.g{-1:14637} state up:standby seq 1 addr [v2:172.21.15.27:6838/1204805980,v1:172.21.15.27:6839/1204805980] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a{-1:14655} state up:standby seq 1 addr [v2:172.21.15.27:6840/1912988733,v1:172.21.15.27:6841/1912988733] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.k{-1:24425} state up:standby seq 1 addr [v2:172.21.15.139:6832/4036036788,v1:172.21.15.139:6833/4036036788] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.l{-1:24433} state up:standby seq 1 addr [v2:172.21.15.153:6832/3706725137,v1:172.21.15.153:6833/3706725137] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.159+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21940a2ca0 msgr2=0x7f21940a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.159+0000 7f21afb19700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21940a2ca0 0x7f21940a5160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2198005950 tx=0x7f219800a300 comp rx=0 tx=0).stop 2024-04-25T01:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 msgr2=0x7f21a8198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 0x7f21a8198480 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f21a40098e0 tx=0x7f21a4009f50 comp rx=0 tx=0).stop 2024-04-25T01:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 shutdown_connections 2024-04-25T01:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f21940a2ca0 0x7f21940a5160 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21a8108040 0x7f21a819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21a8102de0 0x7f21a8198480 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 --2- 172.21.15.27:0/2528765220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f21a8105710 0x7f21a81989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 >> 172.21.15.27:0/2528765220 conn(0x7f21a80fc730 msgr2=0x7f21a81036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 shutdown_connections 2024-04-25T01:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:08.160+0000 7f21afb19700 1 -- 172.21.15.27:0/2528765220 wait complete. 2024-04-25T01:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-25T01:25:08.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:08 smithi153 ceph-mon[32621]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:08.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:08 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2528765220' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:25:08.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:08 smithi139 ceph-mon[32760]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:08.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:08 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2528765220' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:25:08.840 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:08 smithi027 ceph-mon[29197]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:08.841 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:08 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2528765220' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:25:08.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T01:25:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:10.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:10 smithi027 ceph-mon[29197]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:25:10.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:10 smithi153 ceph-mon[32621]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:25:10.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:10 smithi139 ceph-mon[32760]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:25:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.846+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/2712082160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 msgr2=0x7fcb081038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.846+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/2712082160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 0x7fcb081038f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0099c0 tx=0x7fcafc009cd0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.846+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/2712082160 shutdown_connections 2024-04-25T01:25:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.846+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/2712082160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb08103e30 0x7fcb0810a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.846+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/2712082160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb08102290 0x7fcb081026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.846+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/2712082160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 0x7fcb081038f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.846+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/2712082160 >> 172.21.15.27:0/2712082160 conn(0x7fcb080fd830 msgr2=0x7fcb080ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/2712082160 shutdown_connections 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/2712082160 wait complete. 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 Processor -- start 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 -- start start 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb08102290 0x7fcb081984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 0x7fcb08198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb08103e30 0x7fcb0819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0810ded0 con 0x7fcb08103490 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0810dd50 con 0x7fcb08103e30 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0fd8a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0810e050 con 0x7fcb08102290 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.847+0000 7fcb0d325700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 0x7fcb08198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0d325700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 0x7fcb08198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43504/0 (socket says 172.21.15.27:43504) 2024-04-25T01:25:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0d325700 1 -- 172.21.15.27:0/1807888706 learned_addr learned my addr 172.21.15.27:0/1807888706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0db26700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb08102290 0x7fcb081984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0e327700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb08103e30 0x7fcb0819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0d325700 1 -- 172.21.15.27:0/1807888706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb08102290 msgr2=0x7fcb081984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0d325700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb08102290 0x7fcb081984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0d325700 1 -- 172.21.15.27:0/1807888706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb08103e30 msgr2=0x7fcb0819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0d325700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb08103e30 0x7fcb0819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0d325700 1 -- 172.21.15.27:0/1807888706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb081a0140 con 0x7fcb08103490 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0e327700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb08103e30 0x7fcb0819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0d325700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 0x7fcb08198a20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00b580 tx=0x7fcafc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb06ffd700 1 -- 172.21.15.27:0/1807888706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc027070 con 0x7fcb08103490 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb081a03d0 con 0x7fcb08103490 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb081a0900 con 0x7fcb08103490 2024-04-25T01:25:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb06ffd700 1 -- 172.21.15.27:0/1807888706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcafc004150 con 0x7fcb08103490 2024-04-25T01:25:10.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.848+0000 7fcb06ffd700 1 -- 172.21.15.27:0/1807888706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc00f460 con 0x7fcb08103490 2024-04-25T01:25:10.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.849+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf4005360 con 0x7fcb08103490 2024-04-25T01:25:10.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.850+0000 7fcb06ffd700 1 -- 172.21.15.27:0/1807888706 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcafc02a070 con 0x7fcb08103490 2024-04-25T01:25:10.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.851+0000 7fcb06ffd700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcaf00a2c20 0x7fcaf00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:10.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.851+0000 7fcb0db26700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcaf00a2c20 0x7fcaf00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:10.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.851+0000 7fcb06ffd700 1 -- 172.21.15.27:0/1807888706 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fcafc0d01e0 con 0x7fcb08103490 2024-04-25T01:25:10.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.852+0000 7fcb0db26700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcaf00a2c20 0x7fcaf00a50e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb081032f0 tx=0x7fcaf8009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:10.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:10.856+0000 7fcb06ffd700 1 -- 172.21.15.27:0/1807888706 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcafc05e740 con 0x7fcb08103490 2024-04-25T01:25:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.064+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fcaf4005e60 con 0x7fcb08103490 2024-04-25T01:25:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.065+0000 7fcb06ffd700 1 -- 172.21.15.27:0/1807888706 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5676 (secure 0 0 0) 0x7fcafc013070 con 0x7fcb08103490 2024-04-25T01:25:11.045 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 93 2024-04-25T01:25:11.045 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:25:11.045 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-25T01:14:45.961368+0000 2024-04-25T01:25:11.045 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-25T01:24:47.477088+0000 2024-04-25T01:25:11.045 INFO:teuthology.orchestra.run.smithi027.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T01:25:11.045 INFO:teuthology.orchestra.run.smithi027.stdout:crush_version 27 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:full_ratio 0.95 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:backfillfull_ratio 0.9 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:nearfull_ratio 0.85 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:require_min_compat_client luminous 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:min_compat_client luminous 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:require_osd_release squid 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:stretch_mode_enabled false 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:max_osd 12 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6802/4134559555,v1:172.21.15.27:6803/4134559555] [v2:172.21.15.27:6804/4134559555,v1:172.21.15.27:6805/4134559555] exists,up 88cb2314-6b03-44ba-b2ec-e38198da56b8 2024-04-25T01:25:11.046 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6800/4036947883,v1:172.21.15.139:6801/4036947883] [v2:172.21.15.139:6802/4036947883,v1:172.21.15.139:6803/4036947883] exists,up 817954a3-7495-44a0-9983-511339d330f4 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/3400401807,v1:172.21.15.153:6801/3400401807] [v2:172.21.15.153:6802/3400401807,v1:172.21.15.153:6803/3400401807] exists,up cbaef244-9159-4e52-912a-0c7d4857f6fd 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6810/2907204232,v1:172.21.15.27:6811/2907204232] [v2:172.21.15.27:6812/2907204232,v1:172.21.15.27:6813/2907204232] exists,up 205e94f9-7782-426b-b5bd-1b56c93d55c9 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6808/1457327101,v1:172.21.15.139:6809/1457327101] [v2:172.21.15.139:6810/1457327101,v1:172.21.15.139:6811/1457327101] exists,up 57cc1b06-8233-435d-af98-2de145cfcc8d 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/4140079274,v1:172.21.15.153:6809/4140079274] [v2:172.21.15.153:6810/4140079274,v1:172.21.15.153:6811/4140079274] exists,up 087587ee-f7ae-4546-a39a-2a6a774c54ef 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6818/1607133562,v1:172.21.15.27:6819/1607133562] [v2:172.21.15.27:6820/1607133562,v1:172.21.15.27:6821/1607133562] exists,up 6834f4e7-9977-44db-8075-4e7596c95a0d 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 up in weight 1 up_from 51 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6816/3705167644,v1:172.21.15.139:6817/3705167644] [v2:172.21.15.139:6818/3705167644,v1:172.21.15.139:6819/3705167644] exists,up c6f1241c-9012-47d6-80e3-ef957fca03f7 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/4050878685,v1:172.21.15.153:6817/4050878685] [v2:172.21.15.153:6818/4050878685,v1:172.21.15.153:6819/4050878685] exists,up 7ceef8a0-8a21-4256-ab43-ad6e4c1cda16 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 up in weight 1 up_from 67 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6826/91302017,v1:172.21.15.27:6827/91302017] [v2:172.21.15.27:6828/91302017,v1:172.21.15.27:6829/91302017] exists,up 132f821e-9337-40f8-b79b-02322b1a7962 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 up in weight 1 up_from 72 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6824/155195264,v1:172.21.15.139:6825/155195264] [v2:172.21.15.139:6826/155195264,v1:172.21.15.139:6827/155195264] exists,up a6ee88f9-6df0-4fbe-af82-401e2c864c2d 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/3171978672,v1:172.21.15.153:6825/3171978672] [v2:172.21.15.153:6826/3171978672,v1:172.21.15.153:6827/3171978672] exists,up 79072415-ed78-47ee-b950-e02e8c9c30c7 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.5 [8,11] 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.c [2,4] 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.14 [2,4] 2024-04-25T01:25:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.1c [8,11] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.3 [4,9] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.20 [5,9] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.21 [4,6] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.23 [4,8] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.3b [3,9] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.e [9,11] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.f [9,8] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.12 [9,10] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 4.1d [5,8] 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/12403185 expires 2024-04-26T01:15:45.288446+0000 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/354957039 expires 2024-04-26T01:15:45.288446+0000 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/3353153293 expires 2024-04-26T01:15:45.288446+0000 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/234792808 expires 2024-04-26T01:15:45.288446+0000 2024-04-25T01:25:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/3645132749 expires 2024-04-26T01:15:18.102339+0000 2024-04-25T01:25:11.049 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/219526886 expires 2024-04-26T01:15:18.102339+0000 2024-04-25T01:25:11.049 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/219526886 expires 2024-04-26T01:15:18.102339+0000 2024-04-25T01:25:11.049 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/2409557466 expires 2024-04-26T01:15:18.102339+0000 2024-04-25T01:25:11.049 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/4191124222 expires 2024-04-26T01:15:18.102339+0000 2024-04-25T01:25:11.049 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/3353153293 expires 2024-04-26T01:15:45.288446+0000 2024-04-25T01:25:11.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcaf00a2c20 msgr2=0x7fcaf00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcaf00a2c20 0x7fcaf00a50e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb081032f0 tx=0x7fcaf8009040 comp rx=0 tx=0).stop 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 msgr2=0x7fcb08198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 0x7fcb08198a20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00b580 tx=0x7fcafc009580 comp rx=0 tx=0).stop 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 shutdown_connections 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fcaf00a2c20 0x7fcaf00a50e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb08103e30 0x7fcb0819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb08102290 0x7fcb081984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 --2- 172.21.15.27:0/1807888706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb08103490 0x7fcb08198a20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.067+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 >> 172.21.15.27:0/1807888706 conn(0x7fcb080fd830 msgr2=0x7fcb081067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.068+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 shutdown_connections 2024-04-25T01:25:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.068+0000 7fcb0fd8a700 1 -- 172.21.15.27:0/1807888706 wait complete. 2024-04-25T01:25:11.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:11 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1807888706' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:25:11.774 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T01:25:11.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:11.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:11 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1807888706' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:25:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.916+0000 7fc54c09a640 1 -- 172.21.15.27:0/503743596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 msgr2=0x7fc544101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.916+0000 7fc54c09a640 1 --2- 172.21.15.27:0/503743596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 0x7fc544101d10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc534009900 tx=0x7fc53402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.916+0000 7fc54c09a640 1 -- 172.21.15.27:0/503743596 shutdown_connections 2024-04-25T01:25:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.916+0000 7fc54c09a640 1 --2- 172.21.15.27:0/503743596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc544102250 0x7fc544108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.916+0000 7fc54c09a640 1 --2- 172.21.15.27:0/503743596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 0x7fc544101d10 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.916+0000 7fc54c09a640 1 --2- 172.21.15.27:0/503743596 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5441006a0 0x7fc544100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.916+0000 7fc54c09a640 1 -- 172.21.15.27:0/503743596 >> 172.21.15.27:0/503743596 conn(0x7fc5440fbc30 msgr2=0x7fc5440fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 -- 172.21.15.27:0/503743596 shutdown_connections 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 -- 172.21.15.27:0/503743596 wait complete. 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 Processor -- start 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 -- start start 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5441006a0 0x7fc544199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 0x7fc54419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc544102250 0x7fc5441a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc54410d440 con 0x7fc544101890 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc54410d2c0 con 0x7fc544102250 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc54c09a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc54410d5c0 con 0x7fc5441006a0 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc549635640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 0x7fc54419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc549635640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 0x7fc54419a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43530/0 (socket says 172.21.15.27:43530) 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.917+0000 7fc549635640 1 -- 172.21.15.27:0/2966186118 learned_addr learned my addr 172.21.15.27:0/2966186118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc54a637640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc544102250 0x7fc5441a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc549e36640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5441006a0 0x7fc544199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc549635640 1 -- 172.21.15.27:0/2966186118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5441006a0 msgr2=0x7fc544199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc549635640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5441006a0 0x7fc544199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc549635640 1 -- 172.21.15.27:0/2966186118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc544102250 msgr2=0x7fc5441a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc549635640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc544102250 0x7fc5441a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc549635640 1 -- 172.21.15.27:0/2966186118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5441a18d0 con 0x7fc544101890 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc549e36640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5441006a0 0x7fc544199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc54a637640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc544102250 0x7fc5441a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc549635640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 0x7fc54419a220 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc534009230 tx=0x7fc534004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc53affd640 1 -- 172.21.15.27:0/2966186118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53403f030 con 0x7fc544101890 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc53affd640 1 -- 172.21.15.27:0/2966186118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc534003ce0 con 0x7fc544101890 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5441a1b60 con 0x7fc544101890 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc53affd640 1 -- 172.21.15.27:0/2966186118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc534035070 con 0x7fc544101890 2024-04-25T01:25:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.918+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5441a2050 con 0x7fc544101890 2024-04-25T01:25:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.919+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5441a2220 con 0x7fc544101890 2024-04-25T01:25:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.920+0000 7fc53affd640 1 -- 172.21.15.27:0/2966186118 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc534042030 con 0x7fc544101890 2024-04-25T01:25:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.921+0000 7fc53affd640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5300a2b90 0x7fc5300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.921+0000 7fc53affd640 1 -- 172.21.15.27:0/2966186118 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc5340efe70 con 0x7fc544101890 2024-04-25T01:25:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.921+0000 7fc549e36640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5300a2b90 0x7fc5300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.921+0000 7fc549e36640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5300a2b90 0x7fc5300a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc540005fd0 tx=0x7fc540005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:11.924+0000 7fc53affd640 1 -- 172.21.15.27:0/2966186118 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc5340bca20 con 0x7fc544101890 2024-04-25T01:25:12.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:11 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1807888706' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:25:12.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.110+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5441a25b0 con 0x7fc544101890 2024-04-25T01:25:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.112+0000 7fc53affd640 1 -- 172.21.15.27:0/2966186118 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24792 (secure 0 0 0) 0x7fc5340bc320 con 0x7fc544101890 2024-04-25T01:25:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:12.092 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:24:47.477088+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5300a2b90 msgr2=0x7fc5300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5300a2b90 0x7fc5300a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc540005fd0 tx=0x7fc540005850 comp rx=0 tx=0).stop 2024-04-25T01:25:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 msgr2=0x7fc54419a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 0x7fc54419a220 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc534009230 tx=0x7fc534004360 comp rx=0 tx=0).stop 2024-04-25T01:25:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 shutdown_connections 2024-04-25T01:25:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5300a2b90 0x7fc5300a5050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc544102250 0x7fc5441a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc544101890 0x7fc54419a220 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 --2- 172.21.15.27:0/2966186118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5441006a0 0x7fc544199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.114+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 >> 172.21.15.27:0/2966186118 conn(0x7fc5440fbc30 msgr2=0x7fc544107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.115+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 shutdown_connections 2024-04-25T01:25:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.115+0000 7fc54c09a640 1 -- 172.21.15.27:0/2966186118 wait complete. 2024-04-25T01:25:12.105 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:25:12.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:25:12.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.231+0000 7fc59fbb7640 1 -- 172.21.15.27:0/44041761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 msgr2=0x7fc598101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.231+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/44041761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 0x7fc598101cf0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc588009900 tx=0x7fc58802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.231+0000 7fc59fbb7640 1 -- 172.21.15.27:0/44041761 shutdown_connections 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.231+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/44041761 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc598102230 0x7fc598108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.231+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/44041761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 0x7fc598101cf0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.231+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/44041761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc598100680 0x7fc598100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.231+0000 7fc59fbb7640 1 -- 172.21.15.27:0/44041761 >> 172.21.15.27:0/44041761 conn(0x7fc5980fbbf0 msgr2=0x7fc5980fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 -- 172.21.15.27:0/44041761 shutdown_connections 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 -- 172.21.15.27:0/44041761 wait complete. 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 Processor -- start 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 -- start start 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc598100680 0x7fc598197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 0x7fc598197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc598102230 0x7fc59819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810d420 con 0x7fc598101870 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810d2a0 con 0x7fc598102230 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59fbb7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810d5a0 con 0x7fc598100680 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59d152640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 0x7fc598197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59d152640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 0x7fc598197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43554/0 (socket says 172.21.15.27:43554) 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.232+0000 7fc59d152640 1 -- 172.21.15.27:0/927642379 learned_addr learned my addr 172.21.15.27:0/927642379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59d953640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc598100680 0x7fc598197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59e154640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc598102230 0x7fc59819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59d152640 1 -- 172.21.15.27:0/927642379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc598100680 msgr2=0x7fc598197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59d152640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc598100680 0x7fc598197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59d152640 1 -- 172.21.15.27:0/927642379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc598102230 msgr2=0x7fc59819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59d152640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc598102230 0x7fc59819efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59d152640 1 -- 172.21.15.27:0/927642379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59819f640 con 0x7fc598101870 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59d152640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 0x7fc598197f60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc588009230 tx=0x7fc588004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc58effd640 1 -- 172.21.15.27:0/927642379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58803f030 con 0x7fc598101870 2024-04-25T01:25:12.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc58effd640 1 -- 172.21.15.27:0/927642379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc588003ce0 con 0x7fc598101870 2024-04-25T01:25:12.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59819f8d0 con 0x7fc598101870 2024-04-25T01:25:12.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc58effd640 1 -- 172.21.15.27:0/927642379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588035070 con 0x7fc598101870 2024-04-25T01:25:12.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.233+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59819fe10 con 0x7fc598101870 2024-04-25T01:25:12.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.234+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5981079a0 con 0x7fc598101870 2024-04-25T01:25:12.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.235+0000 7fc58effd640 1 -- 172.21.15.27:0/927642379 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc588042030 con 0x7fc598101870 2024-04-25T01:25:12.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.236+0000 7fc58effd640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c10 0x7fc5840a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.236+0000 7fc59d953640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c10 0x7fc5840a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.236+0000 7fc58effd640 1 -- 172.21.15.27:0/927642379 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc5880efa50 con 0x7fc598101870 2024-04-25T01:25:12.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.237+0000 7fc59d953640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c10 0x7fc5840a50d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc5900058c0 tx=0x7fc590005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.241+0000 7fc58effd640 1 -- 172.21.15.27:0/927642379 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc5880bc600 con 0x7fc598101870 2024-04-25T01:25:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.426+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc598062340 con 0x7fc598101870 2024-04-25T01:25:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.427+0000 7fc58effd640 1 -- 172.21.15.27:0/927642379 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fc5880bbf00 con 0x7fc598101870 2024-04-25T01:25:12.406 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c10 msgr2=0x7fc5840a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c10 0x7fc5840a50d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc5900058c0 tx=0x7fc590005850 comp rx=0 tx=0).stop 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 msgr2=0x7fc598197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 0x7fc598197f60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc588009230 tx=0x7fc588004360 comp rx=0 tx=0).stop 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 shutdown_connections 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c10 0x7fc5840a50d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc598102230 0x7fc59819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc598101870 0x7fc598197f60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 --2- 172.21.15.27:0/927642379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc598100680 0x7fc598197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 >> 172.21.15.27:0/927642379 conn(0x7fc5980fbbf0 msgr2=0x7fc598104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 shutdown_connections 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.429+0000 7fc59fbb7640 1 -- 172.21.15.27:0/927642379 wait complete. 2024-04-25T01:25:12.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:25:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 -- 172.21.15.27:0/3072237034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4075720 msgr2=0x7f77a4073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 --2- 172.21.15.27:0/3072237034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4075720 0x7f77a4073b70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7798009900 tx=0x7f779802a480 comp rx=0 tx=0).stop 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 -- 172.21.15.27:0/3072237034 shutdown_connections 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 --2- 172.21.15.27:0/3072237034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77a4109630 0x7f77a410bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 --2- 172.21.15.27:0/3072237034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a4074140 0x7f77a40745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 --2- 172.21.15.27:0/3072237034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4075720 0x7f77a4073b70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 -- 172.21.15.27:0/3072237034 >> 172.21.15.27:0/3072237034 conn(0x7f77a40fcd10 msgr2=0x7f77a40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 -- 172.21.15.27:0/3072237034 shutdown_connections 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 -- 172.21.15.27:0/3072237034 wait complete. 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 Processor -- start 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.546+0000 7f77aa361640 1 -- start start 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77aa361640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77a4074140 0x7f77a4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77aa361640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a4075720 0x7f77a419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77aa361640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4109630 0x7f77a41a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77aa361640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f77a410f610 con 0x7f77a4109630 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77aa361640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f77a410f490 con 0x7f77a4074140 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77aa361640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f77a410f790 con 0x7f77a4075720 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4109630 0x7f77a41a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4109630 0x7f77a41a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43560/0 (socket says 172.21.15.27:43560) 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 -- 172.21.15.27:0/3769393887 learned_addr learned my addr 172.21.15.27:0/3769393887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a37fe640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a4075720 0x7f77a419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a3fff640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77a4074140 0x7f77a4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 -- 172.21.15.27:0/3769393887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a4075720 msgr2=0x7f77a419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a4075720 0x7f77a419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 -- 172.21.15.27:0/3769393887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77a4074140 msgr2=0x7f77a4199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77a4074140 0x7f77a4199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 -- 172.21.15.27:0/3769393887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a41a1930 con 0x7f77a4109630 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.547+0000 7f77a88fe640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4109630 0x7f77a41a1290 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f779c00cad0 tx=0x7f779c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.548+0000 7f77a17fa640 1 -- 172.21.15.27:0/3769393887 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c00ae20 con 0x7f77a4109630 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.548+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a41a1c20 con 0x7f77a4109630 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.548+0000 7f77a17fa640 1 -- 172.21.15.27:0/3769393887 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f779c005020 con 0x7f77a4109630 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.548+0000 7f77a17fa640 1 -- 172.21.15.27:0/3769393887 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c00f6f0 con 0x7f77a4109630 2024-04-25T01:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.548+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a41a2170 con 0x7f77a4109630 2024-04-25T01:25:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.549+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77a41083f0 con 0x7f77a4109630 2024-04-25T01:25:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.550+0000 7f77a17fa640 1 -- 172.21.15.27:0/3769393887 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f779c0051d0 con 0x7f77a4109630 2024-04-25T01:25:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.551+0000 7f77a17fa640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f778c0a2b90 0x7f778c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.551+0000 7f77a3fff640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f778c0a2b90 0x7f778c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.551+0000 7f77a3fff640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f778c0a2b90 0x7f778c0a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7798006010 tx=0x7f77980315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.551+0000 7f77a17fa640 1 -- 172.21.15.27:0/3769393887 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f779c09a0f0 con 0x7f77a4109630 2024-04-25T01:25:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.556+0000 7f77a17fa640 1 -- 172.21.15.27:0/3769393887 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f779c014030 con 0x7f77a4109630 2024-04-25T01:25:12.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:12 smithi027 ceph-mon[29197]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:12.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:12 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2966186118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:12.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:12 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/927642379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:12.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.741+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f77a402d170 con 0x7f77a4109630 2024-04-25T01:25:12.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.742+0000 7f77a17fa640 1 -- 172.21.15.27:0/3769393887 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f779c09c4f0 con 0x7f77a4109630 2024-04-25T01:25:12.721 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:12.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.743+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f778c0a2b90 msgr2=0x7f778c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.743+0000 7f77aa361640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f778c0a2b90 0x7f778c0a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7798006010 tx=0x7f77980315d0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.743+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4109630 msgr2=0x7f77a41a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.743+0000 7f77aa361640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4109630 0x7f77a41a1290 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f779c00cad0 tx=0x7f779c00cf90 comp rx=0 tx=0).stop 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.744+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 shutdown_connections 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.744+0000 7f77aa361640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f778c0a2b90 0x7f778c0a5050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.744+0000 7f77aa361640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77a4109630 0x7f77a41a1290 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.744+0000 7f77aa361640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a4075720 0x7f77a419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.744+0000 7f77aa361640 1 --2- 172.21.15.27:0/3769393887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f77a4074140 0x7f77a4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.744+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 >> 172.21.15.27:0/3769393887 conn(0x7f77a40fcd10 msgr2=0x7f77a41055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.744+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 shutdown_connections 2024-04-25T01:25:12.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.744+0000 7f77aa361640 1 -- 172.21.15.27:0/3769393887 wait complete. 2024-04-25T01:25:12.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:25:12.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:12 smithi139 ceph-mon[32760]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:12.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:12 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2966186118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:12.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:12 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/927642379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:12.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 -- 172.21.15.27:0/2692764939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c102200 msgr2=0x7fbc6c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2692764939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c102200 0x7fbc6c1045f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009900 tx=0x7fbc5c02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:12.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 -- 172.21.15.27:0/2692764939 shutdown_connections 2024-04-25T01:25:12.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2692764939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc6c107460 0x7fbc6c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2692764939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc6c104b30 0x7fbc6c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2692764939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c102200 0x7fbc6c1045f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 -- 172.21.15.27:0/2692764939 >> 172.21.15.27:0/2692764939 conn(0x7fbc6c0fbbf0 msgr2=0x7fbc6c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 -- 172.21.15.27:0/2692764939 shutdown_connections 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 -- 172.21.15.27:0/2692764939 wait complete. 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 Processor -- start 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.861+0000 7fbc717b0640 1 -- start start 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc717b0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc6c102200 0x7fbc6c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc717b0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc6c104b30 0x7fbc6c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc717b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c107460 0x7fbc6c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc717b0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c10d400 con 0x7fbc6c107460 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc717b0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c10d280 con 0x7fbc6c104b30 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc717b0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c10d580 con 0x7fbc6c102200 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c107460 0x7fbc6c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c107460 0x7fbc6c19f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43570/0 (socket says 172.21.15.27:43570) 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 -- 172.21.15.27:0/2183830785 learned_addr learned my addr 172.21.15.27:0/2183830785 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6affd640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc6c102200 0x7fbc6c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc63fff640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc6c104b30 0x7fbc6c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 -- 172.21.15.27:0/2183830785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc6c102200 msgr2=0x7fbc6c197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc6c102200 0x7fbc6c197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 -- 172.21.15.27:0/2183830785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc6c104b30 msgr2=0x7fbc6c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc6c104b30 0x7fbc6c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 -- 172.21.15.27:0/2183830785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6c19f750 con 0x7fbc6c107460 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6affd640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc6c102200 0x7fbc6c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc6b7fe640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c107460 0x7fbc6c19f0e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009fd0 tx=0x7fbc6400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc68ff9640 1 -- 172.21.15.27:0/2183830785 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64017070 con 0x7fbc6c107460 2024-04-25T01:25:12.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6c19fa40 con 0x7fbc6c107460 2024-04-25T01:25:12.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc68ff9640 1 -- 172.21.15.27:0/2183830785 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc64003e50 con 0x7fbc6c107460 2024-04-25T01:25:12.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc68ff9640 1 -- 172.21.15.27:0/2183830785 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc640106c0 con 0x7fbc6c107460 2024-04-25T01:25:12.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.862+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc6c19ff90 con 0x7fbc6c107460 2024-04-25T01:25:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.863+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc6c0fd6e0 con 0x7fbc6c107460 2024-04-25T01:25:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.865+0000 7fbc68ff9640 1 -- 172.21.15.27:0/2183830785 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbc64003970 con 0x7fbc6c107460 2024-04-25T01:25:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.865+0000 7fbc68ff9640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbc580a2b90 0x7fbc580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.865+0000 7fbc6affd640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbc580a2b90 0x7fbc580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.865+0000 7fbc68ff9640 1 -- 172.21.15.27:0/2183830785 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fbc640cf230 con 0x7fbc6c107460 2024-04-25T01:25:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.866+0000 7fbc6affd640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbc580a2b90 0x7fbc580a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c000c00 tx=0x7fbc5c0095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:12.869+0000 7fbc68ff9640 1 -- 172.21.15.27:0/2183830785 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbc6409bde0 con 0x7fbc6c107460 2024-04-25T01:25:13.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:12 smithi153 ceph-mon[32621]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:13.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:12 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2966186118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:13.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:12 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/927642379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.053+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbc6c02d170 con 0x7fbc6c107460 2024-04-25T01:25:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.054+0000 7fbc68ff9640 1 -- 172.21.15.27:0/2183830785 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fbc6409b6e0 con 0x7fbc6c107460 2024-04-25T01:25:13.033 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbc580a2b90 msgr2=0x7fbc580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbc580a2b90 0x7fbc580a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c000c00 tx=0x7fbc5c0095f0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c107460 msgr2=0x7fbc6c19f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c107460 0x7fbc6c19f0e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009fd0 tx=0x7fbc6400b260 comp rx=0 tx=0).stop 2024-04-25T01:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 shutdown_connections 2024-04-25T01:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fbc580a2b90 0x7fbc580a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc6c107460 0x7fbc6c19f0e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc6c104b30 0x7fbc6c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 --2- 172.21.15.27:0/2183830785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc6c102200 0x7fbc6c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 >> 172.21.15.27:0/2183830785 conn(0x7fbc6c0fbbf0 msgr2=0x7fbc6c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 shutdown_connections 2024-04-25T01:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.056+0000 7fbc717b0640 1 -- 172.21.15.27:0/2183830785 wait complete. 2024-04-25T01:25:13.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 -- 172.21.15.27:0/3581128493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 msgr2=0x7f65441044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 --2- 172.21.15.27:0/3581128493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 0x7f65441044f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6534009900 tx=0x7f653402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 -- 172.21.15.27:0/3581128493 shutdown_connections 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 --2- 172.21.15.27:0/3581128493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6544107360 0x7f65441097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 --2- 172.21.15.27:0/3581128493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6544104a30 0x7f6544106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 --2- 172.21.15.27:0/3581128493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 0x7f65441044f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 -- 172.21.15.27:0/3581128493 >> 172.21.15.27:0/3581128493 conn(0x7f65440fbaf0 msgr2=0x7f65440fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 -- 172.21.15.27:0/3581128493 shutdown_connections 2024-04-25T01:25:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.172+0000 7f654359e640 1 -- 172.21.15.27:0/3581128493 wait complete. 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654359e640 1 Processor -- start 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654359e640 1 -- start start 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 0x7f6544101a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6544104a30 0x7f65441000b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654359e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6544107360 0x7f65441005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654359e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f654410d300 con 0x7f6544102100 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654359e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f654410d180 con 0x7f6544107360 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654359e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f654410d480 con 0x7f6544104a30 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 0x7f6544101a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 0x7f6544101a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43598/0 (socket says 172.21.15.27:43598) 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654259c640 1 -- 172.21.15.27:0/3663483424 learned_addr learned my addr 172.21.15.27:0/3663483424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f653bfff640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6544104a30 0x7f65441000b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f6542d9d640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6544107360 0x7f65441005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654259c640 1 -- 172.21.15.27:0/3663483424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6544104a30 msgr2=0x7f65441000b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654259c640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6544104a30 0x7f65441000b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654259c640 1 -- 172.21.15.27:0/3663483424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6544107360 msgr2=0x7f65441005f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654259c640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6544107360 0x7f65441005f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f654259c640 1 -- 172.21.15.27:0/3663483424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6544100e80 con 0x7f6544102100 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f653bfff640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6544104a30 0x7f65441000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.173+0000 7f6542d9d640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6544107360 0x7f65441005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.174+0000 7f654259c640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 0x7f6544101a70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6534009230 tx=0x7f6534004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.174+0000 7f653b7fe640 1 -- 172.21.15.27:0/3663483424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653403f030 con 0x7f6544102100 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.174+0000 7f653b7fe640 1 -- 172.21.15.27:0/3663483424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6534003ce0 con 0x7f6544102100 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.174+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65441a19e0 con 0x7f6544102100 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.174+0000 7f653b7fe640 1 -- 172.21.15.27:0/3663483424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6534035070 con 0x7f6544102100 2024-04-25T01:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.174+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65441a1f30 con 0x7f6544102100 2024-04-25T01:25:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.175+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65440688d0 con 0x7f6544102100 2024-04-25T01:25:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.176+0000 7f653b7fe640 1 -- 172.21.15.27:0/3663483424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6534042030 con 0x7f6544102100 2024-04-25T01:25:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.177+0000 7f653b7fe640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65300a2b90 0x7f65300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.177+0000 7f653bfff640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65300a2b90 0x7f65300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.177+0000 7f653b7fe640 1 -- 172.21.15.27:0/3663483424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f65340efb20 con 0x7f6544102100 2024-04-25T01:25:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.177+0000 7f653bfff640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65300a2b90 0x7f65300a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f652c005d00 tx=0x7f652c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.181+0000 7f653b7fe640 1 -- 172.21.15.27:0/3663483424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f65340bc6d0 con 0x7f6544102100 2024-04-25T01:25:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.366+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6544062340 con 0x7f6544102100 2024-04-25T01:25:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.367+0000 7f653b7fe640 1 -- 172.21.15.27:0/3663483424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f65340bbfd0 con 0x7f6544102100 2024-04-25T01:25:13.346 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.368+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65300a2b90 msgr2=0x7f65300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.368+0000 7f654359e640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65300a2b90 0x7f65300a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f652c005d00 tx=0x7f652c005c70 comp rx=0 tx=0).stop 2024-04-25T01:25:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.368+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 msgr2=0x7f6544101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.368+0000 7f654359e640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 0x7f6544101a70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6534009230 tx=0x7f6534004360 comp rx=0 tx=0).stop 2024-04-25T01:25:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.368+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 shutdown_connections 2024-04-25T01:25:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.369+0000 7f654359e640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f65300a2b90 0x7f65300a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.369+0000 7f654359e640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6544107360 0x7f65441005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.369+0000 7f654359e640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6544104a30 0x7f65441000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.369+0000 7f654359e640 1 --2- 172.21.15.27:0/3663483424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6544102100 0x7f6544101a70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.369+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 >> 172.21.15.27:0/3663483424 conn(0x7f65440fbaf0 msgr2=0x7f6544102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.369+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 shutdown_connections 2024-04-25T01:25:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.369+0000 7f654359e640 1 -- 172.21.15.27:0/3663483424 wait complete. 2024-04-25T01:25:13.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:13.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 -- 172.21.15.27:0/1339382458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 msgr2=0x7f2b681098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 --2- 172.21.15.27:0/1339382458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 0x7f2b681098d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009ca0 tx=0x7f2b6002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 -- 172.21.15.27:0/1339382458 shutdown_connections 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 --2- 172.21.15.27:0/1339382458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 0x7f2b681098d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 --2- 172.21.15.27:0/1339382458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68104bb0 0x7f2b68106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 --2- 172.21.15.27:0/1339382458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b68102280 0x7f2b68104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 -- 172.21.15.27:0/1339382458 >> 172.21.15.27:0/1339382458 conn(0x7f2b680fbbf0 msgr2=0x7f2b680fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 -- 172.21.15.27:0/1339382458 shutdown_connections 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.481+0000 7f2b6e909640 1 -- 172.21.15.27:0/1339382458 wait complete. 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.482+0000 7f2b6e909640 1 Processor -- start 2024-04-25T01:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.482+0000 7f2b6e909640 1 -- start start 2024-04-25T01:25:13.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.482+0000 7f2b6e909640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b68102280 0x7f2b68197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.482+0000 7f2b6e909640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68104bb0 0x7f2b68197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.482+0000 7f2b6e909640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 0x7f2b6819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.482+0000 7f2b6e909640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6810d470 con 0x7f2b681074e0 2024-04-25T01:25:13.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.482+0000 7f2b6e909640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6810d2f0 con 0x7f2b68102280 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.482+0000 7f2b6e909640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6810d5f0 con 0x7f2b68104bb0 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 0x7f2b6819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 0x7f2b6819f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43606/0 (socket says 172.21.15.27:43606) 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 -- 172.21.15.27:0/972057603 learned_addr learned my addr 172.21.15.27:0/972057603 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b67fff640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b68102280 0x7f2b68197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b677fe640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68104bb0 0x7f2b68197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 -- 172.21.15.27:0/972057603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68104bb0 msgr2=0x7f2b68197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68104bb0 0x7f2b68197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 -- 172.21.15.27:0/972057603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b68102280 msgr2=0x7f2b68197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b68102280 0x7f2b68197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 -- 172.21.15.27:0/972057603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6819f6d0 con 0x7f2b681074e0 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b677fe640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68104bb0 0x7f2b68197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:13.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6cea6640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 0x7f2b6819f060 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2b60005e30 tx=0x7f2b60031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b657fa640 1 -- 172.21.15.27:0/972057603 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60003c30 con 0x7f2b681074e0 2024-04-25T01:25:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b6819f960 con 0x7f2b681074e0 2024-04-25T01:25:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b657fa640 1 -- 172.21.15.27:0/972057603 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b60003dd0 con 0x7f2b681074e0 2024-04-25T01:25:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b657fa640 1 -- 172.21.15.27:0/972057603 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6002fde0 con 0x7f2b681074e0 2024-04-25T01:25:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.483+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b6819ff30 con 0x7f2b681074e0 2024-04-25T01:25:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.484+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b680fd6e0 con 0x7f2b681074e0 2024-04-25T01:25:13.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.485+0000 7f2b657fa640 1 -- 172.21.15.27:0/972057603 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2b60004050 con 0x7f2b681074e0 2024-04-25T01:25:13.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.486+0000 7f2b657fa640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2b500a2b90 0x7f2b500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.486+0000 7f2b67fff640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2b500a2b90 0x7f2b500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.486+0000 7f2b657fa640 1 -- 172.21.15.27:0/972057603 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f2b600f0a30 con 0x7f2b681074e0 2024-04-25T01:25:13.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.487+0000 7f2b67fff640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2b500a2b90 0x7f2b500a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2b58005fd0 tx=0x7f2b58005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.490+0000 7f2b657fa640 1 -- 172.21.15.27:0/972057603 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b600bd5e0 con 0x7f2b681074e0 2024-04-25T01:25:13.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:13 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3769393887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:13.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:13 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2183830785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:13.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:13 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3663483424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:13.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.716+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b6802d170 con 0x7f2b681074e0 2024-04-25T01:25:13.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.717+0000 7f2b657fa640 1 -- 172.21.15.27:0/972057603 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f2b600bcee0 con 0x7f2b681074e0 2024-04-25T01:25:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:13.697 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:23:23.940979+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:25:13.697 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-25T01:25:13.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.719+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2b500a2b90 msgr2=0x7f2b500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.719+0000 7f2b6e909640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2b500a2b90 0x7f2b500a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2b58005fd0 tx=0x7f2b58005850 comp rx=0 tx=0).stop 2024-04-25T01:25:13.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.719+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 msgr2=0x7f2b6819f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.719+0000 7f2b6e909640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 0x7f2b6819f060 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2b60005e30 tx=0x7f2b60031ee0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.720+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 shutdown_connections 2024-04-25T01:25:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.720+0000 7f2b6e909640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f2b500a2b90 0x7f2b500a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.720+0000 7f2b6e909640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b681074e0 0x7f2b6819f060 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.720+0000 7f2b6e909640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b68104bb0 0x7f2b68197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.720+0000 7f2b6e909640 1 --2- 172.21.15.27:0/972057603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b68102280 0x7f2b68197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.720+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 >> 172.21.15.27:0/972057603 conn(0x7f2b680fbbf0 msgr2=0x7f2b68102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.720+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 shutdown_connections 2024-04-25T01:25:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.720+0000 7f2b6e909640 1 -- 172.21.15.27:0/972057603 wait complete. 2024-04-25T01:25:13.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:13.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:13 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3769393887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:13.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:13 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2183830785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:13.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:13 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3663483424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 -- 172.21.15.27:0/646430407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a01006a0 msgr2=0x7fe0a0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/646430407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a01006a0 0x7fe0a0100ac0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe094009900 tx=0x7fe09402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 -- 172.21.15.27:0/646430407 shutdown_connections 2024-04-25T01:25:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/646430407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0a0102250 0x7fe0a0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/646430407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0a0101890 0x7fe0a0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/646430407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a01006a0 0x7fe0a0100ac0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 -- 172.21.15.27:0/646430407 >> 172.21.15.27:0/646430407 conn(0x7fe0a00fbc30 msgr2=0x7fe0a00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 -- 172.21.15.27:0/646430407 shutdown_connections 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 -- 172.21.15.27:0/646430407 wait complete. 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 Processor -- start 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.833+0000 7fe0a57f9640 1 -- start start 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe0a57f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0a01006a0 0x7fe0a0197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe0a57f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a0101890 0x7fe0a0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe0a57f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0a0102250 0x7fe0a019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe0a57f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010c320 con 0x7fe0a0101890 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe0a57f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010c1a0 con 0x7fe0a01006a0 2024-04-25T01:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe0a57f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010c4a0 con 0x7fe0a0102250 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a0101890 0x7fe0a0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a0101890 0x7fe0a0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43628/0 (socket says 172.21.15.27:43628) 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 -- 172.21.15.27:0/2838348551 learned_addr learned my addr 172.21.15.27:0/2838348551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09effd640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0a01006a0 0x7fe0a0197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09f7fe640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0a0102250 0x7fe0a019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 -- 172.21.15.27:0/2838348551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0a0102250 msgr2=0x7fe0a019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0a0102250 0x7fe0a019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 -- 172.21.15.27:0/2838348551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0a01006a0 msgr2=0x7fe0a0197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0a01006a0 0x7fe0a0197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 -- 172.21.15.27:0/2838348551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a019f6a0 con 0x7fe0a0101890 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe09e7fc640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a0101890 0x7fe0a0197fc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe09000c920 tx=0x7fe09000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe087fff640 1 -- 172.21.15.27:0/2838348551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090007c70 con 0x7fe0a0101890 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a019f990 con 0x7fe0a0101890 2024-04-25T01:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe087fff640 1 -- 172.21.15.27:0/2838348551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe090007e10 con 0x7fe0a0101890 2024-04-25T01:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe087fff640 1 -- 172.21.15.27:0/2838348551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090018890 con 0x7fe0a0101890 2024-04-25T01:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.834+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a019ff30 con 0x7fe0a0101890 2024-04-25T01:25:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.836+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a01a0220 con 0x7fe0a0101890 2024-04-25T01:25:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.837+0000 7fe087fff640 1 -- 172.21.15.27:0/2838348551 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe090018a50 con 0x7fe0a0101890 2024-04-25T01:25:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.838+0000 7fe087fff640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe0880a2b90 0x7fe0880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.838+0000 7fe09effd640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe0880a2b90 0x7fe0880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.838+0000 7fe087fff640 1 -- 172.21.15.27:0/2838348551 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fe0900cf820 con 0x7fe0a0101890 2024-04-25T01:25:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.838+0000 7fe09effd640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe0880a2b90 0x7fe0880a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe094006010 tx=0x7fe094000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:13.842+0000 7fe087fff640 1 -- 172.21.15.27:0/2838348551 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe090014030 con 0x7fe0a0101890 2024-04-25T01:25:14.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.031+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0a01a03f0 con 0x7fe0a0101890 2024-04-25T01:25:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.033+0000 7fe087fff640 1 -- 172.21.15.27:0/2838348551 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24792 (secure 0 0 0) 0x7fe09009c460 con 0x7fe0a0101890 2024-04-25T01:25:14.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:14.013 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:24:47.477088+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:14.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe0880a2b90 msgr2=0x7fe0880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe0880a2b90 0x7fe0880a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe094006010 tx=0x7fe094000bc0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a0101890 msgr2=0x7fe0a0197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a0101890 0x7fe0a0197fc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe09000c920 tx=0x7fe09000cde0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 shutdown_connections 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fe0880a2b90 0x7fe0880a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0a0102250 0x7fe0a019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a0101890 0x7fe0a0197fc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 --2- 172.21.15.27:0/2838348551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0a01006a0 0x7fe0a0197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 >> 172.21.15.27:0/2838348551 conn(0x7fe0a00fbc30 msgr2=0x7fe0a00fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 shutdown_connections 2024-04-25T01:25:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.035+0000 7fe0a57f9640 1 -- 172.21.15.27:0/2838348551 wait complete. 2024-04-25T01:25:14.025 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:25:14.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:25:14.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:13 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3769393887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:14.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:13 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2183830785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:14.027 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:13 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3663483424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:14.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.151+0000 7f013ef4f640 1 -- 172.21.15.27:0/883049596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 msgr2=0x7f0138101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.151+0000 7f013ef4f640 1 --2- 172.21.15.27:0/883049596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 0x7f0138101cf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0124009900 tx=0x7f012402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:14.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 -- 172.21.15.27:0/883049596 shutdown_connections 2024-04-25T01:25:14.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 --2- 172.21.15.27:0/883049596 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0138102230 0x7f0138108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 --2- 172.21.15.27:0/883049596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 0x7f0138101cf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 --2- 172.21.15.27:0/883049596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0138100680 0x7f0138100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 -- 172.21.15.27:0/883049596 >> 172.21.15.27:0/883049596 conn(0x7f01380fbbf0 msgr2=0x7f01380fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 -- 172.21.15.27:0/883049596 shutdown_connections 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 -- 172.21.15.27:0/883049596 wait complete. 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 Processor -- start 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 -- start start 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0138100680 0x7f0138197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 0x7f01381980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0138102230 0x7f013819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f013810c370 con 0x7f0138101870 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f013810c1f0 con 0x7f0138102230 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f013ef4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f013810c4f0 con 0x7f0138100680 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.152+0000 7f0137fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 0x7f01381980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0137fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 0x7f01381980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43648/0 (socket says 172.21.15.27:43648) 2024-04-25T01:25:14.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0137fff640 1 -- 172.21.15.27:0/3716428798 learned_addr learned my addr 172.21.15.27:0/3716428798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f013cceb640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0138100680 0x7f0138197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f013d4ec640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0138102230 0x7f013819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0137fff640 1 -- 172.21.15.27:0/3716428798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0138100680 msgr2=0x7f0138197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0137fff640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0138100680 0x7f0138197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0137fff640 1 -- 172.21.15.27:0/3716428798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0138102230 msgr2=0x7f013819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0137fff640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0138102230 0x7f013819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0137fff640 1 -- 172.21.15.27:0/3716428798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013819f780 con 0x7f0138101870 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f013d4ec640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0138102230 0x7f013819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0137fff640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 0x7f01381980a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0124009230 tx=0x7f0124004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:14.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0135ffb640 1 -- 172.21.15.27:0/3716428798 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012403f030 con 0x7f0138101870 2024-04-25T01:25:14.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0135ffb640 1 -- 172.21.15.27:0/3716428798 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0124003ce0 con 0x7f0138101870 2024-04-25T01:25:14.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013819fa10 con 0x7f0138101870 2024-04-25T01:25:14.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f0135ffb640 1 -- 172.21.15.27:0/3716428798 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124035070 con 0x7f0138101870 2024-04-25T01:25:14.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.153+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013819ff80 con 0x7f0138101870 2024-04-25T01:25:14.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.155+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01381a01b0 con 0x7f0138101870 2024-04-25T01:25:14.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.156+0000 7f0135ffb640 1 -- 172.21.15.27:0/3716428798 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0124042030 con 0x7f0138101870 2024-04-25T01:25:14.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.156+0000 7f0135ffb640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f01280a2b90 0x7f01280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.156+0000 7f013cceb640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f01280a2b90 0x7f01280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.156+0000 7f0135ffb640 1 -- 172.21.15.27:0/3716428798 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f01240efde0 con 0x7f0138101870 2024-04-25T01:25:14.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.157+0000 7f013cceb640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f01280a2b90 0x7f01280a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f01381016d0 tx=0x7f012c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:14.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.161+0000 7f0135ffb640 1 -- 172.21.15.27:0/3716428798 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f01240bc990 con 0x7f0138101870 2024-04-25T01:25:14.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.344+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01381a0380 con 0x7f0138101870 2024-04-25T01:25:14.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.344+0000 7f0135ffb640 1 -- 172.21.15.27:0/3716428798 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f01240bc290 con 0x7f0138101870 2024-04-25T01:25:14.324 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:25:14.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.346+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f01280a2b90 msgr2=0x7f01280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.346+0000 7f013ef4f640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f01280a2b90 0x7f01280a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f01381016d0 tx=0x7f012c009110 comp rx=0 tx=0).stop 2024-04-25T01:25:14.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.346+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 msgr2=0x7f01381980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.346+0000 7f013ef4f640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 0x7f01381980a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0124009230 tx=0x7f0124004360 comp rx=0 tx=0).stop 2024-04-25T01:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.347+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 shutdown_connections 2024-04-25T01:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.347+0000 7f013ef4f640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f01280a2b90 0x7f01280a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.347+0000 7f013ef4f640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0138102230 0x7f013819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.347+0000 7f013ef4f640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0138101870 0x7f01381980a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.347+0000 7f013ef4f640 1 --2- 172.21.15.27:0/3716428798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0138100680 0x7f0138197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.347+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 >> 172.21.15.27:0/3716428798 conn(0x7f01380fbbf0 msgr2=0x7f01380fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.347+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 shutdown_connections 2024-04-25T01:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.347+0000 7f013ef4f640 1 -- 172.21.15.27:0/3716428798 wait complete. 2024-04-25T01:25:14.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:25:14.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.462+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/3501418888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec100010 msgr2=0x7fd2ec100430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.462+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/3501418888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec100010 0x7fd2ec100430 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009900 tx=0x7fd2e802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.462+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/3501418888 shutdown_connections 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.462+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/3501418888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2ec101310 0x7fd2ec101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.462+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/3501418888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec100970 0x7fd2ec100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.462+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/3501418888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec100010 0x7fd2ec100430 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.462+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/3501418888 >> 172.21.15.27:0/3501418888 conn(0x7fd2ec0fbc30 msgr2=0x7fd2ec0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/3501418888 shutdown_connections 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/3501418888 wait complete. 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 Processor -- start 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 -- start start 2024-04-25T01:25:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec100010 0x7fd2ec199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2ec100970 0x7fd2ec19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec101310 0x7fd2ec1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec10f600 con 0x7fd2ec101310 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec10f480 con 0x7fd2ec100970 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f3ba2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec10f780 con 0x7fd2ec100010 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f213f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec101310 0x7fd2ec1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f213f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec101310 0x7fd2ec1a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43678/0 (socket says 172.21.15.27:43678) 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.463+0000 7fd2f213f640 1 -- 172.21.15.27:0/1589565252 learned_addr learned my addr 172.21.15.27:0/1589565252 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f193e640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec100010 0x7fd2ec199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f113d640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2ec100970 0x7fd2ec19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f213f640 1 -- 172.21.15.27:0/1589565252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec100010 msgr2=0x7fd2ec199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f213f640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec100010 0x7fd2ec199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f213f640 1 -- 172.21.15.27:0/1589565252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2ec100970 msgr2=0x7fd2ec19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f213f640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2ec100970 0x7fd2ec19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f213f640 1 -- 172.21.15.27:0/1589565252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec1a1930 con 0x7fd2ec101310 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f193e640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec100010 0x7fd2ec199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f213f640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec101310 0x7fd2ec1a1290 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400cad0 tx=0x7fd2e400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2e2ffd640 1 -- 172.21.15.27:0/1589565252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e400ae20 con 0x7fd2ec101310 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2e2ffd640 1 -- 172.21.15.27:0/1589565252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e4005020 con 0x7fd2ec101310 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ec1a1c20 con 0x7fd2ec101310 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2e2ffd640 1 -- 172.21.15.27:0/1589565252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e400f6f0 con 0x7fd2ec101310 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.464+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ec1a2170 con 0x7fd2ec101310 2024-04-25T01:25:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.465+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2ec10ad60 con 0x7fd2ec101310 2024-04-25T01:25:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.466+0000 7fd2e2ffd640 1 -- 172.21.15.27:0/1589565252 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd2e40051d0 con 0x7fd2ec101310 2024-04-25T01:25:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.467+0000 7fd2e2ffd640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2d80a2c10 0x7fd2d80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.467+0000 7fd2f193e640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2d80a2c10 0x7fd2d80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.467+0000 7fd2e2ffd640 1 -- 172.21.15.27:0/1589565252 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fd2e40cf8f0 con 0x7fd2ec101310 2024-04-25T01:25:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.467+0000 7fd2f193e640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2d80a2c10 0x7fd2d80a50d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8006010 tx=0x7fd2e80095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:14.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.470+0000 7fd2e2ffd640 1 -- 172.21.15.27:0/1589565252 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd2e4014030 con 0x7fd2ec101310 2024-04-25T01:25:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.653+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd2ec02d3c0 con 0x7fd2ec101310 2024-04-25T01:25:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.654+0000 7fd2e2ffd640 1 -- 172.21.15.27:0/1589565252 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fd2e409c520 con 0x7fd2ec101310 2024-04-25T01:25:14.633 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2d80a2c10 msgr2=0x7fd2d80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2d80a2c10 0x7fd2d80a50d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8006010 tx=0x7fd2e80095f0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec101310 msgr2=0x7fd2ec1a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec101310 0x7fd2ec1a1290 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400cad0 tx=0x7fd2e400cf90 comp rx=0 tx=0).stop 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 shutdown_connections 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fd2d80a2c10 0x7fd2d80a50d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2ec101310 0x7fd2ec1a1290 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2ec100970 0x7fd2ec19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.655+0000 7fd2f3ba2640 1 --2- 172.21.15.27:0/1589565252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec100010 0x7fd2ec199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.656+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 >> 172.21.15.27:0/1589565252 conn(0x7fd2ec0fbc30 msgr2=0x7fd2ec0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.656+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 shutdown_connections 2024-04-25T01:25:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.656+0000 7fd2f3ba2640 1 -- 172.21.15.27:0/1589565252 wait complete. 2024-04-25T01:25:14.645 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:14 smithi027 ceph-mon[29197]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:14.645 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:14 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/972057603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.645 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:14 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2838348551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.645 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:14 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3716428798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:14.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:25:14.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 -- 172.21.15.27:0/2112104724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c101890 msgr2=0x7f6d9c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 --2- 172.21.15.27:0/2112104724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c101890 0x7f6d9c101d10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009900 tx=0x7f6d9802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:14.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 -- 172.21.15.27:0/2112104724 shutdown_connections 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 --2- 172.21.15.27:0/2112104724 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d9c102250 0x7f6d9c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 --2- 172.21.15.27:0/2112104724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c101890 0x7f6d9c101d10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 --2- 172.21.15.27:0/2112104724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c1006a0 0x7f6d9c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 -- 172.21.15.27:0/2112104724 >> 172.21.15.27:0/2112104724 conn(0x7f6d9c0fbc30 msgr2=0x7f6d9c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 -- 172.21.15.27:0/2112104724 shutdown_connections 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 -- 172.21.15.27:0/2112104724 wait complete. 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 Processor -- start 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.772+0000 7f6da328c640 1 -- start start 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da328c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c1006a0 0x7f6d9c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da328c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c101890 0x7f6d9c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da328c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d9c102250 0x7f6d9c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da328c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10d440 con 0x7f6d9c1006a0 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da328c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10d2c0 con 0x7f6d9c102250 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da328c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c10d5c0 con 0x7f6d9c101890 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c1006a0 0x7f6d9c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c1006a0 0x7f6d9c197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43698/0 (socket says 172.21.15.27:43698) 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 -- 172.21.15.27:0/2579775299 learned_addr learned my addr 172.21.15.27:0/2579775299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da0827640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c101890 0x7f6d9c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1829640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d9c102250 0x7f6d9c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 -- 172.21.15.27:0/2579775299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c101890 msgr2=0x7f6d9c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c101890 0x7f6d9c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 -- 172.21.15.27:0/2579775299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d9c102250 msgr2=0x7f6d9c19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d9c102250 0x7f6d9c19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 -- 172.21.15.27:0/2579775299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c19f6e0 con 0x7f6d9c1006a0 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1829640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d9c102250 0x7f6d9c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da1028640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c1006a0 0x7f6d9c197ac0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00e9c0 tx=0x7f6d8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:14.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6d927fc640 1 -- 172.21.15.27:0/2579775299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c00cd80 con 0x7f6d9c1006a0 2024-04-25T01:25:14.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6d927fc640 1 -- 172.21.15.27:0/2579775299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d8c0035f0 con 0x7f6d9c1006a0 2024-04-25T01:25:14.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d9c19f9d0 con 0x7f6d9c1006a0 2024-04-25T01:25:14.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6d927fc640 1 -- 172.21.15.27:0/2579775299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c010690 con 0x7f6d9c1006a0 2024-04-25T01:25:14.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.773+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d9c19fef0 con 0x7f6d9c1006a0 2024-04-25T01:25:14.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.774+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d9c0689b0 con 0x7f6d9c1006a0 2024-04-25T01:25:14.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.775+0000 7f6d927fc640 1 -- 172.21.15.27:0/2579775299 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6d8c0037a0 con 0x7f6d9c1006a0 2024-04-25T01:25:14.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.776+0000 7f6d927fc640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d880a2b90 0x7f6d880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:14.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.776+0000 7f6da0827640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d880a2b90 0x7f6d880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:14.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.776+0000 7f6d927fc640 1 -- 172.21.15.27:0/2579775299 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f6d8c0cf870 con 0x7f6d9c1006a0 2024-04-25T01:25:14.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.777+0000 7f6da0827640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d880a2b90 0x7f6d880a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6d980314e0 tx=0x7f6d98000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:14.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.781+0000 7f6d927fc640 1 -- 172.21.15.27:0/2579775299 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d8c09c420 con 0x7f6d9c1006a0 2024-04-25T01:25:14.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:14 smithi139 ceph-mon[32760]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:14.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:14 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/972057603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:14 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2838348551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:14 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3716428798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.968+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6d9c062440 con 0x7f6d9c1006a0 2024-04-25T01:25:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.968+0000 7f6d927fc640 1 -- 172.21.15.27:0/2579775299 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6d8c09bd20 con 0x7f6d9c1006a0 2024-04-25T01:25:14.948 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d880a2b90 msgr2=0x7f6d880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d880a2b90 0x7f6d880a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6d980314e0 tx=0x7f6d98000bc0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c1006a0 msgr2=0x7f6d9c197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c1006a0 0x7f6d9c197ac0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00e9c0 tx=0x7f6d8c00ee80 comp rx=0 tx=0).stop 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 shutdown_connections 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f6d880a2b90 0x7f6d880a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d9c102250 0x7f6d9c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d9c101890 0x7f6d9c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 --2- 172.21.15.27:0/2579775299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d9c1006a0 0x7f6d9c197ac0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 >> 172.21.15.27:0/2579775299 conn(0x7f6d9c0fbc30 msgr2=0x7f6d9c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 shutdown_connections 2024-04-25T01:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:14.970+0000 7f6da328c640 1 -- 172.21.15.27:0/2579775299 wait complete. 2024-04-25T01:25:14.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:25:15.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:14 smithi153 ceph-mon[32621]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:25:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:14 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/972057603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:14 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2838348551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:15.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:14 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3716428798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:15.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 -- 172.21.15.27:0/1974603647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a20103350 msgr2=0x7f4a20109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 --2- 172.21.15.27:0/1974603647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a20103350 0x7f4a20109be0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009ca0 tx=0x7f4a1802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 -- 172.21.15.27:0/1974603647 shutdown_connections 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 --2- 172.21.15.27:0/1974603647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a20103350 0x7f4a20109be0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 --2- 172.21.15.27:0/1974603647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a20102990 0x7f4a20102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 --2- 172.21.15.27:0/1974603647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a201017a0 0x7f4a20101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 -- 172.21.15.27:0/1974603647 >> 172.21.15.27:0/1974603647 conn(0x7f4a200fcd10 msgr2=0x7f4a200ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 -- 172.21.15.27:0/1974603647 shutdown_connections 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 -- 172.21.15.27:0/1974603647 wait complete. 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.086+0000 7f4a256da640 1 Processor -- start 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a256da640 1 -- start start 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a256da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a201017a0 0x7f4a20197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a256da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a20102990 0x7f4a20198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a256da640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a20103350 0x7f4a2019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a256da640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2010d4c0 con 0x7f4a201017a0 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a256da640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2010d340 con 0x7f4a20103350 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a256da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2010d640 con 0x7f4a20102990 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a201017a0 0x7f4a20197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a201017a0 0x7f4a20197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43720/0 (socket says 172.21.15.27:43720) 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 -- 172.21.15.27:0/1717086623 learned_addr learned my addr 172.21.15.27:0/1717086623 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1f7fe640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a20103350 0x7f4a2019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1e7fc640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a20102990 0x7f4a20198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 -- 172.21.15.27:0/1717086623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a20102990 msgr2=0x7f4a20198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a20102990 0x7f4a20198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 -- 172.21.15.27:0/1717086623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a20103350 msgr2=0x7f4a2019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a20103350 0x7f4a2019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 -- 172.21.15.27:0/1717086623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2019f720 con 0x7f4a201017a0 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1e7fc640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a20102990 0x7f4a20198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:15.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.087+0000 7f4a1effd640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a201017a0 0x7f4a20197b00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000b450 tx=0x7f4a1000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:15.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.088+0000 7f4a07fff640 1 -- 172.21.15.27:0/1717086623 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10015020 con 0x7f4a201017a0 2024-04-25T01:25:15.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.088+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2019fa10 con 0x7f4a201017a0 2024-04-25T01:25:15.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.088+0000 7f4a07fff640 1 -- 172.21.15.27:0/1717086623 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a10011d70 con 0x7f4a201017a0 2024-04-25T01:25:15.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.088+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2019ffb0 con 0x7f4a201017a0 2024-04-25T01:25:15.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.088+0000 7f4a07fff640 1 -- 172.21.15.27:0/1717086623 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1000f340 con 0x7f4a201017a0 2024-04-25T01:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.089+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a20108ac0 con 0x7f4a201017a0 2024-04-25T01:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.090+0000 7f4a07fff640 1 -- 172.21.15.27:0/1717086623 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4a10018020 con 0x7f4a201017a0 2024-04-25T01:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.090+0000 7f4a07fff640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4a080a2b90 0x7f4a080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.091+0000 7f4a1e7fc640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4a080a2b90 0x7f4a080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.091+0000 7f4a07fff640 1 -- 172.21.15.27:0/1717086623 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f4a100cf460 con 0x7f4a201017a0 2024-04-25T01:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.091+0000 7f4a1e7fc640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4a080a2b90 0x7f4a080a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4a20199010 tx=0x7f4a14005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.094+0000 7f4a07fff640 1 -- 172.21.15.27:0/1717086623 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a1001f020 con 0x7f4a201017a0 2024-04-25T01:25:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.281+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4a2002d320 con 0x7f4a201017a0 2024-04-25T01:25:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.282+0000 7f4a07fff640 1 -- 172.21.15.27:0/1717086623 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f4a1000d040 con 0x7f4a201017a0 2024-04-25T01:25:15.261 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:15.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4a080a2b90 msgr2=0x7f4a080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4a080a2b90 0x7f4a080a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4a20199010 tx=0x7f4a14005e50 comp rx=0 tx=0).stop 2024-04-25T01:25:15.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a201017a0 msgr2=0x7f4a20197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a201017a0 0x7f4a20197b00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000b450 tx=0x7f4a1000b910 comp rx=0 tx=0).stop 2024-04-25T01:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 shutdown_connections 2024-04-25T01:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f4a080a2b90 0x7f4a080a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a20103350 0x7f4a2019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a20102990 0x7f4a20198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 --2- 172.21.15.27:0/1717086623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a201017a0 0x7f4a20197b00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 >> 172.21.15.27:0/1717086623 conn(0x7f4a200fcd10 msgr2=0x7f4a20105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 shutdown_connections 2024-04-25T01:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.284+0000 7f4a256da640 1 -- 172.21.15.27:0/1717086623 wait complete. 2024-04-25T01:25:15.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.396+0000 7fa666f63640 1 -- 172.21.15.27:0/816899047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660101870 msgr2=0x7fa660101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.396+0000 7fa666f63640 1 --2- 172.21.15.27:0/816899047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660101870 0x7fa660101cf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009900 tx=0x7fa64c02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 -- 172.21.15.27:0/816899047 shutdown_connections 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 --2- 172.21.15.27:0/816899047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa660102230 0x7fa660108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 --2- 172.21.15.27:0/816899047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660101870 0x7fa660101cf0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 --2- 172.21.15.27:0/816899047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa660100680 0x7fa660100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 -- 172.21.15.27:0/816899047 >> 172.21.15.27:0/816899047 conn(0x7fa6600fbbf0 msgr2=0x7fa6600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 -- 172.21.15.27:0/816899047 shutdown_connections 2024-04-25T01:25:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 -- 172.21.15.27:0/816899047 wait complete. 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 Processor -- start 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 -- start start 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660100680 0x7fa660197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa660101870 0x7fa660197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa660102230 0x7fa66019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa66010d480 con 0x7fa660100680 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa66010d300 con 0x7fa660102230 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa666f63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa66010d600 con 0x7fa660101870 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa664cff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660100680 0x7fa660197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa664cff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660100680 0x7fa660197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43732/0 (socket says 172.21.15.27:43732) 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.397+0000 7fa664cff640 1 -- 172.21.15.27:0/2786557713 learned_addr learned my addr 172.21.15.27:0/2786557713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa664cff640 1 -- 172.21.15.27:0/2786557713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa660101870 msgr2=0x7fa660197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa65ffff640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa660101870 0x7fa660197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa664cff640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa660101870 0x7fa660197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa664cff640 1 -- 172.21.15.27:0/2786557713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa660102230 msgr2=0x7fa66019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa664cff640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa660102230 0x7fa66019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa664cff640 1 -- 172.21.15.27:0/2786557713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66019f680 con 0x7fa660100680 2024-04-25T01:25:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa664cff640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660100680 0x7fa660197a60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa65400d770 tx=0x7fa65400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa65dffb640 1 -- 172.21.15.27:0/2786557713 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654017070 con 0x7fa660100680 2024-04-25T01:25:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa65dffb640 1 -- 172.21.15.27:0/2786557713 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa654009c30 con 0x7fa660100680 2024-04-25T01:25:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66019f970 con 0x7fa660100680 2024-04-25T01:25:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa65dffb640 1 -- 172.21.15.27:0/2786557713 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654003dd0 con 0x7fa660100680 2024-04-25T01:25:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.398+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66019fec0 con 0x7fa660100680 2024-04-25T01:25:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.400+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa660068560 con 0x7fa660100680 2024-04-25T01:25:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.400+0000 7fa65dffb640 1 -- 172.21.15.27:0/2786557713 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa654003fb0 con 0x7fa660100680 2024-04-25T01:25:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.401+0000 7fa65dffb640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa6500a2b90 0x7fa6500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.401+0000 7fa65ffff640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa6500a2b90 0x7fa6500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.401+0000 7fa65dffb640 1 -- 172.21.15.27:0/2786557713 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fa6540cee30 con 0x7fa660100680 2024-04-25T01:25:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.402+0000 7fa65ffff640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa6500a2b90 0x7fa6500a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa64c031610 tx=0x7fa64c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:15.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.406+0000 7fa65dffb640 1 -- 172.21.15.27:0/2786557713 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa65409b9e0 con 0x7fa660100680 2024-04-25T01:25:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.632+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa660062340 con 0x7fa660100680 2024-04-25T01:25:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.633+0000 7fa65dffb640 1 -- 172.21.15.27:0/2786557713 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7fa654009de0 con 0x7fa660100680 2024-04-25T01:25:15.612 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:15.612 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:23:23.940979+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:25:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-25T01:25:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa6500a2b90 msgr2=0x7fa6500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa6500a2b90 0x7fa6500a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa64c031610 tx=0x7fa64c005fd0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660100680 msgr2=0x7fa660197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660100680 0x7fa660197a60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa65400d770 tx=0x7fa65400dc30 comp rx=0 tx=0).stop 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 shutdown_connections 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fa6500a2b90 0x7fa6500a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa660102230 0x7fa66019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa660101870 0x7fa660197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 --2- 172.21.15.27:0/2786557713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa660100680 0x7fa660197a60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 >> 172.21.15.27:0/2786557713 conn(0x7fa6600fbbf0 msgr2=0x7fa660104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 shutdown_connections 2024-04-25T01:25:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.635+0000 7fa666f63640 1 -- 172.21.15.27:0/2786557713 wait complete. 2024-04-25T01:25:15.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 -- 172.21.15.27:0/4102682240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0101790 msgr2=0x7f67a0101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 --2- 172.21.15.27:0/4102682240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0101790 0x7f67a0101bb0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6790009900 tx=0x7f679002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 -- 172.21.15.27:0/4102682240 shutdown_connections 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 --2- 172.21.15.27:0/4102682240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67a0103340 0x7f67a0109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 --2- 172.21.15.27:0/4102682240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67a0102980 0x7f67a0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 --2- 172.21.15.27:0/4102682240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0101790 0x7f67a0101bb0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 -- 172.21.15.27:0/4102682240 >> 172.21.15.27:0/4102682240 conn(0x7f67a00fcd40 msgr2=0x7f67a00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 -- 172.21.15.27:0/4102682240 shutdown_connections 2024-04-25T01:25:15.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.747+0000 7f67a6df4640 1 -- 172.21.15.27:0/4102682240 wait complete. 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a6df4640 1 Processor -- start 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a6df4640 1 -- start start 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a6df4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67a0101790 0x7f67a00754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a6df4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67a0102980 0x7f67a0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a6df4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0103340 0x7f67a0074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a6df4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010d3e0 con 0x7f67a0103340 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a6df4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010d260 con 0x7f67a0101790 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a6df4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010d560 con 0x7f67a0102980 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a5391640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0103340 0x7f67a0074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a5391640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0103340 0x7f67a0074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43752/0 (socket says 172.21.15.27:43752) 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a5391640 1 -- 172.21.15.27:0/1710727038 learned_addr learned my addr 172.21.15.27:0/1710727038 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f67a4b90640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67a0101790 0x7f67a00754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.748+0000 7f679ffff640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67a0102980 0x7f67a0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f67a5391640 1 -- 172.21.15.27:0/1710727038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67a0102980 msgr2=0x7f67a0073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f67a5391640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67a0102980 0x7f67a0073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f67a5391640 1 -- 172.21.15.27:0/1710727038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67a0101790 msgr2=0x7f67a00754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f67a5391640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67a0101790 0x7f67a00754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f67a5391640 1 -- 172.21.15.27:0/1710727038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a0074900 con 0x7f67a0103340 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f67a5391640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0103340 0x7f67a0074070 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f679800cce0 tx=0x7f679800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f679dffb640 1 -- 172.21.15.27:0/1710727038 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679800f450 con 0x7f67a0103340 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f679dffb640 1 -- 172.21.15.27:0/1710727038 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f679800fad0 con 0x7f67a0103340 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f679dffb640 1 -- 172.21.15.27:0/1710727038 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798018c10 con 0x7f67a0103340 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a01a1b50 con 0x7f67a0103340 2024-04-25T01:25:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.749+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a01a20a0 con 0x7f67a0103340 2024-04-25T01:25:15.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.750+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a0108ab0 con 0x7f67a0103340 2024-04-25T01:25:15.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.751+0000 7f679dffb640 1 -- 172.21.15.27:0/1710727038 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f679800f5f0 con 0x7f67a0103340 2024-04-25T01:25:15.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.752+0000 7f679dffb640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67880a2c10 0x7f67880a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.752+0000 7f679dffb640 1 -- 172.21.15.27:0/1710727038 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f67980cf9c0 con 0x7f67a0103340 2024-04-25T01:25:15.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.752+0000 7f67a4b90640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67880a2c10 0x7f67880a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.752+0000 7f67a4b90640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67880a2c10 0x7f67880a50d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6790005ab0 tx=0x7f6790000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:15.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.756+0000 7f679dffb640 1 -- 172.21.15.27:0/1710727038 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f67980d1e20 con 0x7f67a0103340 2024-04-25T01:25:15.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:15 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1589565252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:15.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:15 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2579775299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:15.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:15 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1717086623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:15.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.942+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67a0062340 con 0x7f67a0103340 2024-04-25T01:25:15.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.944+0000 7f679dffb640 1 -- 172.21.15.27:0/1710727038 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24792 (secure 0 0 0) 0x7f6798014030 con 0x7f67a0103340 2024-04-25T01:25:15.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:15.923 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:24:47.477088+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:15.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67880a2c10 msgr2=0x7f67880a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67880a2c10 0x7f67880a50d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6790005ab0 tx=0x7f6790000bc0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0103340 msgr2=0x7f67a0074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0103340 0x7f67a0074070 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f679800cce0 tx=0x7f679800a620 comp rx=0 tx=0).stop 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 shutdown_connections 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f67880a2c10 0x7f67880a50d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67a0103340 0x7f67a0074070 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67a0102980 0x7f67a0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 --2- 172.21.15.27:0/1710727038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67a0101790 0x7f67a00754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 >> 172.21.15.27:0/1710727038 conn(0x7f67a00fcd40 msgr2=0x7f67a0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 shutdown_connections 2024-04-25T01:25:15.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:15.946+0000 7f67a6df4640 1 -- 172.21.15.27:0/1710727038 wait complete. 2024-04-25T01:25:15.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:15 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1589565252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:15.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:15 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2579775299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:15.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:15 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1717086623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:15.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:22:57.180706+0000', 'modified': '2024-04-25T01:23:23.940979+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24521, 'mds_1': 24517, 'mds_2': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24452': {'gid': 24452, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6835/418855472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 418855472}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 418855472}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/1981209233', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 1981209233}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 1981209233}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6837/1472553553', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 1472553553}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 1472553553}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.153:6837/529371680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 529371680}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 529371680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.153:6839/2846322026', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 2846322026}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 2846322026}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.139:6839/2167485871', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 2167485871}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 2167485871}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24521, 'qdb_cluster': [24505, 24517, 24521]} 2024-04-25T01:25:15.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T01:25:15.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T01:25:16.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:15 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1589565252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:16.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:15 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2579775299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:16.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:15 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1717086623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:16.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.062+0000 7f53b8c5d640 1 -- 172.21.15.27:0/3875583624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4102230 msgr2=0x7f53b4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.062+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/3875583624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4102230 0x7f53b4108ac0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009900 tx=0x7f53ac02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:16.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.062+0000 7f53b8c5d640 1 -- 172.21.15.27:0/3875583624 shutdown_connections 2024-04-25T01:25:16.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.062+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/3875583624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4102230 0x7f53b4108ac0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/3875583624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53b4101870 0x7f53b4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/3875583624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53b4100680 0x7f53b4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 -- 172.21.15.27:0/3875583624 >> 172.21.15.27:0/3875583624 conn(0x7f53b40fbbf0 msgr2=0x7f53b40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 -- 172.21.15.27:0/3875583624 shutdown_connections 2024-04-25T01:25:16.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 -- 172.21.15.27:0/3875583624 wait complete. 2024-04-25T01:25:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 Processor -- start 2024-04-25T01:25:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 -- start start 2024-04-25T01:25:16.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53b4100680 0x7f53b41979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4101870 0x7f53b4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53b4102230 0x7f53b419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53b410c2e0 con 0x7f53b4101870 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53b410c160 con 0x7f53b4100680 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.063+0000 7f53b8c5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53b410c460 con 0x7f53b4102230 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4101870 0x7f53b4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4101870 0x7f53b4197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43764/0 (socket says 172.21.15.27:43764) 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 -- 172.21.15.27:0/1325163322 learned_addr learned my addr 172.21.15.27:0/1325163322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53b4102230 0x7f53b419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b259c640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53b4100680 0x7f53b41979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 -- 172.21.15.27:0/1325163322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53b4102230 msgr2=0x7f53b419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53b4102230 0x7f53b419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 -- 172.21.15.27:0/1325163322 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53b4100680 msgr2=0x7f53b41979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53b4100680 0x7f53b41979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 -- 172.21.15.27:0/1325163322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b419f5e0 con 0x7f53b4101870 2024-04-25T01:25:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4101870 0x7f53b4197f00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f539c00c920 tx=0x7f539c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53ab7fe640 1 -- 172.21.15.27:0/1325163322 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c007c70 con 0x7f53b4101870 2024-04-25T01:25:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53ab7fe640 1 -- 172.21.15.27:0/1325163322 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f539c007e10 con 0x7f53b4101870 2024-04-25T01:25:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b8c5d640 1 -- 172.21.15.27:0/1325163322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b419f8d0 con 0x7f53b4101870 2024-04-25T01:25:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53ab7fe640 1 -- 172.21.15.27:0/1325163322 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c018890 con 0x7f53b4101870 2024-04-25T01:25:16.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.064+0000 7f53b8c5d640 1 -- 172.21.15.27:0/1325163322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b419fe20 con 0x7f53b4101870 2024-04-25T01:25:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53ab7fe640 1 -- 172.21.15.27:0/1325163322 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f539c018a70 con 0x7f53b4101870 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53bb96e740 1 Processor -- start 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53bb96e740 1 -- start start 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 0x55f888bfb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c00930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888c00fa0 0x55f888c03430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53bb96e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f888b5ef00 con 0x55f888bfbe30 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53bb96e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f888b07750 con 0x55f888c00fa0 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53bb96e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f888b5aec0 con 0x55f888bfb440 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53b259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c00930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53b259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c00930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43780/0 (socket says 172.21.15.27:43780) 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53b259c640 1 -- 172.21.15.27:0/591872474 learned_addr learned my addr 172.21.15.27:0/591872474 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 0x55f888bfb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53ab7fe640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f53a00a2b90 0x7f53a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.067+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888c00fa0 0x55f888c03430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53b259c640 1 -- 172.21.15.27:0/591872474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 msgr2=0x55f888bfb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f53a00a2b90 0x7f53a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53b259c640 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 0x55f888bfb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53b259c640 1 -- 172.21.15.27:0/591872474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888c00fa0 msgr2=0x55f888c03430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53b259c640 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888c00fa0 0x55f888c03430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53b259c640 1 -- 172.21.15.27:0/591872474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8888732d0 con 0x55f888bfbe30 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888c00fa0 0x55f888c03430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53ab7fe640 1 -- 172.21.15.27:0/1325163322 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f539c0cf490 con 0x7f53b4101870 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53b259c640 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c00930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f53a400af10 tx=0x7f53a400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d0d1bf3db1f9f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53a8ff9640 1 -- 172.21.15.27:0/591872474 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4010840 con 0x55f888bfbe30 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53a8ff9640 1 -- 172.21.15.27:0/591872474 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a40183f0 con 0x55f888bfbe30 2024-04-25T01:25:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.068+0000 7f53a8ff9640 1 -- 172.21.15.27:0/591872474 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40175a0 con 0x55f888bfbe30 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- 172.21.15.27:0/591872474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 msgr2=0x55f888c00930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f53a00a2b90 0x7f53a00a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f53ac006010 tx=0x7f53ac02a410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c00930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f53a400af10 tx=0x7f53a400f790 comp rx=0 tx=0).stop 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- 172.21.15.27:0/591872474 shutdown_connections 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888c00fa0 0x55f888c03430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c00930 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 --2- 172.21.15.27:0/591872474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 0x55f888bfb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- 172.21.15.27:0/591872474 >> 172.21.15.27:0/591872474 conn(0x55f888bf6ab0 msgr2=0x55f888bf6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- 172.21.15.27:0/591872474 shutdown_connections 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- 172.21.15.27:0/591872474 wait complete. 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- start start 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 Processor -- start 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c05860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c05da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888c00fa0 0x55f888c0cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f888c03f50 con 0x55f888bfbe30 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f888b1acf0 con 0x55f888bfb440 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f888c040d0 con 0x55f888c00fa0 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53bb96e740 10 client.0 _ll_register_callbacks cb 0x55f888b1a6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.069+0000 7f53b1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c05da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c05da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43794/0 (socket says 172.21.15.27:43794) 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b1d9b640 1 -- 172.21.15.27:0/1416343013 learned_addr learned my addr 172.21.15.27:0/1416343013 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b259c640 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c05860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b1d9b640 1 -- 172.21.15.27:0/1416343013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888c00fa0 msgr2=0x55f888c0cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888c00fa0 0x55f888c0cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b1d9b640 1 -- 172.21.15.27:0/1416343013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 msgr2=0x55f888c05860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c05860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b1d9b640 1 -- 172.21.15.27:0/1416343013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f88893d6b0 con 0x55f888bfbe30 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b259c640 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c05860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c05da0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f539c007590 tx=0x7f539c09b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa823a870dc59fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f5399ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f5399ffb640 1 -- 172.21.15.27:0/1416343013 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c09bba0 con 0x55f888bfbe30 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f5399ffb640 1 -- 172.21.15.27:0/1416343013 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f539c018d60 con 0x55f888bfbe30 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53bb96e740 1 -- 172.21.15.27:0/1416343013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a40183f0 con 0x55f888bfbe30 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f5399ffb640 1 -- 172.21.15.27:0/1416343013 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f539c09b940 con 0x55f888bfbe30 2024-04-25T01:25:16.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53bb96e740 10 client.15414 fetch_fsmap learned FSMap version 24 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f53bb96e740 1 -- 172.21.15.27:0/1416343013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f888c0fee0 con 0x55f888bfbe30 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.070+0000 7f5399ffb640 1 -- 172.21.15.27:0/1416343013 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a40183f0 con 0x55f888bfbe30 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f5399ffb640 1 -- 172.21.15.27:0/1416343013 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f539c0860e0 con 0x55f888bfbe30 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f53bb96e740 10 client.15414 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f53bb96e740 10 client.15414 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f53bb96e740 20 client.15414 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f53bb96e740 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f888c125d0 0x55f888c13a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f53bb96e740 4 client.15414 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f53bb96e740 1 -- 172.21.15.27:0/1416343013 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f888b6c440 con 0x55f888c125d0 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f53b259c640 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f888c125d0 0x55f888c13a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f53b259c640 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f888c125d0 0x55f888c13a40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f2d95c116d11922 server_cookie=a4b2c03d140d7ec3 in_seq=0 out_seq=0 2024-04-25T01:25:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.071+0000 7f5399ffb640 10 client.15414 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f5399ffb640 1 -- 172.21.15.27:0/1416343013 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f888b6c440 con 0x55f888c125d0 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f5399ffb640 10 client.15414 handle_command_reply: tid=0 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 client.15414 shutdown 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 -- 172.21.15.27:0/1416343013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 msgr2=0x55f888c05da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c05da0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f539c007590 tx=0x7f539c09b7d0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 -- 172.21.15.27:0/1416343013 shutdown_connections 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f888c125d0 0x55f888c13a40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888c00fa0 0x55f888c0cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f5399ffb640 0 client.15414 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888bfbe30 0x55f888c05da0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 --2- 172.21.15.27:0/1416343013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c05860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 -- 172.21.15.27:0/1416343013 >> 172.21.15.27:0/1416343013 conn(0x55f888bf6ab0 msgr2=0x55f888bf5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 -- 172.21.15.27:0/1416343013 shutdown_connections 2024-04-25T01:25:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 1 -- 172.21.15.27:0/1416343013 wait complete. 2024-04-25T01:25:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.072+0000 7f53bb96e740 20 client.15414 trim_cache size 0 max 16384 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.090+0000 7f53bb96e740 1 Processor -- start 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.090+0000 7f53bb96e740 1 -- start start 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c044d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfbe30 0x55f888c0cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888bf6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53bb96e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f888c11d80 con 0x55f888c00fa0 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53bb96e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f888c12450 con 0x55f888bfb440 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53bb96e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f888c12b20 con 0x55f888bfbe30 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888bf6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888bf6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43806/0 (socket says 172.21.15.27:43806) 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 -- 172.21.15.27:0/2358097313 learned_addr learned my addr 172.21.15.27:0/2358097313 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfbe30 0x55f888c0cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 -- 172.21.15.27:0/2358097313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfbe30 msgr2=0x55f888c0cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfbe30 0x55f888c0cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 -- 172.21.15.27:0/2358097313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 msgr2=0x55f888c044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b259c640 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c044d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 -- 172.21.15.27:0/2358097313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f888bf6580 con 0x55f888c00fa0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888bf6040 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009900 tx=0x7f53ac031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3476fc560d85abf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f539bfff640 1 -- 172.21.15.27:0/2358097313 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac035040 con 0x55f888c00fa0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f539bfff640 1 -- 172.21.15.27:0/2358097313 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ac04f070 con 0x55f888c00fa0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.091+0000 7f539bfff640 1 -- 172.21.15.27:0/2358097313 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac003850 con 0x55f888c00fa0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 -- 172.21.15.27:0/2358097313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 msgr2=0x55f888bf6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888bf6040 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009900 tx=0x7f53ac031400 comp rx=0 tx=0).stop 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 -- 172.21.15.27:0/2358097313 shutdown_connections 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888bf6040 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfbe30 0x55f888c0cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 --2- 172.21.15.27:0/2358097313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfb440 0x55f888c044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 -- 172.21.15.27:0/2358097313 >> 172.21.15.27:0/2358097313 conn(0x55f888bf6ab0 msgr2=0x55f888bf6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 -- 172.21.15.27:0/2358097313 shutdown_connections 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 -- 172.21.15.27:0/2358097313 wait complete. 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 -- start start 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.092+0000 7f53bb96e740 1 Processor -- start 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 0x55f888c0afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfbe30 0x55f888c0b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888c0ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f888c138c0 con 0x55f888c00fa0 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f888c131f0 con 0x55f888bfbe30 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f888c13f90 con 0x55f888bfb440 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 10 client.0 _ll_register_callbacks cb 0x7f539c00a560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888c0ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888c0ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43820/0 (socket says 172.21.15.27:43820) 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 -- 172.21.15.27:0/3484350809 learned_addr learned my addr 172.21.15.27:0/3484350809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b259c640 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 0x55f888c0afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b1d9b640 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfbe30 0x55f888c0b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 -- 172.21.15.27:0/3484350809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 msgr2=0x55f888c0afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 0x55f888c0afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 -- 172.21.15.27:0/3484350809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfbe30 msgr2=0x55f888c0b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfbe30 0x55f888c0b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 -- 172.21.15.27:0/3484350809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f888c0c270 con 0x55f888c00fa0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53b2d9d640 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888c0ba60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0314e0 tx=0x7f53ac004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6d882eae5cee9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f5399ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f5399ffb640 1 -- 172.21.15.27:0/3484350809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac035030 con 0x55f888c00fa0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f5399ffb640 1 -- 172.21.15.27:0/3484350809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f53ac005460 con 0x55f888c00fa0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 1 -- 172.21.15.27:0/3484350809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53ac04f070 con 0x55f888c00fa0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f5399ffb640 1 -- 172.21.15.27:0/3484350809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f53ac0035f0 con 0x55f888c00fa0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.093+0000 7f53bb96e740 10 client.15420 fetch_fsmap learned FSMap version 24 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f53bb96e740 1 -- 172.21.15.27:0/3484350809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f888c78720 con 0x55f888c00fa0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f5399ffb640 1 -- 172.21.15.27:0/3484350809 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ac04f070 con 0x55f888c00fa0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f5399ffb640 1 -- 172.21.15.27:0/3484350809 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f53ac03b820 con 0x55f888c00fa0 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f53bb96e740 10 client.15420 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f53bb96e740 10 client.15420 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f53bb96e740 20 client.15420 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f53bb96e740 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f888c125d0 0x55f888c7bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f53bb96e740 4 client.15420 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f53bb96e740 1 -- 172.21.15.27:0/3484350809 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "status"}) -- 0x55f888b66790 con 0x55f888c125d0 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.094+0000 7f53b259c640 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f888c125d0 0x55f888c7bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53b259c640 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f888c125d0 0x55f888c7bc00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fb96b15ed656b6aa server_cookie=ff9a09393542ef71 in_seq=0 out_seq=0 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f5399ffb640 10 client.15420 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f5399ffb640 1 -- 172.21.15.27:0/3484350809 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f888b66790 con 0x55f888c125d0 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f5399ffb640 10 client.15420 handle_command_reply: tid=0 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 client.15420 shutdown 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 -- 172.21.15.27:0/3484350809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 msgr2=0x55f888c0ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888c0ba60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0314e0 tx=0x7f53ac004970 comp rx=0 tx=0).stop 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 -- 172.21.15.27:0/3484350809 shutdown_connections 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f888c125d0 0x55f888c7bc00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f888c00fa0 0x55f888c0ba60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f888bfbe30 0x55f888c0b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f5399ffb640 0 client.15420 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 --2- 172.21.15.27:0/3484350809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f888bfb440 0x55f888c0afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 -- 172.21.15.27:0/3484350809 >> 172.21.15.27:0/3484350809 conn(0x55f888bf6ab0 msgr2=0x55f888bf1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 -- 172.21.15.27:0/3484350809 shutdown_connections 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 1 -- 172.21.15.27:0/3484350809 wait complete. 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.095+0000 7f53bb96e740 20 client.15420 trim_cache size 0 max 16384 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 2, 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24505, 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T01:25:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T01:25:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T01:25:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 113.162357981, 2024-04-25T01:25:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 24, 2024-04-25T01:25:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T01:25:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T01:25:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 158.44591292499999 2024-04-25T01:25:16.082 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 -- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f53a00a2b90 msgr2=0x7f53a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f53a00a2b90 0x7f53a00a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f53ac006010 tx=0x7f53ac02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 -- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4101870 msgr2=0x7f53b4197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4101870 0x7f53b4197f00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f539c00c920 tx=0x7f539c00cde0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 -- 172.21.15.27:0/1325163322 shutdown_connections 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f53a00a2b90 0x7f53a00a5050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53b4102230 0x7f53b419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53b4101870 0x7f53b4197f00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 --2- 172.21.15.27:0/1325163322 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53b4100680 0x7f53b41979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 -- 172.21.15.27:0/1325163322 >> 172.21.15.27:0/1325163322 conn(0x7f53b40fbbf0 msgr2=0x7f53b40fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 -- 172.21.15.27:0/1325163322 shutdown_connections 2024-04-25T01:25:16.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.096+0000 7f53b8c5d640 1 -- 172.21.15.27:0/1325163322 wait complete. 2024-04-25T01:25:16.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:25:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 -- 172.21.15.27:0/2627133551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 msgr2=0x7f43a4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 --2- 172.21.15.27:0/2627133551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 0x7f43a4108ae0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f439c009ca0 tx=0x7f439c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 -- 172.21.15.27:0/2627133551 shutdown_connections 2024-04-25T01:25:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 --2- 172.21.15.27:0/2627133551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 0x7f43a4108ae0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 --2- 172.21.15.27:0/2627133551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a4101890 0x7f43a4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 --2- 172.21.15.27:0/2627133551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43a41006a0 0x7f43a4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 -- 172.21.15.27:0/2627133551 >> 172.21.15.27:0/2627133551 conn(0x7f43a40fbc30 msgr2=0x7f43a40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 -- 172.21.15.27:0/2627133551 shutdown_connections 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.210+0000 7f43ac34c640 1 -- 172.21.15.27:0/2627133551 wait complete. 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43ac34c640 1 Processor -- start 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43ac34c640 1 -- start start 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43ac34c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a41006a0 0x7f43a41046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43ac34c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43a4101890 0x7f43a4102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43ac34c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 0x7f43a4103220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43ac34c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43a410c320 con 0x7f43a4102250 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43ac34c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f43a410c1a0 con 0x7f43a4101890 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43ac34c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43a410c4a0 con 0x7f43a41006a0 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 0x7f43a4103220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 0x7f43a4103220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43840/0 (socket says 172.21.15.27:43840) 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 -- 172.21.15.27:0/4120284379 learned_addr learned my addr 172.21.15.27:0/4120284379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43a98e7640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43a4101890 0x7f43a4102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a41006a0 0x7f43a41046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 -- 172.21.15.27:0/4120284379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a41006a0 msgr2=0x7f43a41046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a41006a0 0x7f43a41046a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 -- 172.21.15.27:0/4120284379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43a4101890 msgr2=0x7f43a4102ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43a4101890 0x7f43a4102ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 -- 172.21.15.27:0/4120284379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a4103ab0 con 0x7f43a4102250 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43a98e7640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43a4101890 0x7f43a4102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.211+0000 7f43aa8e9640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 0x7f43a4103220 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f439c005cc0 tx=0x7f439c009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.212+0000 7f439b7fe640 1 -- 172.21.15.27:0/4120284379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c003c30 con 0x7f43a4102250 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.212+0000 7f439b7fe640 1 -- 172.21.15.27:0/4120284379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f439c003dd0 con 0x7f43a4102250 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.212+0000 7f43ac34c640 1 -- 172.21.15.27:0/4120284379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a41a1c00 con 0x7f43a4102250 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.212+0000 7f439b7fe640 1 -- 172.21.15.27:0/4120284379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c02fc50 con 0x7f43a4102250 2024-04-25T01:25:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.212+0000 7f43ac34c640 1 -- 172.21.15.27:0/4120284379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a41a2150 con 0x7f43a4102250 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.214+0000 7f439b7fe640 1 -- 172.21.15.27:0/4120284379 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f439c004050 con 0x7f43a4102250 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.214+0000 7f43af058740 1 Processor -- start 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.214+0000 7f43af058740 1 -- start start 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43af058740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f527860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43af058740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527e30 0x56241f52c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43af058740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 0x56241f52f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43af058740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56241f48af00 con 0x56241f527e30 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43af058740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56241f433750 con 0x56241f52cfa0 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43af058740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56241f486ec0 con 0x56241f527440 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527e30 0x56241f52c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527e30 0x56241f52c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43844/0 (socket says 172.21.15.27:43844) 2024-04-25T01:25:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 -- 172.21.15.27:0/2025358951 learned_addr learned my addr 172.21.15.27:0/2025358951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43a98e7640 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f527860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43a98e7640 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 0x56241f52f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f439b7fe640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f43900a2b90 0x7f43900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 -- 172.21.15.27:0/2025358951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 msgr2=0x56241f527860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f527860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 -- 172.21.15.27:0/2025358951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 msgr2=0x56241f52f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 0x56241f52f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 -- 172.21.15.27:0/2025358951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56241f19f2d0 con 0x56241f527e30 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa8e9640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f43900a2b90 0x7f43900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f439b7fe640 1 -- 172.21.15.27:0/4120284379 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f439c0f0800 con 0x7f43a4102250 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.215+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527e30 0x56241f52c930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009b70 tx=0x7f43a000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c37595e7af897c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.216+0000 7f4398bfe640 1 -- 172.21.15.27:0/2025358951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0016070 con 0x56241f527e30 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.216+0000 7f4398bfe640 1 -- 172.21.15.27:0/2025358951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43a0005500 con 0x56241f527e30 2024-04-25T01:25:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.216+0000 7f4398bfe640 1 -- 172.21.15.27:0/2025358951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0011470 con 0x56241f527e30 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.216+0000 7f43aa8e9640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f43900a2b90 0x7f43900a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f439c0b97a0 tx=0x7f439c087ac0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.216+0000 7f43af058740 1 -- 172.21.15.27:0/2025358951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527e30 msgr2=0x56241f52c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.216+0000 7f43af058740 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527e30 0x56241f52c930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009b70 tx=0x7f43a000e6c0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.216+0000 7f43af058740 1 -- 172.21.15.27:0/2025358951 shutdown_connections 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.216+0000 7f43af058740 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 0x56241f52f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527e30 0x56241f52c930 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 --2- 172.21.15.27:0/2025358951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f527860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 -- 172.21.15.27:0/2025358951 >> 172.21.15.27:0/2025358951 conn(0x56241f522ab0 msgr2=0x56241f522eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 -- 172.21.15.27:0/2025358951 shutdown_connections 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 -- 172.21.15.27:0/2025358951 wait complete. 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 -- start start 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 Processor -- start 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f5318a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 0x56241f531de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f538e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56241f52ff50 con 0x56241f52cfa0 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56241f446cf0 con 0x56241f527e30 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56241f5300d0 con 0x56241f527440 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43af058740 10 client.0 _ll_register_callbacks cb 0x56241f4466e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43a98e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f538e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43a98e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f538e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43850/0 (socket says 172.21.15.27:43850) 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43a98e7640 1 -- 172.21.15.27:0/327437583 learned_addr learned my addr 172.21.15.27:0/327437583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.217+0000 7f43a98e7640 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f5318a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 0x56241f531de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43a98e7640 1 -- 172.21.15.27:0/327437583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 msgr2=0x56241f5318a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43a98e7640 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f5318a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43a98e7640 1 -- 172.21.15.27:0/327437583 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 msgr2=0x56241f531de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43a98e7640 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 0x56241f531de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43a98e7640 1 -- 172.21.15.27:0/327437583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56241f2696b0 con 0x56241f52cfa0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 0x56241f531de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43a98e7640 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f538e20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4394012c70 tx=0x7f43940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6758c1844b092b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f438dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f438dffb640 1 -- 172.21.15.27:0/327437583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439401b040 con 0x56241f52cfa0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f438dffb640 1 -- 172.21.15.27:0/327437583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f4394020630 con 0x56241f52cfa0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43af058740 1 -- 172.21.15.27:0/327437583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43a0005500 con 0x56241f52cfa0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f438dffb640 1 -- 172.21.15.27:0/327437583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4394011e30 con 0x56241f52cfa0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43af058740 10 client.15432 fetch_fsmap learned FSMap version 24 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43af058740 1 -- 172.21.15.27:0/327437583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56241f53be70 con 0x56241f52cfa0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f438dffb640 1 -- 172.21.15.27:0/327437583 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43a0005500 con 0x56241f52cfa0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f438dffb640 1 -- 172.21.15.27:0/327437583 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f439400f040 con 0x56241f52cfa0 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43af058740 10 client.15432 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43af058740 10 client.15432 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.218+0000 7f43af058740 20 client.15432 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.219+0000 7f43af058740 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56241f53e560 0x56241f53f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.219+0000 7f43af058740 4 client.15432 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.219+0000 7f43af058740 1 -- 172.21.15.27:0/327437583 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56241f498440 con 0x56241f53e560 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.219+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56241f53e560 0x56241f53f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.219+0000 7f438dffb640 10 client.15432 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.219+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56241f53e560 0x56241f53f9d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1a892eef5eac7c9a server_cookie=875d67526ce16bbe in_seq=0 out_seq=0 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f438dffb640 1 -- 172.21.15.27:0/327437583 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56241f498440 con 0x56241f53e560 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f438dffb640 10 client.15432 handle_command_reply: tid=0 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 client.15432 shutdown 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 -- 172.21.15.27:0/327437583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 msgr2=0x56241f538e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f538e20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4394012c70 tx=0x7f43940046c0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 -- 172.21.15.27:0/327437583 shutdown_connections 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56241f53e560 0x56241f53f9d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f538e20 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 0x56241f531de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f438dffb640 0 client.15432 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:25:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 --2- 172.21.15.27:0/327437583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f5318a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 -- 172.21.15.27:0/327437583 >> 172.21.15.27:0/327437583 conn(0x56241f522ab0 msgr2=0x56241f521ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 -- 172.21.15.27:0/327437583 shutdown_connections 2024-04-25T01:25:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 1 -- 172.21.15.27:0/327437583 wait complete. 2024-04-25T01:25:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.220+0000 7f43af058740 20 client.15432 trim_cache size 0 max 16384 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.238+0000 7f43af058740 1 Processor -- start 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.238+0000 7f43af058740 1 -- start start 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43af058740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f51d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43af058740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 0x56241f538d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43af058740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f5291f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43af058740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56241f53dd10 con 0x56241f52cfa0 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43af058740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56241f53e3e0 con 0x56241f527e30 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43af058740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56241f53eab0 con 0x56241f527440 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f5291f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f5291f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43866/0 (socket says 172.21.15.27:43866) 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 -- 172.21.15.27:0/1863685099 learned_addr learned my addr 172.21.15.27:0/1863685099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 -- 172.21.15.27:0/1863685099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 msgr2=0x56241f51d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 --2- 172.21.15.27:0/1863685099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f51d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 -- 172.21.15.27:0/1863685099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 msgr2=0x56241f538d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 --2- 172.21.15.27:0/1863685099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 0x56241f538d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 -- 172.21.15.27:0/1863685099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56241f52c4c0 con 0x56241f52cfa0 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f43a98e7640 1 --2- 172.21.15.27:0/1863685099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f5291f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4394005d00 tx=0x7f439400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a690ee7452b6ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f438ffff640 1 -- 172.21.15.27:0/1863685099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394010600 con 0x56241f52cfa0 2024-04-25T01:25:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f438ffff640 1 -- 172.21.15.27:0/1863685099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4394005510 con 0x56241f52cfa0 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.239+0000 7f438ffff640 1 -- 172.21.15.27:0/1863685099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394004bc0 con 0x56241f52cfa0 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- 172.21.15.27:0/1863685099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 msgr2=0x56241f5291f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 --2- 172.21.15.27:0/1863685099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f5291f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4394005d00 tx=0x7f439400bc80 comp rx=0 tx=0).stop 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- 172.21.15.27:0/1863685099 shutdown_connections 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 --2- 172.21.15.27:0/1863685099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f52cfa0 0x56241f5291f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 --2- 172.21.15.27:0/1863685099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f527e30 0x56241f538d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 --2- 172.21.15.27:0/1863685099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527440 0x56241f51d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- 172.21.15.27:0/1863685099 >> 172.21.15.27:0/1863685099 conn(0x56241f522ab0 msgr2=0x56241f522e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- 172.21.15.27:0/1863685099 shutdown_connections 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- 172.21.15.27:0/1863685099 wait complete. 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- start start 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 Processor -- start 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527440 0x56241f534470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527e30 0x56241f5349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 0x56241f537980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56241f53f850 con 0x56241f527440 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56241f53f180 con 0x56241f52cfa0 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56241f53ff20 con 0x56241f527e30 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43af058740 10 client.0 _ll_register_callbacks cb 0x7f4394012900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.240+0000 7f43a98e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527440 0x56241f534470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43a98e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527440 0x56241f534470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43868/0 (socket says 172.21.15.27:43868) 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43a98e7640 1 -- 172.21.15.27:0/930207099 learned_addr learned my addr 172.21.15.27:0/930207099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527e30 0x56241f5349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43a98e7640 1 -- 172.21.15.27:0/930207099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527e30 msgr2=0x56241f5349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43a98e7640 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527e30 0x56241f5349b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43a98e7640 1 -- 172.21.15.27:0/930207099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 msgr2=0x56241f537980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43a98e7640 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 0x56241f537980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43a98e7640 1 -- 172.21.15.27:0/930207099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56241f537f70 con 0x56241f527440 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43a98e7640 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527440 0x56241f534470 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4394011010 tx=0x7f439400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ec0092fc862836 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f438dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f438dffb640 1 -- 172.21.15.27:0/930207099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394010b30 con 0x56241f527440 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f438dffb640 1 -- 172.21.15.27:0/930207099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f4394005030 con 0x56241f527440 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43af058740 1 -- 172.21.15.27:0/930207099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4394005510 con 0x56241f527440 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f438dffb640 1 -- 172.21.15.27:0/930207099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4394041050 con 0x56241f527440 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43af058740 10 client.15438 fetch_fsmap learned FSMap version 24 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f43af058740 1 -- 172.21.15.27:0/930207099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56241f5a4510 con 0x56241f527440 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.241+0000 7f438dffb640 1 -- 172.21.15.27:0/930207099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4394005510 con 0x56241f527440 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f438dffb640 1 -- 172.21.15.27:0/930207099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f439402d430 con 0x56241f527440 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f43af058740 10 client.15438 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f43af058740 10 client.15438 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:25:16.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f43af058740 20 client.15438 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f43af058740 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56241f5a8ec0 0x56241f5a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f43af058740 4 client.15438 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f43af058740 1 -- 172.21.15.27:0/930207099 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "status"}) -- 0x56241f492790 con 0x56241f5a8ec0 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56241f5a8ec0 0x56241f5a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f43aa0e8640 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56241f5a8ec0 0x56241f5a59d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b6f00981b6547e9 server_cookie=c45cabb5aeaadcdf in_seq=0 out_seq=0 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.242+0000 7f438dffb640 10 client.15438 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f438dffb640 1 -- 172.21.15.27:0/930207099 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56241f492790 con 0x56241f5a8ec0 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f438dffb640 10 client.15438 handle_command_reply: tid=0 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 client.15438 shutdown 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 -- 172.21.15.27:0/930207099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527440 msgr2=0x56241f534470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527440 0x56241f534470 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4394011010 tx=0x7f439400ba40 comp rx=0 tx=0).stop 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 -- 172.21.15.27:0/930207099 shutdown_connections 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56241f5a8ec0 0x56241f5a59d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56241f52cfa0 0x56241f537980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f438dffb640 0 client.15438 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56241f527e30 0x56241f5349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 --2- 172.21.15.27:0/930207099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56241f527440 0x56241f534470 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 -- 172.21.15.27:0/930207099 >> 172.21.15.27:0/930207099 conn(0x56241f522ab0 msgr2=0x56241f51d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 -- 172.21.15.27:0/930207099 shutdown_connections 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 1 -- 172.21.15.27:0/930207099 wait complete. 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43af058740 20 client.15438 trim_cache size 0 max 16384 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:25:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 1, 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24517, 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 114.31494272099999, 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 24, 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 152.972806539 2024-04-25T01:25:16.230 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:25:16.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 -- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f43900a2b90 msgr2=0x7f43900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f43900a2b90 0x7f43900a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f439c0b97a0 tx=0x7f439c087ac0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 -- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 msgr2=0x7f43a4103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 0x7f43a4103220 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f439c005cc0 tx=0x7f439c009890 comp rx=0 tx=0).stop 2024-04-25T01:25:16.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 -- 172.21.15.27:0/4120284379 shutdown_connections 2024-04-25T01:25:16.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f43900a2b90 0x7f43900a5050 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a4102250 0x7f43a4103220 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43a4101890 0x7f43a4102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 --2- 172.21.15.27:0/4120284379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43a41006a0 0x7f43a41046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.243+0000 7f43ac34c640 1 -- 172.21.15.27:0/4120284379 >> 172.21.15.27:0/4120284379 conn(0x7f43a40fbc30 msgr2=0x7f43a4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.244+0000 7f43ac34c640 1 -- 172.21.15.27:0/4120284379 shutdown_connections 2024-04-25T01:25:16.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.244+0000 7f43ac34c640 1 -- 172.21.15.27:0/4120284379 wait complete. 2024-04-25T01:25:16.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:25:16.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 -- 172.21.15.27:0/881742316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 msgr2=0x7f32f8101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/881742316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 0x7f32f8101d10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009900 tx=0x7f32ec02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 -- 172.21.15.27:0/881742316 shutdown_connections 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/881742316 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f8102250 0x7f32f8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/881742316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 0x7f32f8101d10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/881742316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f81006a0 0x7f32f8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 -- 172.21.15.27:0/881742316 >> 172.21.15.27:0/881742316 conn(0x7f32f80fbc30 msgr2=0x7f32f80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 -- 172.21.15.27:0/881742316 shutdown_connections 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.357+0000 7f32ffa4d640 1 -- 172.21.15.27:0/881742316 wait complete. 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32ffa4d640 1 Processor -- start 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32ffa4d640 1 -- start start 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32ffa4d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f81006a0 0x7f32f8197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32ffa4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 0x7f32f8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32ffa4d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8102250 0x7f32f819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32ffa4d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32f810d440 con 0x7f32f8101890 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32ffa4d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f32f810d2c0 con 0x7f32f81006a0 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32ffa4d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32f810d5c0 con 0x7f32f8102250 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 0x7f32f8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 0x7f32f8197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43884/0 (socket says 172.21.15.27:43884) 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 -- 172.21.15.27:0/443296947 learned_addr learned my addr 172.21.15.27:0/443296947 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fdfea640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8102250 0x7f32f819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fd7e9640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f81006a0 0x7f32f8197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 -- 172.21.15.27:0/443296947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8102250 msgr2=0x7f32f819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8102250 0x7f32f819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 -- 172.21.15.27:0/443296947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f81006a0 msgr2=0x7f32f8197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f81006a0 0x7f32f8197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 -- 172.21.15.27:0/443296947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f819f680 con 0x7f32f8101890 2024-04-25T01:25:16.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.358+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 0x7f32f8197fa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009230 tx=0x7f32ec004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.359+0000 7f32ea7fc640 1 -- 172.21.15.27:0/443296947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec03f030 con 0x7f32f8101890 2024-04-25T01:25:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.359+0000 7f32ffa4d640 1 -- 172.21.15.27:0/443296947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f819f910 con 0x7f32f8101890 2024-04-25T01:25:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.359+0000 7f32ea7fc640 1 -- 172.21.15.27:0/443296947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec003ce0 con 0x7f32f8101890 2024-04-25T01:25:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.359+0000 7f32ea7fc640 1 -- 172.21.15.27:0/443296947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec035070 con 0x7f32f8101890 2024-04-25T01:25:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.359+0000 7f32ffa4d640 1 -- 172.21.15.27:0/443296947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f819fe50 con 0x7f32f8101890 2024-04-25T01:25:16.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f32ea7fc640 1 -- 172.21.15.27:0/443296947 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f32ec042030 con 0x7f32f8101890 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f3302839740 1 Processor -- start 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f3302839740 1 -- start start 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f3302839740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9440 0x564e2a3d9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f3302839740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3de930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f3302839740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3defa0 0x564e2a3e1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f3302839740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e2a33cf00 con 0x564e2a3d9440 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f3302839740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564e2a2e5750 con 0x564e2a3defa0 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.361+0000 7f3302839740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e2a338ec0 con 0x564e2a3d9e30 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32ea7fc640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32e40a2b90 0x7f32e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9440 0x564e2a3d9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fd7e9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3de930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fd7e9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3de930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:41150/0 (socket says 172.21.15.27:41150) 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fd7e9640 1 -- 172.21.15.27:0/1878545892 learned_addr learned my addr 172.21.15.27:0/1878545892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32e40a2b90 0x7f32e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32ea7fc640 1 -- 172.21.15.27:0/443296947 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f32ec0efee0 con 0x7f32f8101890 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3defa0 0x564e2a3e1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 -- 172.21.15.27:0/1878545892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 msgr2=0x564e2a3de930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3de930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 -- 172.21.15.27:0/1878545892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3defa0 msgr2=0x564e2a3e1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3defa0 0x564e2a3e1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 -- 172.21.15.27:0/1878545892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e2a0512d0 con 0x564e2a3d9440 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fd7e9640 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3de930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9440 0x564e2a3d9860 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f32f0013da0 tx=0x7f32f00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253db70a5881b063 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32fdfea640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32e40a2b90 0x7f32e40a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f32f0010ca0 tx=0x7f32f0015000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.362+0000 7f32e37fe640 1 -- 172.21.15.27:0/1878545892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0021070 con 0x564e2a3d9440 2024-04-25T01:25:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f32e37fe640 1 -- 172.21.15.27:0/1878545892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f0007b40 con 0x564e2a3d9440 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f32e37fe640 1 -- 172.21.15.27:0/1878545892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f001d3f0 con 0x564e2a3d9440 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 -- 172.21.15.27:0/1878545892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9440 msgr2=0x564e2a3d9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9440 0x564e2a3d9860 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f32f0013da0 tx=0x7f32f00107a0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 -- 172.21.15.27:0/1878545892 shutdown_connections 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3defa0 0x564e2a3e1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3de930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 --2- 172.21.15.27:0/1878545892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9440 0x564e2a3d9860 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 -- 172.21.15.27:0/1878545892 >> 172.21.15.27:0/1878545892 conn(0x564e2a3d4ab0 msgr2=0x564e2a3d4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 -- 172.21.15.27:0/1878545892 shutdown_connections 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.363+0000 7f3302839740 1 -- 172.21.15.27:0/1878545892 wait complete. 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 1 -- start start 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 1 Processor -- start 2024-04-25T01:25:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a3e3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3e3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a3eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e2a3e1ef0 con 0x564e2a3defa0 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564e2a2f8cf0 con 0x564e2a3d9440 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e2a3e2070 con 0x564e2a3d9e30 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f3302839740 10 client.0 _ll_register_callbacks cb 0x564e2a2f86e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fdfea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a3eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fdfea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a3eaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43900/0 (socket says 172.21.15.27:43900) 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fdfea640 1 -- 172.21.15.27:0/1012764996 learned_addr learned my addr 172.21.15.27:0/1012764996 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fd7e9640 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a3e3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3e3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fdfea640 1 -- 172.21.15.27:0/1012764996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 msgr2=0x564e2a3e3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fdfea640 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3e3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fdfea640 1 -- 172.21.15.27:0/1012764996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 msgr2=0x564e2a3e3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fdfea640 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a3e3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.364+0000 7f32fdfea640 1 -- 172.21.15.27:0/1012764996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e2a11b6b0 con 0x564e2a3defa0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f32fdfea640 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a3eaeb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f32f000af50 tx=0x7f32f0005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7fdd2b4b24a6f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f32e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f32e0ff9640 1 -- 172.21.15.27:0/1012764996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0021040 con 0x564e2a3defa0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f32e0ff9640 1 -- 172.21.15.27:0/1012764996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f32f000e640 con 0x564e2a3defa0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 1 -- 172.21.15.27:0/1012764996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564e2a34a440 con 0x564e2a3defa0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f32e0ff9640 1 -- 172.21.15.27:0/1012764996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f32f000e840 con 0x564e2a3defa0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 10 client.15450 fetch_fsmap learned FSMap version 24 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 1 -- 172.21.15.27:0/1012764996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e2a3edf00 con 0x564e2a3defa0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f32e0ff9640 1 -- 172.21.15.27:0/1012764996 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564e2a34a440 con 0x564e2a3defa0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f32e0ff9640 1 -- 172.21.15.27:0/1012764996 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f32f002d070 con 0x564e2a3defa0 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 10 client.15450 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:25:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 10 client.15450 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 20 client.15450 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564e2a3f03c0 0x564e2a3f1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 4 client.15450 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.365+0000 7f3302839740 1 -- 172.21.15.27:0/1012764996 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e2a344790 con 0x564e2a3f03c0 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.366+0000 7f32fd7e9640 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564e2a3f03c0 0x564e2a3f1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.366+0000 7f32fd7e9640 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564e2a3f03c0 0x564e2a3f1830 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4173b80922a3a7e server_cookie=1261719cad4e413e in_seq=0 out_seq=0 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.366+0000 7f32e0ff9640 10 client.15450 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f32e0ff9640 1 -- 172.21.15.27:0/1012764996 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e2a344790 con 0x564e2a3f03c0 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f32e0ff9640 10 client.15450 handle_command_reply: tid=0 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 client.15450 shutdown 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 -- 172.21.15.27:0/1012764996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 msgr2=0x564e2a3eaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a3eaeb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f32f000af50 tx=0x7f32f0005a40 comp rx=0 tx=0).stop 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 -- 172.21.15.27:0/1012764996 shutdown_connections 2024-04-25T01:25:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564e2a3f03c0 0x564e2a3f1830 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a3eaeb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a3e3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 --2- 172.21.15.27:0/1012764996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a3e3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f32e0ff9640 0 client.15450 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 -- 172.21.15.27:0/1012764996 >> 172.21.15.27:0/1012764996 conn(0x564e2a3d4ab0 msgr2=0x564e2a3d3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 -- 172.21.15.27:0/1012764996 shutdown_connections 2024-04-25T01:25:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 1 -- 172.21.15.27:0/1012764996 wait complete. 2024-04-25T01:25:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.367+0000 7f3302839740 20 client.15450 trim_cache size 0 max 16384 2024-04-25T01:25:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.385+0000 7f3302839740 1 Processor -- start 2024-04-25T01:25:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.385+0000 7f3302839740 1 -- start start 2024-04-25T01:25:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.385+0000 7f3302839740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a3d4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.385+0000 7f3302839740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9e30 0x564e2a3e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.385+0000 7f3302839740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3defa0 0x564e2a3e3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.385+0000 7f3302839740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e2a3efb70 con 0x564e2a3d9e30 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.385+0000 7f3302839740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564e2a3f0240 con 0x564e2a3d9440 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.385+0000 7f3302839740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e2a3f0910 con 0x564e2a3defa0 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9e30 0x564e2a3e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9e30 0x564e2a3e38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43912/0 (socket says 172.21.15.27:43912) 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 -- 172.21.15.27:0/3734461690 learned_addr learned my addr 172.21.15.27:0/3734461690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 -- 172.21.15.27:0/3734461690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3defa0 msgr2=0x564e2a3e3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fdfea640 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3defa0 0x564e2a3e3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3defa0 0x564e2a3e3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 -- 172.21.15.27:0/3734461690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 msgr2=0x564e2a3d4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a3d4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 -- 172.21.15.27:0/3734461690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e2a3d48e0 con 0x564e2a3d9e30 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fdfea640 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3defa0 0x564e2a3e3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9e30 0x564e2a3e38f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0ed680 tx=0x7f32ec009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b438b31d9c0ea514 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32e2ffd640 1 -- 172.21.15.27:0/3734461690 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec0d5070 con 0x564e2a3d9e30 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32e2ffd640 1 -- 172.21.15.27:0/3734461690 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec0bca90 con 0x564e2a3d9e30 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.386+0000 7f32e2ffd640 1 -- 172.21.15.27:0/3734461690 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec049040 con 0x564e2a3d9e30 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 -- 172.21.15.27:0/3734461690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9e30 msgr2=0x564e2a3e38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9e30 0x564e2a3e38f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0ed680 tx=0x7f32ec009260 comp rx=0 tx=0).stop 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 -- 172.21.15.27:0/3734461690 shutdown_connections 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3defa0 0x564e2a3e3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3d9e30 0x564e2a3e38f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 --2- 172.21.15.27:0/3734461690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a3d4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 -- 172.21.15.27:0/3734461690 >> 172.21.15.27:0/3734461690 conn(0x564e2a3d4ab0 msgr2=0x564e2a3ef310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 -- 172.21.15.27:0/3734461690 shutdown_connections 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 -- 172.21.15.27:0/3734461690 wait complete. 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 -- start start 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.387+0000 7f3302839740 1 Processor -- start 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a4525a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a452ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a4594c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e2a3f16b0 con 0x564e2a3defa0 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564e2a3f0fe0 con 0x564e2a3d9440 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e2a3f1d80 con 0x564e2a3d9e30 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 10 client.0 _ll_register_callbacks cb 0x7f32f0013a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a4594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a4594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43920/0 (socket says 172.21.15.27:43920) 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 -- 172.21.15.27:0/3590188385 learned_addr learned my addr 172.21.15.27:0/3590188385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a452ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fd7e9640 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a4525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 -- 172.21.15.27:0/3590188385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 msgr2=0x564e2a452ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a452ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 -- 172.21.15.27:0/3590188385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 msgr2=0x564e2a4525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a4525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 -- 172.21.15.27:0/3590188385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e2a459ad0 con 0x564e2a3defa0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fcfe8640 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a452ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32fdfea640 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a4594c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f32f0007ad0 tx=0x7f32f0007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a710525aa4ded637 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32e0ff9640 1 -- 172.21.15.27:0/3590188385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0021040 con 0x564e2a3defa0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32e0ff9640 1 -- 172.21.15.27:0/3590188385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f32f0003b80 con 0x564e2a3defa0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 1 -- 172.21.15.27:0/3590188385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32ec0bca90 con 0x564e2a3defa0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f32e0ff9640 1 -- 172.21.15.27:0/3590188385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f32f0044050 con 0x564e2a3defa0 2024-04-25T01:25:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 10 client.15456 fetch_fsmap learned FSMap version 24 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.388+0000 7f3302839740 1 -- 172.21.15.27:0/3590188385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e2a45ad40 con 0x564e2a3defa0 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f32e0ff9640 1 -- 172.21.15.27:0/3590188385 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec0bca90 con 0x564e2a3defa0 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f32e0ff9640 1 -- 172.21.15.27:0/3590188385 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f32f002d070 con 0x564e2a3defa0 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f3302839740 10 client.15456 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f3302839740 10 client.15456 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f3302839740 20 client.15456 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f3302839740 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564e2a45f920 0x564e2a45e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f3302839740 4 client.15456 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f3302839740 1 -- 172.21.15.27:0/3590188385 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "status"}) -- 0x564e2a2c2b80 con 0x564e2a45f920 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.389+0000 7f32fd7e9640 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564e2a45f920 0x564e2a45e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f32fd7e9640 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564e2a45f920 0x564e2a45e630 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fffa67af21749ff0 server_cookie=f20d3f65bf494fe8 in_seq=0 out_seq=0 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f32e0ff9640 10 client.15456 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f32e0ff9640 1 -- 172.21.15.27:0/3590188385 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564e2a2c2b80 con 0x564e2a45f920 2024-04-25T01:25:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f32e0ff9640 10 client.15456 handle_command_reply: tid=0 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 client.15456 shutdown 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 -- 172.21.15.27:0/3590188385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 msgr2=0x564e2a4594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a4594c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f32f0007ad0 tx=0x7f32f0007870 comp rx=0 tx=0).stop 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 -- 172.21.15.27:0/3590188385 shutdown_connections 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564e2a45f920 0x564e2a45e630 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e2a3defa0 0x564e2a4594c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e2a3d9e30 0x564e2a452ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 --2- 172.21.15.27:0/3590188385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564e2a3d9440 0x564e2a4525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f32e0ff9640 0 client.15456 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 -- 172.21.15.27:0/3590188385 >> 172.21.15.27:0/3590188385 conn(0x564e2a3d4ab0 msgr2=0x564e2a3db330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 -- 172.21.15.27:0/3590188385 shutdown_connections 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 1 -- 172.21.15.27:0/3590188385 wait complete. 2024-04-25T01:25:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.390+0000 7f3302839740 20 client.15456 trim_cache size 0 max 16384 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "1469ec2e-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24521, 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 138.56746888800001, 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 24, 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 93, 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T01:25:16.377 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 143.96515236400001 2024-04-25T01:25:16.378 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:25:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 -- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32e40a2b90 msgr2=0x7f32e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32e40a2b90 0x7f32e40a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f32f0010ca0 tx=0x7f32f0015000 comp rx=0 tx=0).stop 2024-04-25T01:25:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 -- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 msgr2=0x7f32f8197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 0x7f32f8197fa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009230 tx=0x7f32ec004360 comp rx=0 tx=0).stop 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 -- 172.21.15.27:0/443296947 shutdown_connections 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f32e40a2b90 0x7f32e40a5050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8102250 0x7f32f819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f8101890 0x7f32f8197fa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 --2- 172.21.15.27:0/443296947 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32f81006a0 0x7f32f8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 -- 172.21.15.27:0/443296947 >> 172.21.15.27:0/443296947 conn(0x7f32f80fbc30 msgr2=0x7f32f8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 -- 172.21.15.27:0/443296947 shutdown_connections 2024-04-25T01:25:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.391+0000 7f32ffa4d640 1 -- 172.21.15.27:0/443296947 wait complete. 2024-04-25T01:25:16.381 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T01:25:16.391 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T01:25:16.392 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T01:25:16.392 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T01:25:16.392 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T01:25:16.393 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:25:16.393 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:25:16.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T01:25:16.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3533013313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e4101890 msgr2=0x7f83e4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3533013313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e4101890 0x7f83e4101d10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009900 tx=0x7f83cc02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3533013313 shutdown_connections 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3533013313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83e4102250 0x7f83e4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3533013313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e4101890 0x7f83e4101d10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3533013313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83e41006a0 0x7f83e4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3533013313 >> 172.21.15.27:0/3533013313 conn(0x7f83e40fbc10 msgr2=0x7f83e40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3533013313 shutdown_connections 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.515+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3533013313 wait complete. 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e8fd5640 1 Processor -- start 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e8fd5640 1 -- start start 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e8fd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e41006a0 0x7f83e4197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e8fd5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83e4101890 0x7f83e4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e8fd5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83e4102250 0x7f83e419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e8fd5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f83e410d440 con 0x7f83e41006a0 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e8fd5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f83e410d2c0 con 0x7f83e4101890 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e8fd5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83e410d5c0 con 0x7f83e4102250 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e41006a0 0x7f83e4197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e41006a0 0x7f83e4197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43928/0 (socket says 172.21.15.27:43928) 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e259c640 1 -- 172.21.15.27:0/3255406899 learned_addr learned my addr 172.21.15.27:0/3255406899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83dbfff640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83e4101890 0x7f83e4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e259c640 1 -- 172.21.15.27:0/3255406899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83e4102250 msgr2=0x7f83e419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e259c640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83e4102250 0x7f83e419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e259c640 1 -- 172.21.15.27:0/3255406899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83e4101890 msgr2=0x7f83e4197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e259c640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83e4101890 0x7f83e4197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.516+0000 7f83e259c640 1 -- 172.21.15.27:0/3255406899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e419f610 con 0x7f83e41006a0 2024-04-25T01:25:16.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.517+0000 7f83e259c640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e41006a0 0x7f83e4197a40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f83d400c970 tx=0x7f83d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.517+0000 7f83db7fe640 1 -- 172.21.15.27:0/3255406899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4004220 con 0x7f83e41006a0 2024-04-25T01:25:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.517+0000 7f83db7fe640 1 -- 172.21.15.27:0/3255406899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d40043c0 con 0x7f83e41006a0 2024-04-25T01:25:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.517+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e419f900 con 0x7f83e41006a0 2024-04-25T01:25:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.517+0000 7f83db7fe640 1 -- 172.21.15.27:0/3255406899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d400f450 con 0x7f83e41006a0 2024-04-25T01:25:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.517+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e419fe70 con 0x7f83e41006a0 2024-04-25T01:25:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.518+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e41079c0 con 0x7f83e41006a0 2024-04-25T01:25:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.519+0000 7f83db7fe640 1 -- 172.21.15.27:0/3255406899 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f83d40079b0 con 0x7f83e41006a0 2024-04-25T01:25:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.519+0000 7f83db7fe640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f83d00a2cb0 0x7f83d00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.520+0000 7f83dbfff640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f83d00a2cb0 0x7f83d00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.520+0000 7f83db7fe640 1 -- 172.21.15.27:0/3255406899 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f83d40cf1c0 con 0x7f83e41006a0 2024-04-25T01:25:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.520+0000 7f83dbfff640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f83d00a2cb0 0x7f83d00a5170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc0315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.524+0000 7f83db7fe640 1 -- 172.21.15.27:0/3255406899 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f83d4014030 con 0x7f83e41006a0 2024-04-25T01:25:16.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:16.761+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f83e40623e0 con 0x7f83e41006a0 2024-04-25T01:25:16.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:16 smithi027 ceph-mon[29197]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:25:16.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:16 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2786557713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:16.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:16 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1710727038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:17.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:16 smithi153 ceph-mon[32621]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:25:17.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:16 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2786557713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:17.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:16 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1710727038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:17.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:16 smithi139 ceph-mon[32760]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:25:17.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:16 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2786557713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:17.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:16 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1710727038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:17.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.568+0000 7f83db7fe640 1 -- 172.21.15.27:0/3255406899 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f83d409bd70 con 0x7f83e41006a0 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f83d00a2cb0 msgr2=0x7f83d00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f83d00a2cb0 0x7f83d00a5170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc0315d0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e41006a0 msgr2=0x7f83e4197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e41006a0 0x7f83e4197a40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f83d400c970 tx=0x7f83d400ce30 comp rx=0 tx=0).stop 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 shutdown_connections 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f83d00a2cb0 0x7f83d00a5170 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83e4102250 0x7f83e419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83e4101890 0x7f83e4197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 --2- 172.21.15.27:0/3255406899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e41006a0 0x7f83e4197a40 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 >> 172.21.15.27:0/3255406899 conn(0x7f83e40fbc10 msgr2=0x7f83e4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 shutdown_connections 2024-04-25T01:25:17.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.570+0000 7f83e8fd5640 1 -- 172.21.15.27:0/3255406899 wait complete. 2024-04-25T01:25:17.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T01:25:17.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.709+0000 7f858caa5640 1 -- 172.21.15.27:0/330710175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 msgr2=0x7f8588100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.709+0000 7f858caa5640 1 --2- 172.21.15.27:0/330710175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 0x7f8588100ac0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8578009900 tx=0x7f857802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:17.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.709+0000 7f858caa5640 1 -- 172.21.15.27:0/330710175 shutdown_connections 2024-04-25T01:25:17.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.709+0000 7f858caa5640 1 --2- 172.21.15.27:0/330710175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8588102250 0x7f8588108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.709+0000 7f858caa5640 1 --2- 172.21.15.27:0/330710175 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8588101890 0x7f8588101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.709+0000 7f858caa5640 1 --2- 172.21.15.27:0/330710175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 0x7f8588100ac0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.709+0000 7f858caa5640 1 -- 172.21.15.27:0/330710175 >> 172.21.15.27:0/330710175 conn(0x7f85880fbc10 msgr2=0x7f85880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.710+0000 7f858caa5640 1 -- 172.21.15.27:0/330710175 shutdown_connections 2024-04-25T01:25:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.710+0000 7f858caa5640 1 -- 172.21.15.27:0/330710175 wait complete. 2024-04-25T01:25:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.710+0000 7f858caa5640 1 Processor -- start 2024-04-25T01:25:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.710+0000 7f858caa5640 1 -- start start 2024-04-25T01:25:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.710+0000 7f858caa5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 0x7f8588196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.710+0000 7f858caa5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8588101890 0x7f8588196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.710+0000 7f858caa5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8588102250 0x7f858819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.710+0000 7f858caa5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f858810c2f0 con 0x7f85881006a0 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f858caa5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f858810c170 con 0x7f8588102250 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f858caa5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f858810c470 con 0x7f8588101890 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8588101890 0x7f8588196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f858659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 0x7f8588196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8588101890 0x7f8588196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:41220/0 (socket says 172.21.15.27:41220) 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 -- 172.21.15.27:0/2102027539 learned_addr learned my addr 172.21.15.27:0/2102027539 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f858659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 0x7f8588196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43958/0 (socket says 172.21.15.27:43958) 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8586d9d640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8588102250 0x7f858819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 -- 172.21.15.27:0/2102027539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8588102250 msgr2=0x7f858819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8588102250 0x7f858819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 -- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 msgr2=0x7f8588196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 0x7f8588196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 -- 172.21.15.27:0/2102027539 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858819e5b0 con 0x7f8588101890 2024-04-25T01:25:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f8585d9b640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8588101890 0x7f8588196ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f857c00d950 tx=0x7f857c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f85777fe640 1 -- 172.21.15.27:0/2102027539 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c017030 con 0x7f8588101890 2024-04-25T01:25:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f858819e8a0 con 0x7f8588101890 2024-04-25T01:25:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f85777fe640 1 -- 172.21.15.27:0/2102027539 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f857c004020 con 0x7f8588101890 2024-04-25T01:25:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.711+0000 7f85777fe640 1 -- 172.21.15.27:0/2102027539 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c0047a0 con 0x7f8588101890 2024-04-25T01:25:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.712+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f858819ee40 con 0x7f8588101890 2024-04-25T01:25:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.712+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8588107a10 con 0x7f8588101890 2024-04-25T01:25:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.714+0000 7f85777fe640 1 -- 172.21.15.27:0/2102027539 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f857c0041d0 con 0x7f8588101890 2024-04-25T01:25:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.714+0000 7f85777fe640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85700a2c80 0x7f85700a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.714+0000 7f858659c640 1 -- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85700a2c80 msgr2=0x7f85700a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.714+0000 7f85777fe640 1 -- 172.21.15.27:0/2102027539 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f857c0cf0f0 con 0x7f8588101890 2024-04-25T01:25:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.714+0000 7f858659c640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85700a2c80 0x7f85700a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:25:17.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.718+0000 7f85777fe640 1 -- 172.21.15.27:0/2102027539 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f857c09bc40 con 0x7f8588101890 2024-04-25T01:25:17.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.906+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f8588062340 con 0x7f8588101890 2024-04-25T01:25:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.912+0000 7f85777fe640 1 -- 172.21.15.27:0/2102027539 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f857c09b540 con 0x7f8588101890 2024-04-25T01:25:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85700a2c80 msgr2=0x7f85700a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858659c640 1 -- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85700a2c80 msgr2=0x7f85700a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858659c640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85700a2c80 0x7f85700a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85700a2c80 0x7f85700a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8588101890 msgr2=0x7f8588196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8588101890 0x7f8588196ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f857c00d950 tx=0x7f857c00de10 comp rx=0 tx=0).stop 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 shutdown_connections 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f85700a2c80 0x7f85700a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8588102250 0x7f858819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8588101890 0x7f8588196ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 --2- 172.21.15.27:0/2102027539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85881006a0 0x7f8588196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 >> 172.21.15.27:0/2102027539 conn(0x7f85880fbc10 msgr2=0x7f8588104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 shutdown_connections 2024-04-25T01:25:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:17.915+0000 7f858caa5640 1 -- 172.21.15.27:0/2102027539 wait complete. 2024-04-25T01:25:17.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:17 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3255406899' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:25:17.906 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:17 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: ignoring --setuser ceph since I am not root 2024-04-25T01:25:17.906 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:17 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: ignoring --setgroup ceph since I am not root 2024-04-25T01:25:17.906 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:17 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:17.770+0000 7f38aba66200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:25:17.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-25T01:25:17.915 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:17 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3255406899' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:25:17.916 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: ignoring --setuser ceph since I am not root 2024-04-25T01:25:17.916 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: ignoring --setgroup ceph since I am not root 2024-04-25T01:25:17.916 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:17 smithi139 ceph-mgr[34147]: -- 172.21.15.139:0/4157333697 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55847d82a5a0 con 0x55847d7ff000 2024-04-25T01:25:17.916 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:17.751+0000 7f4c85b8f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:25:17.916 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:17.936+0000 7f4c85b8f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:25:17.917 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: ignoring --setuser ceph since I am not root 2024-04-25T01:25:17.917 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: ignoring --setgroup ceph since I am not root 2024-04-25T01:25:17.917 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:17 smithi153 ceph-mgr[34474]: -- 172.21.15.153:0/644991473 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560db79fc5a0 con 0x560db79d1000 2024-04-25T01:25:17.917 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:17.751+0000 7f04e9e32200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:25:17.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:17 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3255406899' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:25:18.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 -- 172.21.15.27:0/3682788588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff158102250 msgr2=0x7ff158108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 --2- 172.21.15.27:0/3682788588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff158102250 0x7ff158108ae0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff150009ca0 tx=0x7ff15002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 -- 172.21.15.27:0/3682788588 shutdown_connections 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 --2- 172.21.15.27:0/3682788588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff158102250 0x7ff158108ae0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 --2- 172.21.15.27:0/3682788588 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff158101890 0x7ff158101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 --2- 172.21.15.27:0/3682788588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1581006a0 0x7ff158100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 -- 172.21.15.27:0/3682788588 >> 172.21.15.27:0/3682788588 conn(0x7ff1580fbc10 msgr2=0x7ff1580fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 -- 172.21.15.27:0/3682788588 shutdown_connections 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.031+0000 7ff15f714640 1 -- 172.21.15.27:0/3682788588 wait complete. 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15f714640 1 Processor -- start 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15f714640 1 -- start start 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15f714640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1581006a0 0x7ff158104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15f714640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff158101890 0x7ff158102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15f714640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff158102250 0x7ff1581031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15f714640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff15810c300 con 0x7ff1581006a0 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15f714640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff15810c180 con 0x7ff158101890 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15f714640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff15810c480 con 0x7ff158102250 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1581006a0 0x7ff158104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1581006a0 0x7ff158104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52574/0 (socket says 172.21.15.27:52574) 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 -- 172.21.15.27:0/2732571582 learned_addr learned my addr 172.21.15.27:0/2732571582 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15ccaf640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff158101890 0x7ff158102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15dcb1640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff158102250 0x7ff1581031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 -- 172.21.15.27:0/2732571582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff158102250 msgr2=0x7ff1581031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff158102250 0x7ff1581031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 -- 172.21.15.27:0/2732571582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff158101890 msgr2=0x7ff158102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff158101890 0x7ff158102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 -- 172.21.15.27:0/2732571582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff158103a40 con 0x7ff1581006a0 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15ccaf640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff158101890 0x7ff158102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.032+0000 7ff15d4b0640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1581006a0 0x7ff158104630 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff15400e9c0 tx=0x7ff15400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.033+0000 7ff14e7fc640 1 -- 172.21.15.27:0/2732571582 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15400cd80 con 0x7ff1581006a0 2024-04-25T01:25:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.033+0000 7ff14e7fc640 1 -- 172.21.15.27:0/2732571582 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1540035f0 con 0x7ff1581006a0 2024-04-25T01:25:18.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.033+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1581a1ac0 con 0x7ff1581006a0 2024-04-25T01:25:18.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.033+0000 7ff14e7fc640 1 -- 172.21.15.27:0/2732571582 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154010690 con 0x7ff1581006a0 2024-04-25T01:25:18.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.033+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1581a2010 con 0x7ff1581006a0 2024-04-25T01:25:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.034+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1581a2240 con 0x7ff1581006a0 2024-04-25T01:25:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.035+0000 7ff14e7fc640 1 -- 172.21.15.27:0/2732571582 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7ff1540037a0 con 0x7ff1581006a0 2024-04-25T01:25:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.036+0000 7ff14e7fc640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff1440a2c80 0x7ff1440a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.036+0000 7ff15ccaf640 1 -- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff1440a2c80 msgr2=0x7ff1440a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.036+0000 7ff14e7fc640 1 -- 172.21.15.27:0/2732571582 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7ff1540cf890 con 0x7ff1581006a0 2024-04-25T01:25:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.036+0000 7ff15ccaf640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff1440a2c80 0x7ff1440a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:25:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.040+0000 7ff14e7fc640 1 -- 172.21.15.27:0/2732571582 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff15409c440 con 0x7ff1581006a0 2024-04-25T01:25:18.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:17 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:17.968+0000 7f38aba66200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:25:18.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:18.108+0000 7f38aba66200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:25:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.237+0000 7ff15ccaf640 1 -- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff1440a2c80 msgr2=0x7ff1440a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.237+0000 7ff15ccaf640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff1440a2c80 0x7ff1440a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:25:18.275 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:17.937+0000 7f04e9e32200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:25:18.275 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:18.042+0000 7f04e9e32200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:25:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.297+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7ff1581a2640 con 0x7ff1581006a0 2024-04-25T01:25:18.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.301+0000 7ff14e7fc640 1 -- 172.21.15.27:0/2732571582 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7ff15409bd40 con 0x7ff1581006a0 2024-04-25T01:25:18.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff1440a2c80 msgr2=0x7ff1440a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:25:18.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff1440a2c80 0x7ff1440a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1581006a0 msgr2=0x7ff158104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1581006a0 0x7ff158104630 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff15400e9c0 tx=0x7ff15400ee80 comp rx=0 tx=0).stop 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 shutdown_connections 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7ff1440a2c80 0x7ff1440a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff158102250 0x7ff1581031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff158101890 0x7ff158102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 --2- 172.21.15.27:0/2732571582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1581006a0 0x7ff158104630 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 >> 172.21.15.27:0/2732571582 conn(0x7ff1580fbc10 msgr2=0x7ff1580fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 shutdown_connections 2024-04-25T01:25:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:18.304+0000 7ff15f714640 1 -- 172.21.15.27:0/2732571582 wait complete. 2024-04-25T01:25:18.292 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:18 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:18.039+0000 7f4c85b8f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:25:18.295 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T01:25:18.309 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T01:25:18.310 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T01:25:18.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:18.310 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T01:25:18.327 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T01:25:18.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:18.327 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:25:18.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:18.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-04-25T01:25:18.483 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:18.456+0000 7f38aba66200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:25:18.488 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-25T01:25:18.488 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:25:18.488 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:25:18.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T01:25:18.748 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:18.775 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:18.351+0000 7f04e9e32200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:25:18.776 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:18.436+0000 7f04e9e32200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:25:18.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:18 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3255406899' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:25:18.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:18 smithi153 ceph-mon[32621]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:25:18.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:18 smithi153 ceph-mon[32621]: from='client.? ' entity='client.admin' 2024-04-25T01:25:18.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:18 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2732571582' entity='client.admin' 2024-04-25T01:25:18.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:18 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3255406899' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:25:18.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:18 smithi139 ceph-mon[32760]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:25:18.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:18 smithi139 ceph-mon[32760]: from='client.? ' entity='client.admin' 2024-04-25T01:25:18.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:18 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2732571582' entity='client.admin' 2024-04-25T01:25:18.792 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:18 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:18.347+0000 7f4c85b8f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:25:18.792 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:18 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:18.432+0000 7f4c85b8f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:25:18.872 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:18.553+0000 7f38aba66200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:25:18.872 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:18 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3255406899' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:25:18.872 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:18 smithi027 ceph-mon[29197]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:25:18.872 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:18 smithi027 ceph-mon[29197]: from='client.? ' entity='client.admin' 2024-04-25T01:25:18.873 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:18 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2732571582' entity='client.admin' 2024-04-25T01:25:19.933 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:19 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:19.612+0000 7f4c85b8f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:25:19.933 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:19 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:19.781+0000 7f4c85b8f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:25:19.943 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:19 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:19.828+0000 7f38aba66200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:25:19.944 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:19.620+0000 7f04e9e32200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:25:19.944 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:19.792+0000 7f04e9e32200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:25:20.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:20.009+0000 7f38aba66200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:25:20.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:20.193+0000 7f38aba66200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:25:20.221 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:19 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:19.953+0000 7f4c85b8f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:25:20.235 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:19.963+0000 7f04e9e32200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:25:20.515 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:20.241+0000 7f4c85b8f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:25:20.516 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:20.331+0000 7f4c85b8f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:25:20.516 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:20.535+0000 7f4c85b8f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:25:20.525 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:20.255+0000 7f04e9e32200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:25:20.525 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:20.348+0000 7f04e9e32200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:25:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.547+0000 7f11ecaef700 1 -- 172.21.15.27:0/2327166507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101130 msgr2=0x7f11e81015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.547+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2327166507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101130 0x7f11e81015b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009990 tx=0x7f11dc009ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 -- 172.21.15.27:0/2327166507 shutdown_connections 2024-04-25T01:25:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2327166507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e8101af0 0x7f11e810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2327166507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11e81071a0 0x7f11e8107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2327166507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101130 0x7f11e81015b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 -- 172.21.15.27:0/2327166507 >> 172.21.15.27:0/2327166507 conn(0x7f11e80780e0 msgr2=0x7f11e80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 -- 172.21.15.27:0/2327166507 shutdown_connections 2024-04-25T01:25:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 -- 172.21.15.27:0/2327166507 wait complete. 2024-04-25T01:25:20.529 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:20.515+0000 7f38aba66200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:25:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 Processor -- start 2024-04-25T01:25:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.548+0000 7f11ecaef700 1 -- start start 2024-04-25T01:25:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11ecaef700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11e8101130 0x7f11e8102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11ecaef700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101af0 0x7f11e8103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11ecaef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81071a0 0x7f11e8103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11ecaef700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8111490 con 0x7f11e8101af0 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11ecaef700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8111310 con 0x7f11e8101130 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11e8101130 0x7f11e8102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11ecaef700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8111610 con 0x7f11e81071a0 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101af0 0x7f11e8103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11e8101130 0x7f11e8102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:41232/0 (socket says 172.21.15.27:41232) 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e659c700 1 -- 172.21.15.27:0/2172322535 learned_addr learned my addr 172.21.15.27:0/2172322535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e6d9d700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81071a0 0x7f11e8103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e6d9d700 1 -- 172.21.15.27:0/2172322535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11e8101130 msgr2=0x7f11e8102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e6d9d700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11e8101130 0x7f11e8102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e6d9d700 1 -- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101af0 msgr2=0x7f11e8103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e6d9d700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101af0 0x7f11e8103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e6d9d700 1 -- 172.21.15.27:0/2172322535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e81a6f30 con 0x7f11e81071a0 2024-04-25T01:25:20.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.549+0000 7f11e5d9b700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101af0 0x7f11e8103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:20.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.550+0000 7f11e6d9d700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81071a0 0x7f11e8103890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f11e000efc0 tx=0x7f11e000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:20.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.550+0000 7f11d77fe700 1 -- 172.21.15.27:0/2172322535 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e00092e0 con 0x7f11e81071a0 2024-04-25T01:25:20.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.550+0000 7f11d77fe700 1 -- 172.21.15.27:0/2172322535 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0004bb0 con 0x7f11e81071a0 2024-04-25T01:25:20.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.550+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e81a71c0 con 0x7f11e81071a0 2024-04-25T01:25:20.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.550+0000 7f11d77fe700 1 -- 172.21.15.27:0/2172322535 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0004280 con 0x7f11e81071a0 2024-04-25T01:25:20.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.550+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e81a7690 con 0x7f11e81071a0 2024-04-25T01:25:20.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.551+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e8068f20 con 0x7f11e81071a0 2024-04-25T01:25:20.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.552+0000 7f11d77fe700 1 -- 172.21.15.27:0/2172322535 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f11e001b030 con 0x7f11e81071a0 2024-04-25T01:25:20.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.553+0000 7f11d77fe700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f11d00a2cf0 0x7f11d00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:20.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.553+0000 7f11e659c700 1 -- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f11d00a2cf0 msgr2=0x7f11d00a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:20.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.553+0000 7f11d77fe700 1 -- 172.21.15.27:0/2172322535 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f11e00cef40 con 0x7f11e81071a0 2024-04-25T01:25:20.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.553+0000 7f11e659c700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f11d00a2cf0 0x7f11d00a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:25:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.557+0000 7f11d77fe700 1 -- 172.21.15.27:0/2172322535 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f11e0022020 con 0x7f11e81071a0 2024-04-25T01:25:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.754+0000 7f11e659c700 1 -- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f11d00a2cf0 msgr2=0x7f11d00a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.754+0000 7f11e659c700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f11d00a2cf0 0x7f11d00a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:25:20.792 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:20.728+0000 7f4c85b8f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:25:20.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.812+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f11e81a79b0 con 0x7f11e81071a0 2024-04-25T01:25:20.793 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:20.617+0000 7f38aba66200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:25:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.812+0000 7f11d77fe700 1 -- 172.21.15.27:0/2172322535 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f11e0010040 con 0x7f11e81071a0 2024-04-25T01:25:20.793 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f11d00a2cf0 msgr2=0x7f11d00a51b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f11d00a2cf0 0x7f11d00a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81071a0 msgr2=0x7f11e8103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81071a0 0x7f11e8103890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f11e000efc0 tx=0x7f11e000c3a0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 shutdown_connections 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7f11d00a2cf0 0x7f11d00a51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81071a0 0x7f11e8103890 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11e8101130 0x7f11e8102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 --2- 172.21.15.27:0/2172322535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8101af0 0x7f11e8103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.814+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 >> 172.21.15.27:0/2172322535 conn(0x7f11e80780e0 msgr2=0x7f11e81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.815+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 shutdown_connections 2024-04-25T01:25:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:20.815+0000 7f11ecaef700 1 -- 172.21.15.27:0/2172322535 wait complete. 2024-04-25T01:25:20.845 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:20.552+0000 7f04e9e32200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:25:20.845 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:20.744+0000 7f04e9e32200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:25:21.119 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:20 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2172322535' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:25:21.120 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:20.832+0000 7f38aba66200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:25:21.120 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:21 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:21.040+0000 7f38aba66200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:25:21.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:20 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2172322535' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:25:21.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:20 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2172322535' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:25:21.456 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:25:21.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:21.457 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-04-25T01:25:21.473 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:25:21.473 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-04-25T01:25:21.474 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42085sec preferred_lft 42085sec 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-04-25T01:25:21.475 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-04-25T01:25:21.475 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:25:21.476 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:25:21.476 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:25:21.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:25:21.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-04-25T01:25:21.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-04-25T01:25:21.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:25:21.476 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:25:21.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:25:21.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:21.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:21.622 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:25:21.646 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-04-25T01:25:21.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:21.648 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-04-25T01:25:21.695 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:21 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:21.682+0000 7f38aba66200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:25:21.704 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-04-25T01:25:21.704 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:25:21.704 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T01:25:21.704 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T01:25:21.705 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:25:21.705 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:25:21.705 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:25:21.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T01:25:21.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T01:25:21.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T01:25:21.705 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:25:21.732 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:21.333+0000 7f4c85b8f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:25:21.750 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:21.349+0000 7f04e9e32200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:25:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:25:21.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:21.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:21.835 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T01:25:21.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:21.851 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-04-25T01:25:21.907 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:25:21.907 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:25:21.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:25:21.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T01:25:21.907 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:25:21.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:25:22.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:22.025 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:21.770+0000 7f04e9e32200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:25:22.025 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:21.870+0000 7f04e9e32200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:25:22.026 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T01:25:22.026 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:25:22.027 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:25:22.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T01:25:22.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:25:22.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T01:25:22.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T01:25:22.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T01:25:22.027 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:25:22.033 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:21.752+0000 7f4c85b8f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:25:22.033 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:21.851+0000 7f4c85b8f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:25:22.033 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:22.053+0000 7f4c85b8f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:25:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:25:22.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:22.142 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:25:22.142 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T01:25:22.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-04-25T01:25:22.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:25:22.142 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T01:25:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:25:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:22.220 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:25:22.221 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:25:22.235 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:25:22.236 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:25:22.236 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:25:22.291 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T01:25:22.291 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T01:25:22.291 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/2c96bf5e-0240-4a00-9816-53b5e1baa41e 2024-04-25T01:25:22.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:22.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/2c96bf5e-0240-4a00-9816-53b5e1baa41e /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-25T01:25:22.352 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:22.153+0000 7f38aba66200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:25:22.352 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:22.270+0000 7f38aba66200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:25:22.374 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-25T01:25:22.374 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-25T01:25:22.374 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-04-25T01:25:22.488 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-25T01:25:22.488 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-25T01:25:22.488 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-04-25T01:25:22.488 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.139:3300,172.21.15.153:3300" 2024-04-25T01:25:22.488 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:3300,172.21.15.139:3300,172.21.15.153:3300:/volumes/qa/sv_1/2c96bf5e-0240-4a00-9816-53b5e1baa41e 2024-04-25T01:25:22.488 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=1469ec2e-02a1-11ef-bc93-c7b262605968" will pass to kernel 2024-04-25T01:25:22.489 INFO:teuthology.orchestra.run.smithi027.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:25:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:25:22.490 INFO:teuthology.orchestra.run.smithi027.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:25:22.490 INFO:teuthology.orchestra.run.smithi027.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:25:22.490 INFO:teuthology.orchestra.run.smithi027.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:25:22.491 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:25:22.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:22.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:25:22.525 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:22.072+0000 7f04e9e32200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:25:22.563 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:25:22.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:25:22.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:22.664 INFO:teuthology.orchestra.run.smithi027.stdout:client25091 2024-04-25T01:25:22.665 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:25:22.665 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:25:22.681 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:25:22.695 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:22.487+0000 7f38aba66200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:25:22.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:25:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:22.753 INFO:teuthology.orchestra.run.smithi027.stdout:client25091 2024-04-25T01:25:22.753 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:25:22.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:25:22.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:22.811 INFO:teuthology.orchestra.run.smithi027.stdout:client25091 2024-04-25T01:25:22.811 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968.client25091/status') 2024-04-25T01:25:22.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:25:22.870 INFO:teuthology.orchestra.run.smithi027.stdout:instance: client.25091 (3)172.21.15.27:0/1181368763 2024-04-25T01:25:22.871 INFO:teuthology.orchestra.run.smithi027.stdout:blocklisted: false 2024-04-25T01:25:22.871 INFO:teuthology.orchestra.run.smithi027.stderr:0+1 records in 2024-04-25T01:25:22.871 INFO:teuthology.orchestra.run.smithi027.stderr:0+1 records out 2024-04-25T01:25:22.871 INFO:teuthology.orchestra.run.smithi027.stderr:71 bytes copied, 2.4479e-05 s, 2.9 MB/s 2024-04-25T01:25:22.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:25:22.878 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:25:22.887 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:25:22.887 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:25:22.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-25T01:25:22.888 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:22.577+0000 7f4c85b8f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:25:22.888 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:22.668+0000 7f4c85b8f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:25:22.909 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:22.596+0000 7f04e9e32200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:25:22.909 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:22.688+0000 7f04e9e32200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:25:22.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 -- 172.21.15.27:0/296937954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60104bb0 msgr2=0x7feb60106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 --2- 172.21.15.27:0/296937954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60104bb0 0x7feb60106fa0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7feb48009900 tx=0x7feb4802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 -- 172.21.15.27:0/296937954 shutdown_connections 2024-04-25T01:25:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 --2- 172.21.15.27:0/296937954 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb601074e0 0x7feb601098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 --2- 172.21.15.27:0/296937954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60104bb0 0x7feb60106fa0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 --2- 172.21.15.27:0/296937954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb60102280 0x7feb60104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 -- 172.21.15.27:0/296937954 >> 172.21.15.27:0/296937954 conn(0x7feb600fbc10 msgr2=0x7feb600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 -- 172.21.15.27:0/296937954 shutdown_connections 2024-04-25T01:25:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.011+0000 7feb64a64640 1 -- 172.21.15.27:0/296937954 wait complete. 2024-04-25T01:25:22.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb64a64640 1 Processor -- start 2024-04-25T01:25:22.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb64a64640 1 -- start start 2024-04-25T01:25:22.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb64a64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60102280 0x7feb60199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb64a64640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb60104bb0 0x7feb6019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb64a64640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb601074e0 0x7feb601a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb64a64640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010d410 con 0x7feb60102280 2024-04-25T01:25:22.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb64a64640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010d290 con 0x7feb60104bb0 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb64a64640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010d590 con 0x7feb601074e0 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60102280 0x7feb60199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60102280 0x7feb60199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52616/0 (socket says 172.21.15.27:52616) 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 -- 172.21.15.27:0/1821175084 learned_addr learned my addr 172.21.15.27:0/1821175084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb57fff640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb60104bb0 0x7feb6019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5ed9d640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb601074e0 0x7feb601a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 -- 172.21.15.27:0/1821175084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb601074e0 msgr2=0x7feb601a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb601074e0 0x7feb601a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 -- 172.21.15.27:0/1821175084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb60104bb0 msgr2=0x7feb6019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb60104bb0 0x7feb6019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 -- 172.21.15.27:0/1821175084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb601a1890 con 0x7feb60102280 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.012+0000 7feb5e59c640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60102280 0x7feb60199c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7feb5000e9f0 tx=0x7feb5000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.013+0000 7feb577fe640 1 -- 172.21.15.27:0/1821175084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5000cd80 con 0x7feb60102280 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.013+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb601a1b80 con 0x7feb60102280 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.013+0000 7feb577fe640 1 -- 172.21.15.27:0/1821175084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb50005500 con 0x7feb60102280 2024-04-25T01:25:22.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.013+0000 7feb577fe640 1 -- 172.21.15.27:0/1821175084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50010690 con 0x7feb60102280 2024-04-25T01:25:22.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.013+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb601a2080 con 0x7feb60102280 2024-04-25T01:25:22.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.014+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb60108fd0 con 0x7feb60102280 2024-04-25T01:25:22.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.015+0000 7feb577fe640 1 -- 172.21.15.27:0/1821175084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7feb500046c0 con 0x7feb60102280 2024-04-25T01:25:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.016+0000 7feb577fe640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7feb4c0a2c80 0x7feb4c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.016+0000 7feb57fff640 1 -- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7feb4c0a2c80 msgr2=0x7feb4c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.017+0000 7feb577fe640 1 -- 172.21.15.27:0/1821175084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7feb500cf900 con 0x7feb60102280 2024-04-25T01:25:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.017+0000 7feb57fff640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7feb4c0a2c80 0x7feb4c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:25:23.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.021+0000 7feb577fe640 1 -- 172.21.15.27:0/1821175084 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb5009c4b0 con 0x7feb60102280 2024-04-25T01:25:23.134 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:23.054+0000 7f38aba66200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:25:23.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.218+0000 7feb57fff640 1 -- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7feb4c0a2c80 msgr2=0x7feb4c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:23.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.218+0000 7feb57fff640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7feb4c0a2c80 0x7feb4c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.248+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7feb600623e0 con 0x7feb60102280 2024-04-25T01:25:23.233 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:22.909+0000 7f4c85b8f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:25:23.234 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:23.044+0000 7f4c85b8f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:25:23.234 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:23.138+0000 7f4c85b8f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:25:23.234 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:23.254+0000 7f4c85b8f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:25:23.256 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:22.928+0000 7f04e9e32200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:25:23.256 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:23.064+0000 7f04e9e32200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:25:23.256 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:23.160+0000 7f04e9e32200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:25:23.256 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:23.275+0000 7f04e9e32200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:25:23.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.300+0000 7feb577fe640 1 -- 172.21.15.27:0/1821175084 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7feb5009bdb0 con 0x7feb60102280 2024-04-25T01:25:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7feb4c0a2c80 msgr2=0x7feb4c0a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:25:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7feb4c0a2c80 0x7feb4c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60102280 msgr2=0x7feb60199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60102280 0x7feb60199c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7feb5000e9f0 tx=0x7feb5000edf0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 shutdown_connections 2024-04-25T01:25:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7feb4c0a2c80 0x7feb4c0a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb601074e0 0x7feb601a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb60104bb0 0x7feb6019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 --2- 172.21.15.27:0/1821175084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb60102280 0x7feb60199c80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 >> 172.21.15.27:0/1821175084 conn(0x7feb600fbc10 msgr2=0x7feb60101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 shutdown_connections 2024-04-25T01:25:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.304+0000 7feb64a64640 1 -- 172.21.15.27:0/1821175084 wait complete. 2024-04-25T01:25:23.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-25T01:25:23.397 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:23 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1821175084' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T01:25:23.398 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:23.154+0000 7f38aba66200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.420+0000 7fc5a106d640 1 -- 172.21.15.27:0/2385481204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1006a0 msgr2=0x7fc59c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.420+0000 7fc5a106d640 1 --2- 172.21.15.27:0/2385481204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1006a0 0x7fc59c100ac0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009900 tx=0x7fc58c02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.420+0000 7fc5a106d640 1 -- 172.21.15.27:0/2385481204 shutdown_connections 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.420+0000 7fc5a106d640 1 --2- 172.21.15.27:0/2385481204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc59c102250 0x7fc59c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.420+0000 7fc5a106d640 1 --2- 172.21.15.27:0/2385481204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59c101890 0x7fc59c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.420+0000 7fc5a106d640 1 --2- 172.21.15.27:0/2385481204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1006a0 0x7fc59c100ac0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.420+0000 7fc5a106d640 1 -- 172.21.15.27:0/2385481204 >> 172.21.15.27:0/2385481204 conn(0x7fc59c0fbc10 msgr2=0x7fc59c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.420+0000 7fc5a106d640 1 -- 172.21.15.27:0/2385481204 shutdown_connections 2024-04-25T01:25:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.421+0000 7fc5a106d640 1 -- 172.21.15.27:0/2385481204 wait complete. 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.421+0000 7fc5a106d640 1 Processor -- start 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.421+0000 7fc5a106d640 1 -- start start 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc5a106d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59c1006a0 0x7fc59c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc5a106d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc59c101890 0x7fc59c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc5a106d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c102250 0x7fc59c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc5a106d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c10c320 con 0x7fc59c102250 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc5a106d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c10c1a0 con 0x7fc59c101890 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc5a106d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c10c4a0 con 0x7fc59c1006a0 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59c1006a0 0x7fc59c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c102250 0x7fc59c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59c1006a0 0x7fc59c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:45160/0 (socket says 172.21.15.27:45160) 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59ad9d640 1 -- 172.21.15.27:0/1013182402 learned_addr learned my addr 172.21.15.27:0/1013182402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59a59c640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc59c101890 0x7fc59c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59b59e640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59c1006a0 msgr2=0x7fc59c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59b59e640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59c1006a0 0x7fc59c197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59b59e640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc59c101890 msgr2=0x7fc59c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59b59e640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc59c101890 0x7fc59c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59b59e640 1 -- 172.21.15.27:0/1013182402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c19f6d0 con 0x7fc59c102250 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59ad9d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc59c1006a0 0x7fc59c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.422+0000 7fc59b59e640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c102250 0x7fc59c19f060 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc59400e940 tx=0x7fc59400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.423+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594004d20 con 0x7fc59c102250 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.423+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc594005020 con 0x7fc59c102250 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.423+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59c19f9c0 con 0x7fc59c102250 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.423+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940116b0 con 0x7fc59c102250 2024-04-25T01:25:23.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.423+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59c19fec0 con 0x7fc59c102250 2024-04-25T01:25:23.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.424+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59c1a0150 con 0x7fc59c102250 2024-04-25T01:25:23.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.425+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fc5940054d0 con 0x7fc59c102250 2024-04-25T01:25:23.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.426+0000 7fc583fff640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 0x7fc5840a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.426+0000 7fc59ad9d640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 msgr2=0x7fc5840a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:23.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.426+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc5940cf4b0 con 0x7fc59c102250 2024-04-25T01:25:23.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.426+0000 7fc59ad9d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 0x7fc5840a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:25:23.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.431+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc594016030 con 0x7fc59c102250 2024-04-25T01:25:23.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:23 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1821175084' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T01:25:23.525 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:23.369+0000 7f04e9e32200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:25:23.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:23 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1821175084' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T01:25:23.542 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:23.346+0000 7f4c85b8f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:25:23.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.627+0000 7fc59ad9d640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 msgr2=0x7fc5840a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:23.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.627+0000 7fc59ad9d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 0x7fc5840a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:25:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:23.643+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 --> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fc59c0fddb0 con 0x7fc5840a2c80 2024-04-25T01:25:23.695 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:23.417+0000 7f38aba66200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:25:23.695 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:23.562+0000 7f38aba66200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:25:23.695 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:23.664+0000 7f38aba66200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:25:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:24.028+0000 7fc59ad9d640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 msgr2=0x7fc5840a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:24.028+0000 7fc59ad9d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 0x7fc5840a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:25:24.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:23.786+0000 7f38aba66200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:25:24.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:23.885+0000 7f38aba66200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:25:24.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:24 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1821175084' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T01:25:24.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:24 smithi153 ceph-mon[32621]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:25:24.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:24.187+0000 7f04e9e32200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:25:24.526 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:25:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z[34470]: 2024-04-25T01:25:24.401+0000 7f04e9e32200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:25:24.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:24 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1821175084' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T01:25:24.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:24 smithi139 ceph-mon[32760]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:25:24.542 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:24 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:24.156+0000 7f4c85b8f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:25:24.542 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:25:24 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y[34143]: 2024-04-25T01:25:24.371+0000 7f4c85b8f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:25:24.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:24 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1821175084' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T01:25:24.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:24 smithi027 ceph-mon[29197]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:25:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:24.829+0000 7fc59ad9d640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 msgr2=0x7fc5840a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4052240654 2024-04-25T01:25:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:24.829+0000 7fc59ad9d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 0x7fc5840a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:25:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:24.988+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7fc594098080 con 0x7fc59c102250 2024-04-25T01:25:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:24.988+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 msgr2=0x7fc5840a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:25:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:24.988+0000 7fc583fff640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/4052240654,v1:172.21.15.27:6801/4052240654] conn(0x7fc5840a2c80 0x7fc5840a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:25.137 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:24.743+0000 7f38aba66200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:25:25.137 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:24.973+0000 7f38aba66200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:25:25.445 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.231+0000 7f380cc18700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.445 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.231+0000 7f380cc18700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.446 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.231+0000 7f380cc18700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.446 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.231+0000 7f380cc18700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.446 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.231+0000 7f380cc18700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.446 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.232+0000 7f380ac14700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.446 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.232+0000 7f380ac14700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.446 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.232+0000 7f380ac14700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.446 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.232+0000 7f380ac14700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.446 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:25.232+0000 7f380ac14700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: Standby manager daemon y restarted 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: Standby manager daemon y started 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: Standby manager daemon z restarted 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: Standby manager daemon z started 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:25:25.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: Active manager daemon x restarted 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: Activating manager daemon x 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: mgrmap e20: x(active, starting, since 0.0144643s), standbys: y, z 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:25:25.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:25:25.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:25:25.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: Manager daemon x is now available 2024-04-25T01:25:25.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:25.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:25:25.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:25 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: Standby manager daemon y restarted 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: Standby manager daemon y started 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: Standby manager daemon z restarted 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: Standby manager daemon z started 2024-04-25T01:25:25.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: Active manager daemon x restarted 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: Activating manager daemon x 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: mgrmap e20: x(active, starting, since 0.0144643s), standbys: y, z 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:25:25.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:25:25.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: Manager daemon x is now available 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:25:25.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:25 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:25:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: Standby manager daemon y restarted 2024-04-25T01:25:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: Standby manager daemon y started 2024-04-25T01:25:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:25:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:25:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:25:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.139:0/654373462' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:25:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: Standby manager daemon z restarted 2024-04-25T01:25:25.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: Standby manager daemon z started 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.? 172.21.15.153:0/1485733345' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: Active manager daemon x restarted 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: Activating manager daemon x 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: mgrmap e20: x(active, starting, since 0.0144643s), standbys: y, z 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:25:25.543 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:25:25.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:25:25.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:25:25.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:25:25.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:25:25.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:25:25.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: Manager daemon x is now available 2024-04-25T01:25:25.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:25.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:25:25.546 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:25 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:25:25.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:25.997+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7fc594051310 con 0x7fc59c102250 2024-04-25T01:25:25.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:25.997+0000 7fc583fff640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc584051920 0x7fc584051d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:25.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:25.997+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fc594060d90 con 0x7fc584051920 2024-04-25T01:25:25.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:25.999+0000 7fc59ad9d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc584051920 0x7fc584051d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:25.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.003+0000 7fc59ad9d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc584051920 0x7fc584051d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc58c000c00 tx=0x7fc58c005d50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.017+0000 7fc583fff640 1 -- 172.21.15.27:0/1013182402 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fc594060d90 con 0x7fc584051920 2024-04-25T01:25:25.997 INFO:teuthology.orchestra.run.smithi027.stdout:{} 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc584051920 msgr2=0x7fc584051d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc5a106d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc584051920 0x7fc584051d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc58c000c00 tx=0x7fc58c005d50 comp rx=0 tx=0).stop 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c102250 msgr2=0x7fc59c19f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc5a106d640 1 --2- 172.21.15.27:0/1013182402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c102250 0x7fc59c19f060 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc59400e940 tx=0x7fc59400ee00 comp rx=0 tx=0).stop 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc59b59e640 1 -- 172.21.15.27:0/1013182402 reap_dead start 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 shutdown_connections 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 >> 172.21.15.27:0/1013182402 conn(0x7fc59c0fbc10 msgr2=0x7fc59c0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 shutdown_connections 2024-04-25T01:25:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.020+0000 7fc5a106d640 1 -- 172.21.15.27:0/1013182402 wait complete. 2024-04-25T01:25:26.015 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T01:25:26.027 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T01:25:26.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 -- 172.21.15.27:0/441633315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc102260 msgr2=0x7f69dc104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 --2- 172.21.15.27:0/441633315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc102260 0x7f69dc104650 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f69c4009900 tx=0x7f69c402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 -- 172.21.15.27:0/441633315 shutdown_connections 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 --2- 172.21.15.27:0/441633315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69dc1074c0 0x7f69dc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 --2- 172.21.15.27:0/441633315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc104b90 0x7f69dc106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 --2- 172.21.15.27:0/441633315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc102260 0x7f69dc104650 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 -- 172.21.15.27:0/441633315 >> 172.21.15.27:0/441633315 conn(0x7f69dc0fbbf0 msgr2=0x7f69dc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 -- 172.21.15.27:0/441633315 shutdown_connections 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 -- 172.21.15.27:0/441633315 wait complete. 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 Processor -- start 2024-04-25T01:25:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.169+0000 7f69e249a640 1 -- start start 2024-04-25T01:25:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69e249a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69dc102260 0x7f69dc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69e249a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc104b90 0x7f69dc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69e249a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1074c0 0x7f69dc19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69e249a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10d3f0 con 0x7f69dc104b90 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69e249a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10d270 con 0x7f69dc102260 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69e249a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10d570 con 0x7f69dc1074c0 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc104b90 0x7f69dc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc104b90 0x7f69dc198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52794/0 (socket says 172.21.15.27:52794) 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 -- 172.21.15.27:0/3113349664 learned_addr learned my addr 172.21.15.27:0/3113349664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69dbfff640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69dc102260 0x7f69dc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69e0a37640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1074c0 0x7f69dc19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 -- 172.21.15.27:0/3113349664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1074c0 msgr2=0x7f69dc19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1074c0 0x7f69dc19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 -- 172.21.15.27:0/3113349664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69dc102260 msgr2=0x7f69dc197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69dc102260 0x7f69dc197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 -- 172.21.15.27:0/3113349664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69dc19f6e0 con 0x7f69dc104b90 2024-04-25T01:25:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69db7fe640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc104b90 0x7f69dc198000 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00c970 tx=0x7f69cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.170+0000 7f69d97fa640 1 -- 172.21.15.27:0/3113349664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc007bb0 con 0x7f69dc104b90 2024-04-25T01:25:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.171+0000 7f69d97fa640 1 -- 172.21.15.27:0/3113349664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69cc007d50 con 0x7f69dc104b90 2024-04-25T01:25:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.171+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69dc19f9d0 con 0x7f69dc104b90 2024-04-25T01:25:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.171+0000 7f69d97fa640 1 -- 172.21.15.27:0/3113349664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc00f450 con 0x7f69dc104b90 2024-04-25T01:25:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.171+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69dc19ff20 con 0x7f69dc104b90 2024-04-25T01:25:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.172+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69dc0fd6e0 con 0x7f69dc104b90 2024-04-25T01:25:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.173+0000 7f69d97fa640 1 -- 172.21.15.27:0/3113349664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f69cc0046c0 con 0x7f69dc104b90 2024-04-25T01:25:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.173+0000 7f69d97fa640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69c80a2990 0x7f69c80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.173+0000 7f69dbfff640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69c80a2990 0x7f69c80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.173+0000 7f69d97fa640 1 -- 172.21.15.27:0/3113349664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f69cc0cf090 con 0x7f69dc104b90 2024-04-25T01:25:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.174+0000 7f69dbfff640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69c80a2990 0x7f69c80a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f69c40314e0 tx=0x7f69c4000bc0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.178+0000 7f69d97fa640 1 -- 172.21.15.27:0/3113349664 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f69cc014030 con 0x7f69dc104b90 2024-04-25T01:25:26.253 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:26.009+0000 7f380840f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:26.253 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:26.009+0000 7f380840f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:26.253 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:26.009+0000 7f380840f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:26.253 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:26.009+0000 7f380840f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:26.253 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:25:26.009+0000 7f380840f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.369+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69dc062340 con 0x7f69dc104b90 2024-04-25T01:25:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.370+0000 7f69d97fa640 1 -- 172.21.15.27:0/3113349664 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25170 (secure 0 0 0) 0x7f69cc09b7e0 con 0x7f69dc104b90 2024-04-25T01:25:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:26.351 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:25:24.976745+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2238094164":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/3292898694":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/4288305036":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6800/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2504114047":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.372+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69c80a2990 msgr2=0x7f69c80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.372+0000 7f69e249a640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69c80a2990 0x7f69c80a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f69c40314e0 tx=0x7f69c4000bc0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.372+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc104b90 msgr2=0x7f69dc198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.372+0000 7f69e249a640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc104b90 0x7f69dc198000 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00c970 tx=0x7f69cc00ce30 comp rx=0 tx=0).stop 2024-04-25T01:25:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.373+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 shutdown_connections 2024-04-25T01:25:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.373+0000 7f69e249a640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1074c0 0x7f69dc19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.373+0000 7f69e249a640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69c80a2990 0x7f69c80a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.373+0000 7f69e249a640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc104b90 0x7f69dc198000 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.373+0000 7f69e249a640 1 --2- 172.21.15.27:0/3113349664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69dc102260 0x7f69dc197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.373+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 >> 172.21.15.27:0/3113349664 conn(0x7f69dc0fbbf0 msgr2=0x7f69dc102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.373+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 shutdown_connections 2024-04-25T01:25:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.373+0000 7f69e249a640 1 -- 172.21.15.27:0/3113349664 wait complete. 2024-04-25T01:25:26.365 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:25:26.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 -- 172.21.15.27:0/247180211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe814102990 msgr2=0x7fe814102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/247180211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe814102990 0x7fe814102e10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe808009900 tx=0x7fe80802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 -- 172.21.15.27:0/247180211 shutdown_connections 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/247180211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe814103350 0x7fe814109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/247180211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe814102990 0x7fe814102e10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/247180211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8141017a0 0x7fe814101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 -- 172.21.15.27:0/247180211 >> 172.21.15.27:0/247180211 conn(0x7fe8140fcd10 msgr2=0x7fe8140ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 -- 172.21.15.27:0/247180211 shutdown_connections 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.493+0000 7fe81ae2c640 1 -- 172.21.15.27:0/247180211 wait complete. 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe81ae2c640 1 Processor -- start 2024-04-25T01:25:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe81ae2c640 1 -- start start 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe81ae2c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8141017a0 0x7fe814197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe81ae2c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe814102990 0x7fe814198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe81ae2c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe814103350 0x7fe81419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe81ae2c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe81410d400 con 0x7fe8141017a0 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe81ae2c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe81410d280 con 0x7fe814103350 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe81ae2c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe81410d580 con 0x7fe814102990 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe818bc8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8141017a0 0x7fe814197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe818bc8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8141017a0 0x7fe814197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52822/0 (socket says 172.21.15.27:52822) 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe818bc8640 1 -- 172.21.15.27:0/3138935311 learned_addr learned my addr 172.21.15.27:0/3138935311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe8193c9640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe814103350 0x7fe81419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe813fff640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe814102990 0x7fe814198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe818bc8640 1 -- 172.21.15.27:0/3138935311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe814102990 msgr2=0x7fe814198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe818bc8640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe814102990 0x7fe814198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe818bc8640 1 -- 172.21.15.27:0/3138935311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe814103350 msgr2=0x7fe81419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe818bc8640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe814103350 0x7fe81419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.494+0000 7fe818bc8640 1 -- 172.21.15.27:0/3138935311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81419f6f0 con 0x7fe8141017a0 2024-04-25T01:25:26.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.495+0000 7fe818bc8640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8141017a0 0x7fe814197ad0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe80400d950 tx=0x7fe80400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:26.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.495+0000 7fe811ffb640 1 -- 172.21.15.27:0/3138935311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804017020 con 0x7fe8141017a0 2024-04-25T01:25:26.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.495+0000 7fe811ffb640 1 -- 172.21.15.27:0/3138935311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe804009c30 con 0x7fe8141017a0 2024-04-25T01:25:26.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.495+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81419f9e0 con 0x7fe8141017a0 2024-04-25T01:25:26.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.495+0000 7fe811ffb640 1 -- 172.21.15.27:0/3138935311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80400f740 con 0x7fe8141017a0 2024-04-25T01:25:26.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.495+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81419ff00 con 0x7fe8141017a0 2024-04-25T01:25:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.496+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8140689b0 con 0x7fe8141017a0 2024-04-25T01:25:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.497+0000 7fe811ffb640 1 -- 172.21.15.27:0/3138935311 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7fe80401a020 con 0x7fe8141017a0 2024-04-25T01:25:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.498+0000 7fe811ffb640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7fc0a2990 0x7fe7fc0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.498+0000 7fe813fff640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7fc0a2990 0x7fe7fc0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.498+0000 7fe811ffb640 1 -- 172.21.15.27:0/3138935311 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe8040ceb10 con 0x7fe8141017a0 2024-04-25T01:25:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.498+0000 7fe813fff640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7fc0a2990 0x7fe7fc0a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe808000c00 tx=0x7fe8080315d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:26.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.502+0000 7fe811ffb640 1 -- 172.21.15.27:0/3138935311 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe804021070 con 0x7fe8141017a0 2024-04-25T01:25:26.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.686+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe814062340 con 0x7fe8141017a0 2024-04-25T01:25:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.686+0000 7fe811ffb640 1 -- 172.21.15.27:0/3138935311 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fe804009de0 con 0x7fe8141017a0 2024-04-25T01:25:26.668 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:25:26.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7fc0a2990 msgr2=0x7fe7fc0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7fc0a2990 0x7fe7fc0a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe808000c00 tx=0x7fe8080315d0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8141017a0 msgr2=0x7fe814197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8141017a0 0x7fe814197ad0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe80400d950 tx=0x7fe80400de10 comp rx=0 tx=0).stop 2024-04-25T01:25:26.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 shutdown_connections 2024-04-25T01:25:26.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe814103350 0x7fe81419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe814102990 0x7fe814198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7fc0a2990 0x7fe7fc0a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.688+0000 7fe81ae2c640 1 --2- 172.21.15.27:0/3138935311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8141017a0 0x7fe814197ad0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.689+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 >> 172.21.15.27:0/3138935311 conn(0x7fe8140fcd10 msgr2=0x7fe814105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:26.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.689+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 shutdown_connections 2024-04-25T01:25:26.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.689+0000 7fe81ae2c640 1 -- 172.21.15.27:0/3138935311 wait complete. 2024-04-25T01:25:26.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:25:26.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 -- 172.21.15.27:0/3033481880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c102980 msgr2=0x7f366c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 --2- 172.21.15.27:0/3033481880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c102980 0x7f366c102e00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3660009900 tx=0x7f366002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:26.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 -- 172.21.15.27:0/3033481880 shutdown_connections 2024-04-25T01:25:26.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 --2- 172.21.15.27:0/3033481880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f366c103340 0x7f366c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 --2- 172.21.15.27:0/3033481880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c102980 0x7f366c102e00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 --2- 172.21.15.27:0/3033481880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c101790 0x7f366c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 -- 172.21.15.27:0/3033481880 >> 172.21.15.27:0/3033481880 conn(0x7f366c0fcd20 msgr2=0x7f366c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 -- 172.21.15.27:0/3033481880 shutdown_connections 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.825+0000 7f3671f61640 1 -- 172.21.15.27:0/3033481880 wait complete. 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f3671f61640 1 Processor -- start 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f3671f61640 1 -- start start 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f3671f61640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f366c101790 0x7f366c19e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f3671f61640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c102980 0x7f366c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f3671f61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c103340 0x7f366c1a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f3671f61640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f366c10d410 con 0x7f366c103340 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f3671f61640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f366c10d290 con 0x7f366c101790 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f3671f61640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f366c10d590 con 0x7f366c102980 2024-04-25T01:25:26.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f366bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c103340 0x7f366c1a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f366b7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f366c101790 0x7f366c19e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f366bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c103340 0x7f366c1a56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52852/0 (socket says 172.21.15.27:52852) 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f366bfff640 1 -- 172.21.15.27:0/3337762059 learned_addr learned my addr 172.21.15.27:0/3337762059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.826+0000 7f366affd640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c102980 0x7f366c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f366bfff640 1 -- 172.21.15.27:0/3337762059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c102980 msgr2=0x7f366c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f366bfff640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c102980 0x7f366c19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f366bfff640 1 -- 172.21.15.27:0/3337762059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f366c101790 msgr2=0x7f366c19e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f366bfff640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f366c101790 0x7f366c19e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f366bfff640 1 -- 172.21.15.27:0/3337762059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f366c1a5d40 con 0x7f366c103340 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f366bfff640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c103340 0x7f366c1a56d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f366400eb10 tx=0x7f366400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f3668ff9640 1 -- 172.21.15.27:0/3337762059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664004b20 con 0x7f366c103340 2024-04-25T01:25:26.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f3668ff9640 1 -- 172.21.15.27:0/3337762059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3664004cc0 con 0x7f366c103340 2024-04-25T01:25:26.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f366c1a5fd0 con 0x7f366c103340 2024-04-25T01:25:26.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f3668ff9640 1 -- 172.21.15.27:0/3337762059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664010760 con 0x7f366c103340 2024-04-25T01:25:26.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.827+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f366c1a64a0 con 0x7f366c103340 2024-04-25T01:25:26.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.828+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f366c1069d0 con 0x7f366c103340 2024-04-25T01:25:26.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.830+0000 7f3668ff9640 1 -- 172.21.15.27:0/3337762059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f3664005020 con 0x7f366c103340 2024-04-25T01:25:26.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.830+0000 7f3668ff9640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f36540a2990 0x7f36540a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.830+0000 7f366b7fe640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f36540a2990 0x7f36540a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.830+0000 7f3668ff9640 1 -- 172.21.15.27:0/3337762059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f36640cf6a0 con 0x7f366c103340 2024-04-25T01:25:26.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.831+0000 7f366b7fe640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f36540a2990 0x7f36540a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f366c1027e0 tx=0x7f365c005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:26.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:26.838+0000 7f3668ff9640 1 -- 172.21.15.27:0/3337762059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f366409c440 con 0x7f366c103340 2024-04-25T01:25:26.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.004+0000 7f3668ff9640 1 -- 172.21.15.27:0/3337762059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 22) ==== 138004+0+0 (secure 0 0 0) 0x7f36640983b0 con 0x7f366c103340 2024-04-25T01:25:27.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.078+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f366c062340 con 0x7f366c103340 2024-04-25T01:25:27.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.078+0000 7f3668ff9640 1 -- 172.21.15.27:0/3337762059 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f36640576b0 con 0x7f366c103340 2024-04-25T01:25:27.059 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f36540a2990 msgr2=0x7f36540a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f36540a2990 0x7f36540a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f366c1027e0 tx=0x7f365c005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c103340 msgr2=0x7f366c1a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c103340 0x7f366c1a56d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f366400eb10 tx=0x7f366400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 shutdown_connections 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f36540a2990 0x7f36540a4e50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c103340 0x7f366c1a56d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f366c102980 0x7f366c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 --2- 172.21.15.27:0/3337762059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f366c101790 0x7f366c19e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 >> 172.21.15.27:0/3337762059 conn(0x7f366c0fcd20 msgr2=0x7f366c103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 shutdown_connections 2024-04-25T01:25:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.080+0000 7f3671f61640 1 -- 172.21.15.27:0/3337762059 wait complete. 2024-04-25T01:25:27.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 -- 172.21.15.27:0/3173398502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 msgr2=0x7ff67c1017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 --2- 172.21.15.27:0/3173398502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 0x7ff67c1017f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff674009ca0 tx=0x7ff67402a950 comp rx=0 tx=0).stop 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 -- 172.21.15.27:0/3173398502 shutdown_connections 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 --2- 172.21.15.27:0/3173398502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 0x7ff67c1017f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 --2- 172.21.15.27:0/3173398502 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff67c1009d0 0x7ff67c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 --2- 172.21.15.27:0/3173398502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff67c100070 0x7ff67c100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 -- 172.21.15.27:0/3173398502 >> 172.21.15.27:0/3173398502 conn(0x7ff67c0fbbf0 msgr2=0x7ff67c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 -- 172.21.15.27:0/3173398502 shutdown_connections 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.218+0000 7ff6824bd640 1 -- 172.21.15.27:0/3173398502 wait complete. 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff6824bd640 1 Processor -- start 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff6824bd640 1 -- start start 2024-04-25T01:25:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff6824bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff67c100070 0x7ff67c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff6824bd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff67c1009d0 0x7ff67c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff6824bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 0x7ff67c1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff6824bd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10f650 con 0x7ff67c101370 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff6824bd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10f4d0 con 0x7ff67c1009d0 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff6824bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10f7d0 con 0x7ff67c100070 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff680a5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 0x7ff67c1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff680a5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 0x7ff67c1a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52884/0 (socket says 172.21.15.27:52884) 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff680a5a640 1 -- 172.21.15.27:0/2197549798 learned_addr learned my addr 172.21.15.27:0/2197549798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff67bfff640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff67c100070 0x7ff67c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.219+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff67c1009d0 0x7ff67c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff680a5a640 1 -- 172.21.15.27:0/2197549798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff67c100070 msgr2=0x7ff67c199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff680a5a640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff67c100070 0x7ff67c199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff680a5a640 1 -- 172.21.15.27:0/2197549798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff67c1009d0 msgr2=0x7ff67c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff680a5a640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff67c1009d0 0x7ff67c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff680a5a640 1 -- 172.21.15.27:0/2197549798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c1a18b0 con 0x7ff67c101370 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff680a5a640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 0x7ff67c1a1210 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff67402af40 tx=0x7ff67402fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff6797fa640 1 -- 172.21.15.27:0/2197549798 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674034040 con 0x7ff67c101370 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff6797fa640 1 -- 172.21.15.27:0/2197549798 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff674030920 con 0x7ff67c101370 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67c1a1b40 con 0x7ff67c101370 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff6797fa640 1 -- 172.21.15.27:0/2197549798 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674045480 con 0x7ff67c101370 2024-04-25T01:25:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.220+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff67c1a2030 con 0x7ff67c101370 2024-04-25T01:25:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.222+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff67c10af10 con 0x7ff67c101370 2024-04-25T01:25:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.222+0000 7ff6797fa640 1 -- 172.21.15.27:0/2197549798 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) ==== 138004+0+0 (secure 0 0 0) 0x7ff674037020 con 0x7ff67c101370 2024-04-25T01:25:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.223+0000 7ff6797fa640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6640a2d90 0x7ff6640a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.223+0000 7ff6797fa640 1 -- 172.21.15.27:0/2197549798 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff6740f07f0 con 0x7ff67c101370 2024-04-25T01:25:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.223+0000 7ff67bfff640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6640a2d90 0x7ff6640a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.223+0000 7ff67bfff640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6640a2d90 0x7ff6640a5250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff67c19ab40 tx=0x7ff66c00b390 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.227+0000 7ff6797fa640 1 -- 172.21.15.27:0/2197549798 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff6740bd020 con 0x7ff67c101370 2024-04-25T01:25:27.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:26 smithi153 ceph-mon[32621]: mgrmap e21: x(active, since 1.02142s), standbys: y, z 2024-04-25T01:25:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:26 smithi153 ceph-mon[32621]: [25/Apr/2024:01:25:26] ENGINE Bus STARTING 2024-04-25T01:25:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:26 smithi153 ceph-mon[32621]: [25/Apr/2024:01:25:26] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T01:25:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:26 smithi153 ceph-mon[32621]: [25/Apr/2024:01:25:26] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T01:25:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:26 smithi153 ceph-mon[32621]: [25/Apr/2024:01:25:26] ENGINE Bus STARTED 2024-04-25T01:25:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:26 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3113349664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:27.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:26 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3138935311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:27.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:26 smithi139 ceph-mon[32760]: mgrmap e21: x(active, since 1.02142s), standbys: y, z 2024-04-25T01:25:27.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:26 smithi139 ceph-mon[32760]: [25/Apr/2024:01:25:26] ENGINE Bus STARTING 2024-04-25T01:25:27.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:26 smithi139 ceph-mon[32760]: [25/Apr/2024:01:25:26] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T01:25:27.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:26 smithi139 ceph-mon[32760]: [25/Apr/2024:01:25:26] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T01:25:27.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:26 smithi139 ceph-mon[32760]: [25/Apr/2024:01:25:26] ENGINE Bus STARTED 2024-04-25T01:25:27.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:26 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3113349664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:27.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:26 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3138935311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.428+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff67c02d170 con 0x7ff67c101370 2024-04-25T01:25:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.428+0000 7ff6797fa640 1 -- 172.21.15.27:0/2197549798 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7ff6740bc920 con 0x7ff67c101370 2024-04-25T01:25:27.409 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:25:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.430+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6640a2d90 msgr2=0x7ff6640a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.430+0000 7ff6824bd640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6640a2d90 0x7ff6640a5250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff67c19ab40 tx=0x7ff66c00b390 comp rx=0 tx=0).stop 2024-04-25T01:25:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 msgr2=0x7ff67c1a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 0x7ff67c1a1210 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff67402af40 tx=0x7ff67402fd40 comp rx=0 tx=0).stop 2024-04-25T01:25:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 shutdown_connections 2024-04-25T01:25:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6640a2d90 0x7ff6640a5250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c101370 0x7ff67c1a1210 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff67c1009d0 0x7ff67c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 --2- 172.21.15.27:0/2197549798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff67c100070 0x7ff67c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 >> 172.21.15.27:0/2197549798 conn(0x7ff67c0fbbf0 msgr2=0x7ff67c0fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 shutdown_connections 2024-04-25T01:25:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.431+0000 7ff6824bd640 1 -- 172.21.15.27:0/2197549798 wait complete. 2024-04-25T01:25:27.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-mon[29197]: mgrmap e21: x(active, since 1.02142s), standbys: y, z 2024-04-25T01:25:27.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-mon[29197]: [25/Apr/2024:01:25:26] ENGINE Bus STARTING 2024-04-25T01:25:27.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-mon[29197]: [25/Apr/2024:01:25:26] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T01:25:27.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-mon[29197]: [25/Apr/2024:01:25:26] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T01:25:27.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-mon[29197]: [25/Apr/2024:01:25:26] ENGINE Bus STARTED 2024-04-25T01:25:27.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3113349664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:27.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:26 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3138935311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:27.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:25:27.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 -- 172.21.15.27:0/3447941957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 msgr2=0x7fc69c106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 --2- 172.21.15.27:0/3447941957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 0x7fc69c106fe0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009900 tx=0x7fc68c02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 -- 172.21.15.27:0/3447941957 shutdown_connections 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 --2- 172.21.15.27:0/3447941957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c107520 0x7fc69c109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 --2- 172.21.15.27:0/3447941957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 0x7fc69c106fe0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 --2- 172.21.15.27:0/3447941957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc69c1022c0 0x7fc69c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 -- 172.21.15.27:0/3447941957 >> 172.21.15.27:0/3447941957 conn(0x7fc69c0fbbf0 msgr2=0x7fc69c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 -- 172.21.15.27:0/3447941957 shutdown_connections 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.548+0000 7fc6a4114640 1 -- 172.21.15.27:0/3447941957 wait complete. 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a4114640 1 Processor -- start 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a4114640 1 -- start start 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a4114640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc69c1022c0 0x7fc69c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a4114640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 0x7fc69c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a4114640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c107520 0x7fc69c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a4114640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c10d410 con 0x7fc69c104bf0 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a4114640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c10d290 con 0x7fc69c1022c0 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a4114640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c10d590 con 0x7fc69c107520 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a16af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 0x7fc69c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a16af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 0x7fc69c198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52904/0 (socket says 172.21.15.27:52904) 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a16af640 1 -- 172.21.15.27:0/1576056972 learned_addr learned my addr 172.21.15.27:0/1576056972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a1eb0640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc69c1022c0 0x7fc69c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a26b1640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c107520 0x7fc69c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a16af640 1 -- 172.21.15.27:0/1576056972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c107520 msgr2=0x7fc69c19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a16af640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c107520 0x7fc69c19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a16af640 1 -- 172.21.15.27:0/1576056972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc69c1022c0 msgr2=0x7fc69c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a16af640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc69c1022c0 0x7fc69c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a16af640 1 -- 172.21.15.27:0/1576056972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c19f730 con 0x7fc69c104bf0 2024-04-25T01:25:27.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.549+0000 7fc6a26b1640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c107520 0x7fc69c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:27.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.550+0000 7fc6a16af640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 0x7fc69c198050 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc68c031610 tx=0x7fc68c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:27.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.550+0000 7fc692ffd640 1 -- 172.21.15.27:0/1576056972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c03f030 con 0x7fc69c104bf0 2024-04-25T01:25:27.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.550+0000 7fc692ffd640 1 -- 172.21.15.27:0/1576056972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc68c003ad0 con 0x7fc69c104bf0 2024-04-25T01:25:27.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.550+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c19f9c0 con 0x7fc69c104bf0 2024-04-25T01:25:27.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.550+0000 7fc692ffd640 1 -- 172.21.15.27:0/1576056972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c035070 con 0x7fc69c104bf0 2024-04-25T01:25:27.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.550+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c19feb0 con 0x7fc69c104bf0 2024-04-25T01:25:27.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.551+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc69c0fd6e0 con 0x7fc69c104bf0 2024-04-25T01:25:27.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.552+0000 7fc692ffd640 1 -- 172.21.15.27:0/1576056972 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) ==== 138004+0+0 (secure 0 0 0) 0x7fc68c042030 con 0x7fc69c104bf0 2024-04-25T01:25:27.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.553+0000 7fc692ffd640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6880a2d90 0x7fc6880a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.553+0000 7fc6a1eb0640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6880a2d90 0x7fc6880a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.553+0000 7fc692ffd640 1 -- 172.21.15.27:0/1576056972 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc68c0eff30 con 0x7fc69c104bf0 2024-04-25T01:25:27.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.553+0000 7fc6a1eb0640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6880a2d90 0x7fc6880a5250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc698009bd0 tx=0x7fc698005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:27.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.557+0000 7fc692ffd640 1 -- 172.21.15.27:0/1576056972 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc68c0bc000 con 0x7fc69c104bf0 2024-04-25T01:25:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.745+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc69c02d390 con 0x7fc69c104bf0 2024-04-25T01:25:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.745+0000 7fc692ffd640 1 -- 172.21.15.27:0/1576056972 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fc68c0c0300 con 0x7fc69c104bf0 2024-04-25T01:25:27.726 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.747+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6880a2d90 msgr2=0x7fc6880a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.747+0000 7fc6a4114640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6880a2d90 0x7fc6880a5250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc698009bd0 tx=0x7fc698005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 msgr2=0x7fc69c198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 0x7fc69c198050 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc68c031610 tx=0x7fc68c004270 comp rx=0 tx=0).stop 2024-04-25T01:25:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 shutdown_connections 2024-04-25T01:25:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c107520 0x7fc69c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6880a2d90 0x7fc6880a5250 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c104bf0 0x7fc69c198050 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 --2- 172.21.15.27:0/1576056972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc69c1022c0 0x7fc69c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 >> 172.21.15.27:0/1576056972 conn(0x7fc69c0fbbf0 msgr2=0x7fc69c102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 shutdown_connections 2024-04-25T01:25:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.748+0000 7fc6a4114640 1 -- 172.21.15.27:0/1576056972 wait complete. 2024-04-25T01:25:27.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.862+0000 7f10e0129640 1 -- 172.21.15.27:0/350886258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d8104b70 msgr2=0x7f10d8106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.862+0000 7f10e0129640 1 --2- 172.21.15.27:0/350886258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d8104b70 0x7f10d8106f60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009900 tx=0x7f10c802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.862+0000 7f10e0129640 1 -- 172.21.15.27:0/350886258 shutdown_connections 2024-04-25T01:25:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.862+0000 7f10e0129640 1 --2- 172.21.15.27:0/350886258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d81074a0 0x7f10d81098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.862+0000 7f10e0129640 1 --2- 172.21.15.27:0/350886258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d8104b70 0x7f10d8106f60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.862+0000 7f10e0129640 1 --2- 172.21.15.27:0/350886258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10d8102240 0x7f10d8104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.862+0000 7f10e0129640 1 -- 172.21.15.27:0/350886258 >> 172.21.15.27:0/350886258 conn(0x7f10d80fbc30 msgr2=0x7f10d80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.862+0000 7f10e0129640 1 -- 172.21.15.27:0/350886258 shutdown_connections 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 -- 172.21.15.27:0/350886258 wait complete. 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 Processor -- start 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 -- start start 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d8102240 0x7f10d8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10d8104b70 0x7f10d8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d81074a0 0x7f10d819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f10d810d440 con 0x7f10d81074a0 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f10d810d2c0 con 0x7f10d8104b70 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10e0129640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10d810d5c0 con 0x7f10d8102240 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10de6c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d81074a0 0x7f10d819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10de6c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d81074a0 0x7f10d819f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50230/0 (socket says 172.21.15.27:50230) 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10ddec5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d8102240 0x7f10d8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10de6c6640 1 -- 172.21.15.27:0/1017577266 learned_addr learned my addr 172.21.15.27:0/1017577266 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.863+0000 7f10dd6c4640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10d8104b70 0x7f10d8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10de6c6640 1 -- 172.21.15.27:0/1017577266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d8102240 msgr2=0x7f10d8197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10de6c6640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d8102240 0x7f10d8197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10de6c6640 1 -- 172.21.15.27:0/1017577266 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10d8104b70 msgr2=0x7f10d8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10de6c6640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10d8104b70 0x7f10d8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10de6c6640 1 -- 172.21.15.27:0/1017577266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d819f730 con 0x7f10d81074a0 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10de6c6640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d81074a0 0x7f10d819f0c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f10d000eb10 tx=0x7f10d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10ceffd640 1 -- 172.21.15.27:0/1017577266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0004b20 con 0x7f10d81074a0 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d819fa20 con 0x7f10d81074a0 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10ceffd640 1 -- 172.21.15.27:0/1017577266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10d0004cc0 con 0x7f10d81074a0 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10ceffd640 1 -- 172.21.15.27:0/1017577266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0010760 con 0x7f10d81074a0 2024-04-25T01:25:27.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.864+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d819ff90 con 0x7f10d81074a0 2024-04-25T01:25:27.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.865+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d80fd720 con 0x7f10d81074a0 2024-04-25T01:25:27.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.867+0000 7f10ceffd640 1 -- 172.21.15.27:0/1017577266 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) ==== 138004+0+0 (secure 0 0 0) 0x7f10d0005000 con 0x7f10d81074a0 2024-04-25T01:25:27.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.868+0000 7f10ceffd640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10c40a2d90 0x7f10c40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:27.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.868+0000 7f10ddec5640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10c40a2d90 0x7f10c40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:27.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.868+0000 7f10ceffd640 1 -- 172.21.15.27:0/1017577266 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f10d00d03a0 con 0x7f10d81074a0 2024-04-25T01:25:27.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.868+0000 7f10ddec5640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10c40a2d90 0x7f10c40a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009bd0 tx=0x7f10d4005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:27.873+0000 7f10ceffd640 1 -- 172.21.15.27:0/1017577266 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f10d009cac0 con 0x7f10d81074a0 2024-04-25T01:25:28.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.063+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10d8062340 con 0x7f10d81074a0 2024-04-25T01:25:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.065+0000 7f10ceffd640 1 -- 172.21.15.27:0/1017577266 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25170 (secure 0 0 0) 0x7f10d009c3c0 con 0x7f10d81074a0 2024-04-25T01:25:28.045 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:28.046 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:25:24.976745+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2238094164":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/3292898694":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/4288305036":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6800/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2504114047":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10c40a2d90 msgr2=0x7f10c40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10c40a2d90 0x7f10c40a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009bd0 tx=0x7f10d4005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d81074a0 msgr2=0x7f10d819f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d81074a0 0x7f10d819f0c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f10d000eb10 tx=0x7f10d000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 shutdown_connections 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10c40a2d90 0x7f10c40a5250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10d81074a0 0x7f10d819f0c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10d8104b70 0x7f10d8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 --2- 172.21.15.27:0/1017577266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d8102240 0x7f10d8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 >> 172.21.15.27:0/1017577266 conn(0x7f10d80fbc30 msgr2=0x7f10d8102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 shutdown_connections 2024-04-25T01:25:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.067+0000 7f10e0129640 1 -- 172.21.15.27:0/1017577266 wait complete. 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3337762059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.059 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2197549798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.059 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:27 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1576056972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.060 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:25:28.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:25:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 -- 172.21.15.27:0/2765078984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec101870 msgr2=0x7efcec101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 --2- 172.21.15.27:0/2765078984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec101870 0x7efcec101cf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009900 tx=0x7efcdc02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 -- 172.21.15.27:0/2765078984 shutdown_connections 2024-04-25T01:25:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 --2- 172.21.15.27:0/2765078984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcec102230 0x7efcec108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 --2- 172.21.15.27:0/2765078984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec101870 0x7efcec101cf0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 --2- 172.21.15.27:0/2765078984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcec100680 0x7efcec100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 -- 172.21.15.27:0/2765078984 >> 172.21.15.27:0/2765078984 conn(0x7efcec0fbbf0 msgr2=0x7efcec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 -- 172.21.15.27:0/2765078984 shutdown_connections 2024-04-25T01:25:28.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.186+0000 7efcf0af5640 1 -- 172.21.15.27:0/2765078984 wait complete. 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcf0af5640 1 Processor -- start 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcf0af5640 1 -- start start 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcf0af5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec100680 0x7efcec069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcf0af5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcec101870 0x7efcec19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcf0af5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcec102230 0x7efcec19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcf0af5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efcec10c300 con 0x7efcec100680 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcf0af5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efcec10c180 con 0x7efcec102230 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcf0af5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efcec10c480 con 0x7efcec101870 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcea59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec100680 0x7efcec069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcea59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec100680 0x7efcec069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50260/0 (socket says 172.21.15.27:50260) 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcea59c640 1 -- 172.21.15.27:0/3589922754 learned_addr learned my addr 172.21.15.27:0/3589922754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efce9d9b640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcec101870 0x7efcec19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.187+0000 7efcead9d640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcec102230 0x7efcec19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efcea59c640 1 -- 172.21.15.27:0/3589922754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcec101870 msgr2=0x7efcec19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efcea59c640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcec101870 0x7efcec19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efcea59c640 1 -- 172.21.15.27:0/3589922754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcec102230 msgr2=0x7efcec19ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efcea59c640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcec102230 0x7efcec19ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efcea59c640 1 -- 172.21.15.27:0/3589922754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcec19f770 con 0x7efcec100680 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efcea59c640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec100680 0x7efcec069440 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efcd400e9c0 tx=0x7efcd400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efce37fe640 1 -- 172.21.15.27:0/3589922754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd400cd80 con 0x7efcec100680 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efce37fe640 1 -- 172.21.15.27:0/3589922754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcd40035f0 con 0x7efcec100680 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcec19fa00 con 0x7efcec100680 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efce37fe640 1 -- 172.21.15.27:0/3589922754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4010690 con 0x7efcec100680 2024-04-25T01:25:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.188+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcec19fed0 con 0x7efcec100680 2024-04-25T01:25:28.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.189+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcec107a90 con 0x7efcec100680 2024-04-25T01:25:28.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.190+0000 7efce37fe640 1 -- 172.21.15.27:0/3589922754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) ==== 138004+0+0 (secure 0 0 0) 0x7efcd40037a0 con 0x7efcec100680 2024-04-25T01:25:28.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.191+0000 7efce37fe640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efcd80a2d90 0x7efcd80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.191+0000 7efce9d9b640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efcd80a2d90 0x7efcd80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.191+0000 7efce37fe640 1 -- 172.21.15.27:0/3589922754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7efcd40cfcd0 con 0x7efcec100680 2024-04-25T01:25:28.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.192+0000 7efce9d9b640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efcd80a2d90 0x7efcd80a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0314e0 tx=0x7efcdc000bc0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.196+0000 7efce37fe640 1 -- 172.21.15.27:0/3589922754 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efcd409c3f0 con 0x7efcec100680 2024-04-25T01:25:28.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3337762059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2197549798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:27 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1576056972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3337762059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2197549798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:27 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1576056972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:28.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.391+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efcec0623e0 con 0x7efcec100680 2024-04-25T01:25:28.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.392+0000 7efce37fe640 1 -- 172.21.15.27:0/3589922754 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7efcd409bcf0 con 0x7efcec100680 2024-04-25T01:25:28.372 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efcd80a2d90 msgr2=0x7efcd80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efcd80a2d90 0x7efcd80a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0314e0 tx=0x7efcdc000bc0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec100680 msgr2=0x7efcec069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec100680 0x7efcec069440 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efcd400e9c0 tx=0x7efcd400ee80 comp rx=0 tx=0).stop 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 shutdown_connections 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efcec102230 0x7efcec19ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcec101870 0x7efcec19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efcd80a2d90 0x7efcd80a5250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 --2- 172.21.15.27:0/3589922754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcec100680 0x7efcec069440 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 >> 172.21.15.27:0/3589922754 conn(0x7efcec0fbbf0 msgr2=0x7efcec104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 shutdown_connections 2024-04-25T01:25:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.394+0000 7efcf0af5640 1 -- 172.21.15.27:0/3589922754 wait complete. 2024-04-25T01:25:28.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:25:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 -- 172.21.15.27:0/3933486352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80100680 msgr2=0x7f4d80100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 --2- 172.21.15.27:0/3933486352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80100680 0x7f4d80100aa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009900 tx=0x7f4d7002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 -- 172.21.15.27:0/3933486352 shutdown_connections 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 --2- 172.21.15.27:0/3933486352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d80102230 0x7f4d80108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 --2- 172.21.15.27:0/3933486352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d80101870 0x7f4d80101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 --2- 172.21.15.27:0/3933486352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80100680 0x7f4d80100aa0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 -- 172.21.15.27:0/3933486352 >> 172.21.15.27:0/3933486352 conn(0x7f4d800fbbf0 msgr2=0x7f4d800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 -- 172.21.15.27:0/3933486352 shutdown_connections 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.517+0000 7f4d8495a640 1 -- 172.21.15.27:0/3933486352 wait complete. 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d8495a640 1 Processor -- start 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d8495a640 1 -- start start 2024-04-25T01:25:28.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d8495a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d80100680 0x7f4d80069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d8495a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d80101870 0x7f4d8019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d8495a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80102230 0x7f4d8019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d8495a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8010c3a0 con 0x7f4d80102230 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d8495a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8010c220 con 0x7f4d80101870 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d8495a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8010c520 con 0x7f4d80100680 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d7ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80102230 0x7f4d8019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d7e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d80100680 0x7f4d80069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d7ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80102230 0x7f4d8019eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50278/0 (socket says 172.21.15.27:50278) 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d7ed9d640 1 -- 172.21.15.27:0/2235675084 learned_addr learned my addr 172.21.15.27:0/2235675084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.518+0000 7f4d7dd9b640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d80101870 0x7f4d8019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d7ed9d640 1 -- 172.21.15.27:0/2235675084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d80100680 msgr2=0x7f4d80069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d7ed9d640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d80100680 0x7f4d80069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d7ed9d640 1 -- 172.21.15.27:0/2235675084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d80101870 msgr2=0x7f4d8019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d7ed9d640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d80101870 0x7f4d8019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d7ed9d640 1 -- 172.21.15.27:0/2235675084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8019f7d0 con 0x7f4d80102230 2024-04-25T01:25:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d7ed9d640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80102230 0x7f4d8019eec0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800b970 tx=0x7f4d7800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d6f7fe640 1 -- 172.21.15.27:0/2235675084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78019070 con 0x7f4d80102230 2024-04-25T01:25:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d6f7fe640 1 -- 172.21.15.27:0/2235675084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d7800cb40 con 0x7f4d80102230 2024-04-25T01:25:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d6f7fe640 1 -- 172.21.15.27:0/2235675084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78007650 con 0x7f4d80102230 2024-04-25T01:25:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8019fa60 con 0x7f4d80102230 2024-04-25T01:25:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.519+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8019ffb0 con 0x7f4d80102230 2024-04-25T01:25:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.520+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d801079a0 con 0x7f4d80102230 2024-04-25T01:25:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.522+0000 7f4d6f7fe640 1 -- 172.21.15.27:0/2235675084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) ==== 138004+0+0 (secure 0 0 0) 0x7f4d7801c070 con 0x7f4d80102230 2024-04-25T01:25:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.522+0000 7f4d6f7fe640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d90 0x7f4d680a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.523+0000 7f4d7e59c640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d90 0x7f4d680a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.523+0000 7f4d6f7fe640 1 -- 172.21.15.27:0/2235675084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4d780ced50 con 0x7f4d80102230 2024-04-25T01:25:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.523+0000 7f4d7e59c640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d90 0x7f4d680a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d7002aa00 tx=0x7f4d700315b0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:28.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.527+0000 7f4d6f7fe640 1 -- 172.21.15.27:0/2235675084 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d780d2030 con 0x7f4d80102230 2024-04-25T01:25:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.715+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d8002d370 con 0x7f4d80102230 2024-04-25T01:25:28.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.716+0000 7f4d6f7fe640 1 -- 172.21.15.27:0/2235675084 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f4d7809afe0 con 0x7f4d80102230 2024-04-25T01:25:28.696 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d90 msgr2=0x7f4d680a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d90 0x7f4d680a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d7002aa00 tx=0x7f4d700315b0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80102230 msgr2=0x7f4d8019eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80102230 0x7f4d8019eec0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800b970 tx=0x7f4d7800be30 comp rx=0 tx=0).stop 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 shutdown_connections 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d90 0x7f4d680a5250 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d80102230 0x7f4d8019eec0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d80101870 0x7f4d8019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 --2- 172.21.15.27:0/2235675084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d80100680 0x7f4d80069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 >> 172.21.15.27:0/2235675084 conn(0x7f4d800fbbf0 msgr2=0x7f4d80104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 shutdown_connections 2024-04-25T01:25:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.718+0000 7f4d8495a640 1 -- 172.21.15.27:0/2235675084 wait complete. 2024-04-25T01:25:28.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 -- 172.21.15.27:0/2284195971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 msgr2=0x7f60dc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2284195971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 0x7f60dc100aa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009900 tx=0x7f60d802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 -- 172.21.15.27:0/2284195971 shutdown_connections 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2284195971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60dc102230 0x7f60dc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2284195971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60dc101870 0x7f60dc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2284195971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 0x7f60dc100aa0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 -- 172.21.15.27:0/2284195971 >> 172.21.15.27:0/2284195971 conn(0x7f60dc0fbbf0 msgr2=0x7f60dc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 -- 172.21.15.27:0/2284195971 shutdown_connections 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.838+0000 7f60e3df9640 1 -- 172.21.15.27:0/2284195971 wait complete. 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e3df9640 1 Processor -- start 2024-04-25T01:25:28.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e3df9640 1 -- start start 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e3df9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 0x7f60dc069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e3df9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60dc101870 0x7f60dc19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e3df9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60dc102230 0x7f60dc1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e3df9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc10c300 con 0x7f60dc100680 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e3df9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc10c180 con 0x7f60dc102230 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e3df9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc10c480 con 0x7f60dc101870 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e1b95640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 0x7f60dc069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e1b95640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 0x7f60dc069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50300/0 (socket says 172.21.15.27:50300) 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e1b95640 1 -- 172.21.15.27:0/2245360085 learned_addr learned my addr 172.21.15.27:0/2245360085 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e1394640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60dc101870 0x7f60dc19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.839+0000 7f60e2396640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60dc102230 0x7f60dc1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e1394640 1 -- 172.21.15.27:0/2245360085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60dc102230 msgr2=0x7f60dc1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e1394640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60dc102230 0x7f60dc1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e1394640 1 -- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 msgr2=0x7f60dc069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e1394640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 0x7f60dc069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e1394640 1 -- 172.21.15.27:0/2245360085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc1a1960 con 0x7f60dc101870 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e1b95640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 0x7f60dc069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e1394640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60dc101870 0x7f60dc19ec20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00e9f0 tx=0x7f60cc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60d2ffd640 1 -- 172.21.15.27:0/2245360085 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00ccc0 con 0x7f60dc101870 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60d2ffd640 1 -- 172.21.15.27:0/2245360085 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60cc010910 con 0x7f60dc101870 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60dc1a1bf0 con 0x7f60dc101870 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60d2ffd640 1 -- 172.21.15.27:0/2245360085 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc018960 con 0x7f60dc101870 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.840+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60dc1a20f0 con 0x7f60dc101870 2024-04-25T01:25:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.841+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60dc107a70 con 0x7f60dc101870 2024-04-25T01:25:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.843+0000 7f60d2ffd640 1 -- 172.21.15.27:0/2245360085 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 22) ==== 138004+0+0 (secure 0 0 0) 0x7f60cc010ac0 con 0x7f60dc101870 2024-04-25T01:25:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.843+0000 7f60d2ffd640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60c80a2dc0 0x7f60c80a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.844+0000 7f60e1b95640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60c80a2dc0 0x7f60c80a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.844+0000 7f60d2ffd640 1 -- 172.21.15.27:0/2245360085 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f60cc0d01c0 con 0x7f60dc101870 2024-04-25T01:25:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.844+0000 7f60e1b95640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60c80a2dc0 0x7f60c80a5280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60d8000c00 tx=0x7f60d80046c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:28.848+0000 7f60d2ffd640 1 -- 172.21.15.27:0/2245360085 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f60cc09c290 con 0x7f60dc101870 2024-04-25T01:25:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.042+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f60dc02d3c0 con 0x7f60dc101870 2024-04-25T01:25:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.042+0000 7f60d2ffd640 1 -- 172.21.15.27:0/2245360085 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f60cc0bae20 con 0x7f60dc101870 2024-04-25T01:25:29.022 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:25:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60c80a2dc0 msgr2=0x7f60c80a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60c80a2dc0 0x7f60c80a5280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60d8000c00 tx=0x7f60d80046c0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60dc101870 msgr2=0x7f60dc19ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60dc101870 0x7f60dc19ec20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00e9f0 tx=0x7f60cc00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 shutdown_connections 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60dc102230 0x7f60dc1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60dc101870 0x7f60dc19ec20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60c80a2dc0 0x7f60c80a5280 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 --2- 172.21.15.27:0/2245360085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60dc100680 0x7f60dc069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 >> 172.21.15.27:0/2245360085 conn(0x7f60dc0fbbf0 msgr2=0x7f60dc104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 shutdown_connections 2024-04-25T01:25:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.044+0000 7f60e3df9640 1 -- 172.21.15.27:0/2245360085 wait complete. 2024-04-25T01:25:29.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:25:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 -- 172.21.15.27:0/2995532677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 msgr2=0x7faf88101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 --2- 172.21.15.27:0/2995532677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 0x7faf88101d00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009900 tx=0x7faf7c02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 -- 172.21.15.27:0/2995532677 shutdown_connections 2024-04-25T01:25:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 --2- 172.21.15.27:0/2995532677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf88102240 0x7faf88108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 --2- 172.21.15.27:0/2995532677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 0x7faf88101d00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 --2- 172.21.15.27:0/2995532677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf88100690 0x7faf88100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 -- 172.21.15.27:0/2995532677 >> 172.21.15.27:0/2995532677 conn(0x7faf880fbc00 msgr2=0x7faf880fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:29.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 -- 172.21.15.27:0/2995532677 shutdown_connections 2024-04-25T01:25:29.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.163+0000 7faf8dff9640 1 -- 172.21.15.27:0/2995532677 wait complete. 2024-04-25T01:25:29.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.164+0000 7faf8dff9640 1 Processor -- start 2024-04-25T01:25:29.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.164+0000 7faf8dff9640 1 -- start start 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.164+0000 7faf8dff9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf88100690 0x7faf88197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.164+0000 7faf8dff9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 0x7faf88198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.164+0000 7faf8dff9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf88102240 0x7faf8819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.164+0000 7faf8dff9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faf8810c310 con 0x7faf88101880 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.164+0000 7faf8dff9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7faf8810c190 con 0x7faf88102240 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.164+0000 7faf8dff9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf8810c490 con 0x7faf88100690 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 0x7faf88198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 0x7faf88198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50318/0 (socket says 172.21.15.27:50318) 2024-04-25T01:25:29.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf877fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf88100690 0x7faf88197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 -- 172.21.15.27:0/3281170569 learned_addr learned my addr 172.21.15.27:0/3281170569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf87fff640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf88102240 0x7faf8819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 -- 172.21.15.27:0/3281170569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf88100690 msgr2=0x7faf88197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf88100690 0x7faf88197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 -- 172.21.15.27:0/3281170569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf88102240 msgr2=0x7faf8819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf88102240 0x7faf8819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 -- 172.21.15.27:0/3281170569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8819f700 con 0x7faf88101880 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf877fe640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf88100690 0x7faf88197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf86ffd640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 0x7faf88198020 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009230 tx=0x7faf7c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf84ff9640 1 -- 172.21.15.27:0/3281170569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c03f030 con 0x7faf88101880 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf84ff9640 1 -- 172.21.15.27:0/3281170569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf7c003ce0 con 0x7faf88101880 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf8819f990 con 0x7faf88101880 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf84ff9640 1 -- 172.21.15.27:0/3281170569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c035070 con 0x7faf88101880 2024-04-25T01:25:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.165+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf8819fe80 con 0x7faf88101880 2024-04-25T01:25:29.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.166+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf881a0110 con 0x7faf88101880 2024-04-25T01:25:29.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.168+0000 7faf84ff9640 1 -- 172.21.15.27:0/3281170569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) ==== 138004+0+0 (secure 0 0 0) 0x7faf7c042030 con 0x7faf88101880 2024-04-25T01:25:29.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.168+0000 7faf84ff9640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7faf700a2d90 0x7faf700a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.168+0000 7faf877fe640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7faf700a2d90 0x7faf700a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.168+0000 7faf84ff9640 1 -- 172.21.15.27:0/3281170569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7faf7c0f01e0 con 0x7faf88101880 2024-04-25T01:25:29.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.169+0000 7faf877fe640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7faf700a2d90 0x7faf700a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faf881016e0 tx=0x7faf78005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:29.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.172+0000 7faf84ff9640 1 -- 172.21.15.27:0/3281170569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faf7c0bc2b0 con 0x7faf88101880 2024-04-25T01:25:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1017577266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: Unable to set osd_memory_target on smithi139 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-25T01:25:29.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: Unable to set osd_memory_target on smithi153 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3589922754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:28 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.278 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T01:25:29.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: Unable to set osd_memory_target on smithi027 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:25:29.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:29.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:29.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T01:25:29.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:25:29.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:25:29.279 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:29 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2235675084' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:29.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1017577266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:29.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:28 smithi139 ceph-mon[32760]: Unable to set osd_memory_target on smithi139 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-25T01:25:29.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: Unable to set osd_memory_target on smithi153 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3589922754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: Unable to set osd_memory_target on smithi027 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:25:29.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:29.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:29.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T01:25:29.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:25:29.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:25:29.295 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:29 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2235675084' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:29.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.368+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7faf881a04a0 con 0x7faf88101880 2024-04-25T01:25:29.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.368+0000 7faf84ff9640 1 -- 172.21.15.27:0/3281170569 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7faf7c0b6e20 con 0x7faf88101880 2024-04-25T01:25:29.348 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7faf700a2d90 msgr2=0x7faf700a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7faf700a2d90 0x7faf700a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faf881016e0 tx=0x7faf78005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 msgr2=0x7faf88198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 0x7faf88198020 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009230 tx=0x7faf7c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 shutdown_connections 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf88102240 0x7faf8819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7faf700a2d90 0x7faf700a5250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf88101880 0x7faf88198020 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 --2- 172.21.15.27:0/3281170569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf88100690 0x7faf88197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 >> 172.21.15.27:0/3281170569 conn(0x7faf880fbc00 msgr2=0x7faf880fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 shutdown_connections 2024-04-25T01:25:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.370+0000 7faf8dff9640 1 -- 172.21.15.27:0/3281170569 wait complete. 2024-04-25T01:25:29.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1017577266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:29.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:28 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Unable to set osd_memory_target on smithi139 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Unable to set osd_memory_target on smithi153 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T01:25:29.363 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3589922754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Unable to set osd_memory_target on smithi027 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Updating smithi139:/etc/ceph/ceph.conf 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-25T01:25:29.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:29 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2235675084' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:29.365 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T01:25:29.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 -- 172.21.15.27:0/3926888727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8100fe0 msgr2=0x7f35e810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 --2- 172.21.15.27:0/3926888727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8100fe0 0x7f35e810d4f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009cd0 tx=0x7f35e002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 -- 172.21.15.27:0/3926888727 shutdown_connections 2024-04-25T01:25:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 --2- 172.21.15.27:0/3926888727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8100fe0 0x7f35e810d4f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 --2- 172.21.15.27:0/3926888727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e8100620 0x7f35e8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 --2- 172.21.15.27:0/3926888727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e8106630 0x7f35e8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 -- 172.21.15.27:0/3926888727 >> 172.21.15.27:0/3926888727 conn(0x7f35e8077740 msgr2=0x7f35e8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 -- 172.21.15.27:0/3926888727 shutdown_connections 2024-04-25T01:25:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.496+0000 7f35ef170640 1 -- 172.21.15.27:0/3926888727 wait complete. 2024-04-25T01:25:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.497+0000 7f35ef170640 1 Processor -- start 2024-04-25T01:25:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.497+0000 7f35ef170640 1 -- start start 2024-04-25T01:25:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.497+0000 7f35ef170640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e8100620 0x7f35e8196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.497+0000 7f35ef170640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e8100fe0 0x7f35e8197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.497+0000 7f35ef170640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8106630 0x7f35e819b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.497+0000 7f35ef170640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f35e8110a00 con 0x7f35e8106630 2024-04-25T01:25:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.497+0000 7f35ef170640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f35e8110880 con 0x7f35e8100620 2024-04-25T01:25:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.497+0000 7f35ef170640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35e8110b80 con 0x7f35e8100fe0 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8106630 0x7f35e819b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8106630 0x7f35e819b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50334/0 (socket says 172.21.15.27:50334) 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 -- 172.21.15.27:0/3793438795 learned_addr learned my addr 172.21.15.27:0/3793438795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35e7fff640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e8100fe0 0x7f35e8197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ecf0c640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e8100620 0x7f35e8196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 -- 172.21.15.27:0/3793438795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e8100fe0 msgr2=0x7f35e8197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e8100fe0 0x7f35e8197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 -- 172.21.15.27:0/3793438795 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e8100620 msgr2=0x7f35e8196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e8100620 0x7f35e8196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 -- 172.21.15.27:0/3793438795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e819be60 con 0x7f35e8106630 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ecf0c640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e8100620 0x7f35e8196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:29.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ed70d640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8106630 0x7f35e819b730 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f35e0005cc0 tx=0x7f35e0031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35e5ffb640 1 -- 172.21.15.27:0/3793438795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e004a070 con 0x7f35e8106630 2024-04-25T01:25:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35e5ffb640 1 -- 172.21.15.27:0/3793438795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e0004040 con 0x7f35e8106630 2024-04-25T01:25:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35e5ffb640 1 -- 172.21.15.27:0/3793438795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e0030800 con 0x7f35e8106630 2024-04-25T01:25:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.498+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e819c0f0 con 0x7f35e8106630 2024-04-25T01:25:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.499+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e80688d0 con 0x7f35e8106630 2024-04-25T01:25:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.500+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e8101d60 con 0x7f35e8106630 2024-04-25T01:25:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.501+0000 7f35e5ffb640 1 -- 172.21.15.27:0/3793438795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f35e00041f0 con 0x7f35e8106630 2024-04-25T01:25:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.502+0000 7f35e5ffb640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35d80a2e20 0x7f35d80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.502+0000 7f35e5ffb640 1 -- 172.21.15.27:0/3793438795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f35e002e020 con 0x7f35e8106630 2024-04-25T01:25:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.502+0000 7f35ecf0c640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35d80a2e20 0x7f35d80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.503+0000 7f35ecf0c640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35d80a2e20 0x7f35d80a52e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009ba0 tx=0x7f35dc005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:29.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.508+0000 7f35e5ffb640 1 -- 172.21.15.27:0/3793438795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f35e0080740 con 0x7f35e8106630 2024-04-25T01:25:29.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.737+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35e80693b0 con 0x7f35e8106630 2024-04-25T01:25:29.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.738+0000 7f35e5ffb640 1 -- 172.21.15.27:0/3793438795 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f35e00be470 con 0x7f35e8106630 2024-04-25T01:25:29.718 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:29.719 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:25:23.250708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24505,24517]},"id":1}]} 2024-04-25T01:25:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35d80a2e20 msgr2=0x7f35d80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35d80a2e20 0x7f35d80a52e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009ba0 tx=0x7f35dc005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8106630 msgr2=0x7f35e819b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8106630 0x7f35e819b730 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f35e0005cc0 tx=0x7f35e0031e60 comp rx=0 tx=0).stop 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 shutdown_connections 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35d80a2e20 0x7f35d80a52e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8106630 0x7f35e819b730 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f35e8100fe0 0x7f35e8197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 --2- 172.21.15.27:0/3793438795 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35e8100620 0x7f35e8196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.740+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 >> 172.21.15.27:0/3793438795 conn(0x7f35e8077740 msgr2=0x7f35e8102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:29.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.741+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 shutdown_connections 2024-04-25T01:25:29.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.741+0000 7f35ef170640 1 -- 172.21.15.27:0/3793438795 wait complete. 2024-04-25T01:25:29.732 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T01:25:29.732 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T01:25:29.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:25:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.858+0000 7f0e164df640 1 -- 172.21.15.27:0/2535222097 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100620 msgr2=0x7f0e10100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 --2- 172.21.15.27:0/2535222097 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100620 0x7f0e10100aa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400d910 tx=0x7f0e0402c380 comp rx=0 tx=0).stop 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 -- 172.21.15.27:0/2535222097 shutdown_connections 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 --2- 172.21.15.27:0/2535222097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e10100fe0 0x7f0e1010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 --2- 172.21.15.27:0/2535222097 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100620 0x7f0e10100aa0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 --2- 172.21.15.27:0/2535222097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e10106630 0x7f0e10106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 -- 172.21.15.27:0/2535222097 >> 172.21.15.27:0/2535222097 conn(0x7f0e10077740 msgr2=0x7f0e10077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 -- 172.21.15.27:0/2535222097 shutdown_connections 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 -- 172.21.15.27:0/2535222097 wait complete. 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 Processor -- start 2024-04-25T01:25:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.859+0000 7f0e164df640 1 -- start start 2024-04-25T01:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e164df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e10100620 0x7f0e10195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e164df640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100fe0 0x7f0e10196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e164df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e10106630 0x7f0e1019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e164df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1010f930 con 0x7f0e10106630 2024-04-25T01:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e164df640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1010f7b0 con 0x7f0e10100fe0 2024-04-25T01:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e164df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1010fab0 con 0x7f0e10100620 2024-04-25T01:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e14a7c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e10106630 0x7f0e1019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0f7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100fe0 0x7f0e10196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e10100620 0x7f0e10195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0f7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100fe0 0x7f0e10196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:48008/0 (socket says 172.21.15.27:48008) 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e10100620 0x7f0e10195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:48612/0 (socket says 172.21.15.27:48612) 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e14a7c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e10106630 0x7f0e1019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50348/0 (socket says 172.21.15.27:50348) 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0f7fe640 1 -- 172.21.15.27:0/3724991037 learned_addr learned my addr 172.21.15.27:0/3724991037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e14a7c640 1 -- 172.21.15.27:0/3724991037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e10100620 msgr2=0x7f0e10195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e14a7c640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e10100620 0x7f0e10195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e14a7c640 1 -- 172.21.15.27:0/3724991037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100fe0 msgr2=0x7f0e10196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e14a7c640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100fe0 0x7f0e10196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e14a7c640 1 -- 172.21.15.27:0/3724991037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1019add0 con 0x7f0e10106630 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0ffff640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e10100620 0x7f0e10195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e14a7c640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e10106630 0x7f0e1019a6a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009fd0 tx=0x7f0e0800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0d7fa640 1 -- 172.21.15.27:0/3724991037 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0800ce20 con 0x7f0e10106630 2024-04-25T01:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1019b0c0 con 0x7f0e10106630 2024-04-25T01:25:29.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0d7fa640 1 -- 172.21.15.27:0/3724991037 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e08005020 con 0x7f0e10106630 2024-04-25T01:25:29.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.860+0000 7f0e0d7fa640 1 -- 172.21.15.27:0/3724991037 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08010720 con 0x7f0e10106630 2024-04-25T01:25:29.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.861+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e101a2950 con 0x7f0e10106630 2024-04-25T01:25:29.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.862+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e10068560 con 0x7f0e10106630 2024-04-25T01:25:29.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.863+0000 7f0e0d7fa640 1 -- 172.21.15.27:0/3724991037 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f0e08005540 con 0x7f0e10106630 2024-04-25T01:25:29.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.863+0000 7f0e0d7fa640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0df80a2e20 0x7f0df80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.864+0000 7f0e0ffff640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0df80a2e20 0x7f0df80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.864+0000 7f0e0d7fa640 1 -- 172.21.15.27:0/3724991037 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0e080d0170 con 0x7f0e10106630 2024-04-25T01:25:29.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.864+0000 7f0e0ffff640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0df80a2e20 0x7f0df80a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0e000095f0 tx=0x7f0e00005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:29.867+0000 7f0e0d7fa640 1 -- 172.21.15.27:0/3724991037 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e0809c840 con 0x7f0e10106630 2024-04-25T01:25:30.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.050+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e10108c10 con 0x7f0df80a2e20 2024-04-25T01:25:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.060+0000 7f0e0d7fa640 1 -- 172.21.15.27:0/3724991037 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339699 (secure 0 0 0) 0x7f0e10108c10 con 0x7f0df80a2e20 2024-04-25T01:25:30.044 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:30.046 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T01:25:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0df80a2e20 msgr2=0x7f0df80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0df80a2e20 0x7f0df80a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0e000095f0 tx=0x7f0e00005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e10106630 msgr2=0x7f0e1019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e10106630 0x7f0e1019a6a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009fd0 tx=0x7f0e0800ef10 comp rx=0 tx=0).stop 2024-04-25T01:25:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 shutdown_connections 2024-04-25T01:25:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0df80a2e20 0x7f0df80a52e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e10106630 0x7f0e1019a6a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e10100fe0 0x7f0e10196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 --2- 172.21.15.27:0/3724991037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e10100620 0x7f0e10195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 >> 172.21.15.27:0/3724991037 conn(0x7f0e10077740 msgr2=0x7f0e10102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:30.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 shutdown_connections 2024-04-25T01:25:30.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.063+0000 7f0e164df640 1 -- 172.21.15.27:0/3724991037 wait complete. 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2245360085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3281170569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3793438795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:30.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.056 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:25:28.998037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190019,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":986,"num_read_kb":654,"num_write":472,"num_write_kb":4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1244945,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361864,"kb_used_data":12672,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579960,"statfs":{"total":1151940427776,"available":1151569879040,"internally_reserved":0,"allocated":12976128,"data_stored":9890553,"data_compressed":549245,"data_compressed_allocated":741376,"data_compressed_original":4370432,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990220"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800504+0000","last_change":"2024-04-25T01:22:57.847700+0000","last_active":"2024-04-25T01:25:25.800504+0000","last_peered":"2024-04-25T01:25:25.800504+0000","last_clean":"2024-04-25T01:25:25.800504+0000","last_became_active":"2024-04-25T01:22:57.846877+0000","last_became_peered":"2024-04-25T01:22:57.846877+0000","last_unstale":"2024-04-25T01:25:25.800504+0000","last_undegraded":"2024-04-25T01:25:25.800504+0000","last_fullsized":"2024-04-25T01:25:25.800504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:05.918261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241338+0000","last_change":"2024-04-25T01:22:57.848396+0000","last_active":"2024-04-25T01:25:25.241338+0000","last_peered":"2024-04-25T01:25:25.241338+0000","last_clean":"2024-04-25T01:25:25.241338+0000","last_became_active":"2024-04-25T01:22:57.847971+0000","last_became_peered":"2024-04-25T01:22:57.847971+0000","last_unstale":"2024-04-25T01:25:25.241338+0000","last_undegraded":"2024-04-25T01:25:25.241338+0000","last_fullsized":"2024-04-25T01:25:25.241338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:13.143828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240866+0000","last_change":"2024-04-25T01:22:57.856908+0000","last_active":"2024-04-25T01:25:25.240866+0000","last_peered":"2024-04-25T01:25:25.240866+0000","last_clean":"2024-04-25T01:25:25.240866+0000","last_became_active":"2024-04-25T01:22:57.856218+0000","last_became_peered":"2024-04-25T01:22:57.856218+0000","last_unstale":"2024-04-25T01:25:25.240866+0000","last_undegraded":"2024-04-25T01:25:25.240866+0000","last_fullsized":"2024-04-25T01:25:25.240866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:51.971222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119108+0000","last_change":"2024-04-25T01:22:57.849022+0000","last_active":"2024-04-25T01:25:25.119108+0000","last_peered":"2024-04-25T01:25:25.119108+0000","last_clean":"2024-04-25T01:25:25.119108+0000","last_became_active":"2024-04-25T01:22:57.848303+0000","last_became_peered":"2024-04-25T01:22:57.848303+0000","last_unstale":"2024-04-25T01:25:25.119108+0000","last_undegraded":"2024-04-25T01:25:25.119108+0000","last_fullsized":"2024-04-25T01:25:25.119108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:52.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087107+0000","last_change":"2024-04-25T01:23:47.488312+0000","last_active":"2024-04-25T01:25:25.087107+0000","last_peered":"2024-04-25T01:25:25.087107+0000","last_clean":"2024-04-25T01:25:25.087107+0000","last_became_active":"2024-04-25T01:23:47.487923+0000","last_became_peered":"2024-04-25T01:23:47.487923+0000","last_unstale":"2024-04-25T01:25:25.087107+0000","last_undegraded":"2024-04-25T01:25:25.087107+0000","last_fullsized":"2024-04-25T01:25:25.087107+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:30.276853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983361+0000","last_change":"2024-04-25T01:22:57.855733+0000","last_active":"2024-04-25T01:25:24.983361+0000","last_peered":"2024-04-25T01:25:24.983361+0000","last_clean":"2024-04-25T01:25:24.983361+0000","last_became_active":"2024-04-25T01:22:57.855075+0000","last_became_peered":"2024-04-25T01:22:57.855075+0000","last_unstale":"2024-04-25T01:25:24.983361+0000","last_undegraded":"2024-04-25T01:25:24.983361+0000","last_fullsized":"2024-04-25T01:25:24.983361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:24.563461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238372+0000","last_change":"2024-04-25T01:22:57.849693+0000","last_active":"2024-04-25T01:25:25.238372+0000","last_peered":"2024-04-25T01:25:25.238372+0000","last_clean":"2024-04-25T01:25:25.238372+0000","last_became_active":"2024-04-25T01:22:57.849324+0000","last_became_peered":"2024-04-25T01:22:57.849324+0000","last_unstale":"2024-04-25T01:25:25.238372+0000","last_undegraded":"2024-04-25T01:25:25.238372+0000","last_fullsized":"2024-04-25T01:25:25.238372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:09.936496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983951+0000","last_change":"2024-04-25T01:22:57.847962+0000","last_active":"2024-04-25T01:25:24.983951+0000","last_peered":"2024-04-25T01:25:24.983951+0000","last_clean":"2024-04-25T01:25:24.983951+0000","last_became_active":"2024-04-25T01:22:57.847592+0000","last_became_peered":"2024-04-25T01:22:57.847592+0000","last_unstale":"2024-04-25T01:25:24.983951+0000","last_undegraded":"2024-04-25T01:25:24.983951+0000","last_fullsized":"2024-04-25T01:25:24.983951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:45.594118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239964+0000","last_change":"2024-04-25T01:22:57.860711+0000","last_active":"2024-04-25T01:25:25.239964+0000","last_peered":"2024-04-25T01:25:25.239964+0000","last_clean":"2024-04-25T01:25:25.239964+0000","last_became_active":"2024-04-25T01:22:57.860393+0000","last_became_peered":"2024-04-25T01:22:57.860393+0000","last_unstale":"2024-04-25T01:25:25.239964+0000","last_undegraded":"2024-04-25T01:25:25.239964+0000","last_fullsized":"2024-04-25T01:25:25.239964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:31.068095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984143+0000","last_change":"2024-04-25T01:22:57.853741+0000","last_active":"2024-04-25T01:25:24.984143+0000","last_peered":"2024-04-25T01:25:24.984143+0000","last_clean":"2024-04-25T01:25:24.984143+0000","last_became_active":"2024-04-25T01:22:57.851015+0000","last_became_peered":"2024-04-25T01:22:57.851015+0000","last_unstale":"2024-04-25T01:25:24.984143+0000","last_undegraded":"2024-04-25T01:25:24.984143+0000","last_fullsized":"2024-04-25T01:25:24.984143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:45.831335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239112+0000","last_change":"2024-04-25T01:22:57.853354+0000","last_active":"2024-04-25T01:25:25.239112+0000","last_peered":"2024-04-25T01:25:25.239112+0000","last_clean":"2024-04-25T01:25:25.239112+0000","last_became_active":"2024-04-25T01:22:57.853043+0000","last_became_peered":"2024-04-25T01:22:57.853043+0000","last_unstale":"2024-04-25T01:25:25.239112+0000","last_undegraded":"2024-04-25T01:25:25.239112+0000","last_fullsized":"2024-04-25T01:25:25.239112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:54.378751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119156+0000","last_change":"2024-04-25T01:22:57.855542+0000","last_active":"2024-04-25T01:25:25.119156+0000","last_peered":"2024-04-25T01:25:25.119156+0000","last_clean":"2024-04-25T01:25:25.119156+0000","last_became_active":"2024-04-25T01:22:57.855196+0000","last_became_peered":"2024-04-25T01:22:57.855196+0000","last_unstale":"2024-04-25T01:25:25.119156+0000","last_undegraded":"2024-04-25T01:25:25.119156+0000","last_fullsized":"2024-04-25T01:25:25.119156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:56.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117568+0000","last_change":"2024-04-25T01:22:57.829353+0000","last_active":"2024-04-25T01:25:25.117568+0000","last_peered":"2024-04-25T01:25:25.117568+0000","last_clean":"2024-04-25T01:25:25.117568+0000","last_became_active":"2024-04-25T01:22:57.828937+0000","last_became_peered":"2024-04-25T01:22:57.828937+0000","last_unstale":"2024-04-25T01:25:25.117568+0000","last_undegraded":"2024-04-25T01:25:25.117568+0000","last_fullsized":"2024-04-25T01:25:25.117568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.541556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241354+0000","last_change":"2024-04-25T01:22:57.854635+0000","last_active":"2024-04-25T01:25:25.241354+0000","last_peered":"2024-04-25T01:25:25.241354+0000","last_clean":"2024-04-25T01:25:25.241354+0000","last_became_active":"2024-04-25T01:22:57.854292+0000","last_became_peered":"2024-04-25T01:22:57.854292+0000","last_unstale":"2024-04-25T01:25:25.241354+0000","last_undegraded":"2024-04-25T01:25:25.241354+0000","last_fullsized":"2024-04-25T01:25:25.241354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:48.209059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240635+0000","last_change":"2024-04-25T01:22:57.852210+0000","last_active":"2024-04-25T01:25:25.240635+0000","last_peered":"2024-04-25T01:25:25.240635+0000","last_clean":"2024-04-25T01:25:25.240635+0000","last_became_active":"2024-04-25T01:22:57.851833+0000","last_became_peered":"2024-04-25T01:22:57.851833+0000","last_unstale":"2024-04-25T01:25:25.240635+0000","last_undegraded":"2024-04-25T01:25:25.240635+0000","last_fullsized":"2024-04-25T01:25:25.240635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:14.664785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239054+0000","last_change":"2024-04-25T01:22:57.849357+0000","last_active":"2024-04-25T01:25:25.239054+0000","last_peered":"2024-04-25T01:25:25.239054+0000","last_clean":"2024-04-25T01:25:25.239054+0000","last_became_active":"2024-04-25T01:22:57.849050+0000","last_became_peered":"2024-04-25T01:22:57.849050+0000","last_unstale":"2024-04-25T01:25:25.239054+0000","last_undegraded":"2024-04-25T01:25:25.239054+0000","last_fullsized":"2024-04-25T01:25:25.239054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:37.669424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800559+0000","last_change":"2024-04-25T01:22:57.844790+0000","last_active":"2024-04-25T01:25:25.800559+0000","last_peered":"2024-04-25T01:25:25.800559+0000","last_clean":"2024-04-25T01:25:25.800559+0000","last_became_active":"2024-04-25T01:22:57.844442+0000","last_became_peered":"2024-04-25T01:22:57.844442+0000","last_unstale":"2024-04-25T01:25:25.800559+0000","last_undegraded":"2024-04-25T01:25:25.800559+0000","last_fullsized":"2024-04-25T01:25:25.800559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:33.766034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085227+0000","last_change":"2024-04-25T01:22:57.844430+0000","last_active":"2024-04-25T01:25:25.085227+0000","last_peered":"2024-04-25T01:25:25.085227+0000","last_clean":"2024-04-25T01:25:25.085227+0000","last_became_active":"2024-04-25T01:22:57.844110+0000","last_became_peered":"2024-04-25T01:22:57.844110+0000","last_unstale":"2024-04-25T01:25:25.085227+0000","last_undegraded":"2024-04-25T01:25:25.085227+0000","last_fullsized":"2024-04-25T01:25:25.085227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:50.348249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984509+0000","last_change":"2024-04-25T01:22:57.852705+0000","last_active":"2024-04-25T01:25:24.984509+0000","last_peered":"2024-04-25T01:25:24.984509+0000","last_clean":"2024-04-25T01:25:24.984509+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:25:24.984509+0000","last_undegraded":"2024-04-25T01:25:24.984509+0000","last_fullsized":"2024-04-25T01:25:24.984509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:10.320459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983615+0000","last_change":"2024-04-25T01:23:01.879341+0000","last_active":"2024-04-25T01:25:24.983615+0000","last_peered":"2024-04-25T01:25:24.983615+0000","last_clean":"2024-04-25T01:25:24.983615+0000","last_became_active":"2024-04-25T01:23:01.878881+0000","last_became_peered":"2024-04-25T01:23:01.878881+0000","last_unstale":"2024-04-25T01:25:24.983615+0000","last_undegraded":"2024-04-25T01:25:24.983615+0000","last_fullsized":"2024-04-25T01:25:24.983615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:46.036868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085363+0000","last_change":"2024-04-25T01:22:56.839867+0000","last_active":"2024-04-25T01:25:25.085363+0000","last_peered":"2024-04-25T01:25:25.085363+0000","last_clean":"2024-04-25T01:25:25.085363+0000","last_became_active":"2024-04-25T01:22:56.839521+0000","last_became_peered":"2024-04-25T01:22:56.839521+0000","last_unstale":"2024-04-25T01:25:25.085363+0000","last_undegraded":"2024-04-25T01:25:25.085363+0000","last_fullsized":"2024-04-25T01:25:25.085363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:00.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984330+0000","last_change":"2024-04-25T01:22:57.855922+0000","last_active":"2024-04-25T01:25:24.984330+0000","last_peered":"2024-04-25T01:25:24.984330+0000","last_clean":"2024-04-25T01:25:24.984330+0000","last_became_active":"2024-04-25T01:22:57.855584+0000","last_became_peered":"2024-04-25T01:22:57.855584+0000","last_unstale":"2024-04-25T01:25:24.984330+0000","last_undegraded":"2024-04-25T01:25:24.984330+0000","last_fullsized":"2024-04-25T01:25:24.984330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:02.791269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118747+0000","last_change":"2024-04-25T01:23:01.872340+0000","last_active":"2024-04-25T01:25:25.118747+0000","last_peered":"2024-04-25T01:25:25.118747+0000","last_clean":"2024-04-25T01:25:25.118747+0000","last_became_active":"2024-04-25T01:23:01.871900+0000","last_became_peered":"2024-04-25T01:23:01.871900+0000","last_unstale":"2024-04-25T01:25:25.118747+0000","last_undegraded":"2024-04-25T01:25:25.118747+0000","last_fullsized":"2024-04-25T01:25:25.118747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:11.918022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984380+0000","last_change":"2024-04-25T01:22:56.843851+0000","last_active":"2024-04-25T01:25:24.984380+0000","last_peered":"2024-04-25T01:25:24.984380+0000","last_clean":"2024-04-25T01:25:24.984380+0000","last_became_active":"2024-04-25T01:22:56.843232+0000","last_became_peered":"2024-04-25T01:22:56.843232+0000","last_unstale":"2024-04-25T01:25:24.984380+0000","last_undegraded":"2024-04-25T01:25:24.984380+0000","last_fullsized":"2024-04-25T01:25:24.984380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:13.758328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241152+0000","last_change":"2024-04-25T01:22:57.857686+0000","last_active":"2024-04-25T01:25:25.241152+0000","last_peered":"2024-04-25T01:25:25.241152+0000","last_clean":"2024-04-25T01:25:25.241152+0000","last_became_active":"2024-04-25T01:22:57.857175+0000","last_became_peered":"2024-04-25T01:22:57.857175+0000","last_unstale":"2024-04-25T01:25:25.241152+0000","last_undegraded":"2024-04-25T01:25:25.241152+0000","last_fullsized":"2024-04-25T01:25:25.241152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:24.777861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118799+0000","last_change":"2024-04-25T01:23:01.867914+0000","last_active":"2024-04-25T01:25:25.118799+0000","last_peered":"2024-04-25T01:25:25.118799+0000","last_clean":"2024-04-25T01:25:25.118799+0000","last_became_active":"2024-04-25T01:23:01.867103+0000","last_became_peered":"2024-04-25T01:23:01.867103+0000","last_unstale":"2024-04-25T01:25:25.118799+0000","last_undegraded":"2024-04-25T01:25:25.118799+0000","last_fullsized":"2024-04-25T01:25:25.118799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:26.425433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087318+0000","last_change":"2024-04-25T01:22:56.841310+0000","last_active":"2024-04-25T01:25:25.087318+0000","last_peered":"2024-04-25T01:25:25.087318+0000","last_clean":"2024-04-25T01:25:25.087318+0000","last_became_active":"2024-04-25T01:22:56.840882+0000","last_became_peered":"2024-04-25T01:22:56.840882+0000","last_unstale":"2024-04-25T01:25:25.087318+0000","last_undegraded":"2024-04-25T01:25:25.087318+0000","last_fullsized":"2024-04-25T01:25:25.087318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:29.971448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239551+0000","last_change":"2024-04-25T01:22:57.842810+0000","last_active":"2024-04-25T01:25:25.239551+0000","last_peered":"2024-04-25T01:25:25.239551+0000","last_clean":"2024-04-25T01:25:25.239551+0000","last_became_active":"2024-04-25T01:22:57.842457+0000","last_became_peered":"2024-04-25T01:22:57.842457+0000","last_unstale":"2024-04-25T01:25:25.239551+0000","last_undegraded":"2024-04-25T01:25:25.239551+0000","last_fullsized":"2024-04-25T01:25:25.239551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:45.512037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119017+0000","last_change":"2024-04-25T01:23:01.877298+0000","last_active":"2024-04-25T01:25:25.119017+0000","last_peered":"2024-04-25T01:25:25.119017+0000","last_clean":"2024-04-25T01:25:25.119017+0000","last_became_active":"2024-04-25T01:23:01.876874+0000","last_became_peered":"2024-04-25T01:23:01.876874+0000","last_unstale":"2024-04-25T01:25:25.119017+0000","last_undegraded":"2024-04-25T01:25:25.119017+0000","last_fullsized":"2024-04-25T01:25:25.119017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:35:08.221388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800229+0000","last_change":"2024-04-25T01:24:47.495071+0000","last_active":"2024-04-25T01:25:25.800229+0000","last_peered":"2024-04-25T01:25:25.800229+0000","last_clean":"2024-04-25T01:25:25.800229+0000","last_became_active":"2024-04-25T01:24:47.491300+0000","last_became_peered":"2024-04-25T01:24:47.491300+0000","last_unstale":"2024-04-25T01:25:25.800229+0000","last_undegraded":"2024-04-25T01:25:25.800229+0000","last_fullsized":"2024-04-25T01:25:25.800229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:33:00.552625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093517+0000","last_change":"2024-04-25T01:22:57.851210+0000","last_active":"2024-04-25T01:25:25.093517+0000","last_peered":"2024-04-25T01:25:25.093517+0000","last_clean":"2024-04-25T01:25:25.093517+0000","last_became_active":"2024-04-25T01:22:57.850867+0000","last_became_peered":"2024-04-25T01:22:57.850867+0000","last_unstale":"2024-04-25T01:25:25.093517+0000","last_undegraded":"2024-04-25T01:25:25.093517+0000","last_fullsized":"2024-04-25T01:25:25.093517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:15.998433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984328+0000","last_change":"2024-04-25T01:23:47.489730+0000","last_active":"2024-04-25T01:25:24.984328+0000","last_peered":"2024-04-25T01:25:24.984328+0000","last_clean":"2024-04-25T01:25:24.984328+0000","last_became_active":"2024-04-25T01:23:47.489190+0000","last_became_peered":"2024-04-25T01:23:47.489190+0000","last_unstale":"2024-04-25T01:25:24.984328+0000","last_undegraded":"2024-04-25T01:25:24.984328+0000","last_fullsized":"2024-04-25T01:25:24.984328+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.101073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.237569+0000","last_change":"2024-04-25T01:22:56.834741+0000","last_active":"2024-04-25T01:25:25.237569+0000","last_peered":"2024-04-25T01:25:25.237569+0000","last_clean":"2024-04-25T01:25:25.237569+0000","last_became_active":"2024-04-25T01:22:56.834296+0000","last_became_peered":"2024-04-25T01:22:56.834296+0000","last_unstale":"2024-04-25T01:25:25.237569+0000","last_undegraded":"2024-04-25T01:25:25.237569+0000","last_fullsized":"2024-04-25T01:25:25.237569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:38.940603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087133+0000","last_change":"2024-04-25T01:22:57.854646+0000","last_active":"2024-04-25T01:25:25.087133+0000","last_peered":"2024-04-25T01:25:25.087133+0000","last_clean":"2024-04-25T01:25:25.087133+0000","last_became_active":"2024-04-25T01:22:57.853839+0000","last_became_peered":"2024-04-25T01:22:57.853839+0000","last_unstale":"2024-04-25T01:25:25.087133+0000","last_undegraded":"2024-04-25T01:25:25.087133+0000","last_fullsized":"2024-04-25T01:25:25.087133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:36.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984172+0000","last_change":"2024-04-25T01:23:01.875478+0000","last_active":"2024-04-25T01:25:24.984172+0000","last_peered":"2024-04-25T01:25:24.984172+0000","last_clean":"2024-04-25T01:25:24.984172+0000","last_became_active":"2024-04-25T01:23:01.874643+0000","last_became_peered":"2024-04-25T01:23:01.874643+0000","last_unstale":"2024-04-25T01:25:24.984172+0000","last_undegraded":"2024-04-25T01:25:24.984172+0000","last_fullsized":"2024-04-25T01:25:24.984172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.953507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983490+0000","last_change":"2024-04-25T01:22:56.843118+0000","last_active":"2024-04-25T01:25:24.983490+0000","last_peered":"2024-04-25T01:25:24.983490+0000","last_clean":"2024-04-25T01:25:24.983490+0000","last_became_active":"2024-04-25T01:22:56.842798+0000","last_became_peered":"2024-04-25T01:22:56.842798+0000","last_unstale":"2024-04-25T01:25:24.983490+0000","last_undegraded":"2024-04-25T01:25:24.983490+0000","last_fullsized":"2024-04-25T01:25:24.983490+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:00.054913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093690+0000","last_change":"2024-04-25T01:22:57.850265+0000","last_active":"2024-04-25T01:25:25.093690+0000","last_peered":"2024-04-25T01:25:25.093690+0000","last_clean":"2024-04-25T01:25:25.093690+0000","last_became_active":"2024-04-25T01:22:57.849480+0000","last_became_peered":"2024-04-25T01:22:57.849480+0000","last_unstale":"2024-04-25T01:25:25.093690+0000","last_undegraded":"2024-04-25T01:25:25.093690+0000","last_fullsized":"2024-04-25T01:25:25.093690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:48.577252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800384+0000","last_change":"2024-04-25T01:23:01.865369+0000","last_active":"2024-04-25T01:25:25.800384+0000","last_peered":"2024-04-25T01:25:25.800384+0000","last_clean":"2024-04-25T01:25:25.800384+0000","last_became_active":"2024-04-25T01:23:01.864872+0000","last_became_peered":"2024-04-25T01:23:01.864872+0000","last_unstale":"2024-04-25T01:25:25.800384+0000","last_undegraded":"2024-04-25T01:25:25.800384+0000","last_fullsized":"2024-04-25T01:25:25.800384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:56.266514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116988+0000","last_change":"2024-04-25T01:22:56.842576+0000","last_active":"2024-04-25T01:25:25.116988+0000","last_peered":"2024-04-25T01:25:25.116988+0000","last_clean":"2024-04-25T01:25:25.116988+0000","last_became_active":"2024-04-25T01:22:56.841081+0000","last_became_peered":"2024-04-25T01:22:56.841081+0000","last_unstale":"2024-04-25T01:25:25.116988+0000","last_undegraded":"2024-04-25T01:25:25.116988+0000","last_fullsized":"2024-04-25T01:25:25.116988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:08.174426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983862+0000","last_change":"2024-04-25T01:22:57.842812+0000","last_active":"2024-04-25T01:25:24.983862+0000","last_peered":"2024-04-25T01:25:24.983862+0000","last_clean":"2024-04-25T01:25:24.983862+0000","last_became_active":"2024-04-25T01:22:57.842462+0000","last_became_peered":"2024-04-25T01:22:57.842462+0000","last_unstale":"2024-04-25T01:25:24.983862+0000","last_undegraded":"2024-04-25T01:25:24.983862+0000","last_fullsized":"2024-04-25T01:25:24.983862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:14:13.444850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241074+0000","last_change":"2024-04-25T01:23:01.870401+0000","last_active":"2024-04-25T01:25:25.241074+0000","last_peered":"2024-04-25T01:25:25.241074+0000","last_clean":"2024-04-25T01:25:25.241074+0000","last_became_active":"2024-04-25T01:23:01.869932+0000","last_became_peered":"2024-04-25T01:23:01.869932+0000","last_unstale":"2024-04-25T01:25:25.241074+0000","last_undegraded":"2024-04-25T01:25:25.241074+0000","last_fullsized":"2024-04-25T01:25:25.241074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:13.323209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093295+0000","last_change":"2024-04-25T01:22:56.835946+0000","last_active":"2024-04-25T01:25:25.093295+0000","last_peered":"2024-04-25T01:25:25.093295+0000","last_clean":"2024-04-25T01:25:25.093295+0000","last_became_active":"2024-04-25T01:22:56.835630+0000","last_became_peered":"2024-04-25T01:22:56.835630+0000","last_unstale":"2024-04-25T01:25:25.093295+0000","last_undegraded":"2024-04-25T01:25:25.093295+0000","last_fullsized":"2024-04-25T01:25:25.093295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:02.128275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.084835+0000","last_change":"2024-04-25T01:22:57.844462+0000","last_active":"2024-04-25T01:25:25.084835+0000","last_peered":"2024-04-25T01:25:25.084835+0000","last_clean":"2024-04-25T01:25:25.084835+0000","last_became_active":"2024-04-25T01:22:57.843426+0000","last_became_peered":"2024-04-25T01:22:57.843426+0000","last_unstale":"2024-04-25T01:25:25.084835+0000","last_undegraded":"2024-04-25T01:25:25.084835+0000","last_fullsized":"2024-04-25T01:25:25.084835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:42.896622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984063+0000","last_change":"2024-04-25T01:23:01.871387+0000","last_active":"2024-04-25T01:25:24.984063+0000","last_peered":"2024-04-25T01:25:24.984063+0000","last_clean":"2024-04-25T01:25:24.984063+0000","last_became_active":"2024-04-25T01:23:01.870901+0000","last_became_peered":"2024-04-25T01:23:01.870901+0000","last_unstale":"2024-04-25T01:25:24.984063+0000","last_undegraded":"2024-04-25T01:25:24.984063+0000","last_fullsized":"2024-04-25T01:25:24.984063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:49.582515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238114+0000","last_change":"2024-04-25T01:22:56.834440+0000","last_active":"2024-04-25T01:25:25.238114+0000","last_peered":"2024-04-25T01:25:25.238114+0000","last_clean":"2024-04-25T01:25:25.238114+0000","last_became_active":"2024-04-25T01:22:56.834116+0000","last_became_peered":"2024-04-25T01:22:56.834116+0000","last_unstale":"2024-04-25T01:25:25.238114+0000","last_undegraded":"2024-04-25T01:25:25.238114+0000","last_fullsized":"2024-04-25T01:25:25.238114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:01.176601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239994+0000","last_change":"2024-04-25T01:22:57.853560+0000","last_active":"2024-04-25T01:25:25.239994+0000","last_peered":"2024-04-25T01:25:25.239994+0000","last_clean":"2024-04-25T01:25:25.239994+0000","last_became_active":"2024-04-25T01:22:57.853234+0000","last_became_peered":"2024-04-25T01:22:57.853234+0000","last_unstale":"2024-04-25T01:25:25.239994+0000","last_undegraded":"2024-04-25T01:25:25.239994+0000","last_fullsized":"2024-04-25T01:25:25.239994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:51.139371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.084730+0000","last_change":"2024-04-25T01:23:01.873856+0000","last_active":"2024-04-25T01:25:25.084730+0000","last_peered":"2024-04-25T01:25:25.084730+0000","last_clean":"2024-04-25T01:25:25.084730+0000","last_became_active":"2024-04-25T01:23:01.873413+0000","last_became_peered":"2024-04-25T01:23:01.873413+0000","last_unstale":"2024-04-25T01:25:25.084730+0000","last_undegraded":"2024-04-25T01:25:25.084730+0000","last_fullsized":"2024-04-25T01:25:25.084730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:11.802479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239188+0000","last_change":"2024-04-25T01:22:56.831080+0000","last_active":"2024-04-25T01:25:25.239188+0000","last_peered":"2024-04-25T01:25:25.239188+0000","last_clean":"2024-04-25T01:25:25.239188+0000","last_became_active":"2024-04-25T01:22:56.830735+0000","last_became_peered":"2024-04-25T01:22:56.830735+0000","last_unstale":"2024-04-25T01:25:25.239188+0000","last_undegraded":"2024-04-25T01:25:25.239188+0000","last_fullsized":"2024-04-25T01:25:25.239188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:38.418439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241002+0000","last_change":"2024-04-25T01:22:57.853838+0000","last_active":"2024-04-25T01:25:25.241002+0000","last_peered":"2024-04-25T01:25:25.241002+0000","last_clean":"2024-04-25T01:25:25.241002+0000","last_became_active":"2024-04-25T01:22:57.853023+0000","last_became_peered":"2024-04-25T01:22:57.853023+0000","last_unstale":"2024-04-25T01:25:25.241002+0000","last_undegraded":"2024-04-25T01:25:25.241002+0000","last_fullsized":"2024-04-25T01:25:25.241002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:18.515831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118964+0000","last_change":"2024-04-25T01:23:01.865446+0000","last_active":"2024-04-25T01:25:25.118964+0000","last_peered":"2024-04-25T01:25:25.118964+0000","last_clean":"2024-04-25T01:25:25.118964+0000","last_became_active":"2024-04-25T01:23:01.864937+0000","last_became_peered":"2024-04-25T01:23:01.864937+0000","last_unstale":"2024-04-25T01:25:25.118964+0000","last_undegraded":"2024-04-25T01:25:25.118964+0000","last_fullsized":"2024-04-25T01:25:25.118964+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:18.041430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093597+0000","last_change":"2024-04-25T01:22:56.836055+0000","last_active":"2024-04-25T01:25:25.093597+0000","last_peered":"2024-04-25T01:25:25.093597+0000","last_clean":"2024-04-25T01:25:25.093597+0000","last_became_active":"2024-04-25T01:22:56.835760+0000","last_became_peered":"2024-04-25T01:22:56.835760+0000","last_unstale":"2024-04-25T01:25:25.093597+0000","last_undegraded":"2024-04-25T01:25:25.093597+0000","last_fullsized":"2024-04-25T01:25:25.093597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:59.249996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239754+0000","last_change":"2024-04-25T01:22:57.852444+0000","last_active":"2024-04-25T01:25:25.239754+0000","last_peered":"2024-04-25T01:25:25.239754+0000","last_clean":"2024-04-25T01:25:25.239754+0000","last_became_active":"2024-04-25T01:22:57.852127+0000","last_became_peered":"2024-04-25T01:22:57.852127+0000","last_unstale":"2024-04-25T01:25:25.239754+0000","last_undegraded":"2024-04-25T01:25:25.239754+0000","last_fullsized":"2024-04-25T01:25:25.239754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:11.718512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983143+0000","last_change":"2024-04-25T01:23:47.039057+0000","last_active":"2024-04-25T01:25:24.983143+0000","last_peered":"2024-04-25T01:25:24.983143+0000","last_clean":"2024-04-25T01:25:24.983143+0000","last_became_active":"2024-04-25T01:23:47.038568+0000","last_became_peered":"2024-04-25T01:23:47.038568+0000","last_unstale":"2024-04-25T01:25:24.983143+0000","last_undegraded":"2024-04-25T01:25:24.983143+0000","last_fullsized":"2024-04-25T01:25:24.983143+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:23.828975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"94'39","reported_seq":346,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.549084+0000","last_change":"2024-04-25T01:24:47.501745+0000","last_active":"2024-04-25T01:25:25.549084+0000","last_peered":"2024-04-25T01:25:25.549084+0000","last_clean":"2024-04-25T01:25:25.549084+0000","last_became_active":"2024-04-25T01:24:47.494829+0000","last_became_peered":"2024-04-25T01:24:47.494829+0000","last_unstale":"2024-04-25T01:25:25.549084+0000","last_undegraded":"2024-04-25T01:25:25.549084+0000","last_fullsized":"2024-04-25T01:25:25.549084+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:54.482539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69677,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":252,"num_read_kb":194,"num_write":40,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":63605,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087512+0000","last_change":"2024-04-25T01:22:57.854554+0000","last_active":"2024-04-25T01:25:25.087512+0000","last_peered":"2024-04-25T01:25:25.087512+0000","last_clean":"2024-04-25T01:25:25.087512+0000","last_became_active":"2024-04-25T01:22:57.853328+0000","last_became_peered":"2024-04-25T01:22:57.853328+0000","last_unstale":"2024-04-25T01:25:25.087512+0000","last_undegraded":"2024-04-25T01:25:25.087512+0000","last_fullsized":"2024-04-25T01:25:25.087512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:21.774148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093976+0000","last_change":"2024-04-25T01:23:01.870571+0000","last_active":"2024-04-25T01:25:25.093976+0000","last_peered":"2024-04-25T01:25:25.093976+0000","last_clean":"2024-04-25T01:25:25.093976+0000","last_became_active":"2024-04-25T01:23:01.869982+0000","last_became_peered":"2024-04-25T01:23:01.869982+0000","last_unstale":"2024-04-25T01:25:25.093976+0000","last_undegraded":"2024-04-25T01:25:25.093976+0000","last_fullsized":"2024-04-25T01:25:25.093976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:17.294254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984043+0000","last_change":"2024-04-25T01:22:56.835134+0000","last_active":"2024-04-25T01:25:24.984043+0000","last_peered":"2024-04-25T01:25:24.984043+0000","last_clean":"2024-04-25T01:25:24.984043+0000","last_became_active":"2024-04-25T01:22:56.834207+0000","last_became_peered":"2024-04-25T01:22:56.834207+0000","last_unstale":"2024-04-25T01:25:24.984043+0000","last_undegraded":"2024-04-25T01:25:24.984043+0000","last_fullsized":"2024-04-25T01:25:24.984043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:07.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116949+0000","last_change":"2024-04-25T01:22:57.847693+0000","last_active":"2024-04-25T01:25:25.116949+0000","last_peered":"2024-04-25T01:25:25.116949+0000","last_clean":"2024-04-25T01:25:25.116949+0000","last_became_active":"2024-04-25T01:22:57.847211+0000","last_became_peered":"2024-04-25T01:22:57.847211+0000","last_unstale":"2024-04-25T01:25:25.116949+0000","last_undegraded":"2024-04-25T01:25:25.116949+0000","last_fullsized":"2024-04-25T01:25:25.116949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:48.295608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800279+0000","last_change":"2024-04-25T01:23:01.870340+0000","last_active":"2024-04-25T01:25:25.800279+0000","last_peered":"2024-04-25T01:25:25.800279+0000","last_clean":"2024-04-25T01:25:25.800279+0000","last_became_active":"2024-04-25T01:23:01.869662+0000","last_became_peered":"2024-04-25T01:23:01.869662+0000","last_unstale":"2024-04-25T01:25:25.800279+0000","last_undegraded":"2024-04-25T01:25:25.800279+0000","last_fullsized":"2024-04-25T01:25:25.800279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:05:57.313991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093495+0000","last_change":"2024-04-25T01:22:56.838146+0000","last_active":"2024-04-25T01:25:25.093495+0000","last_peered":"2024-04-25T01:25:25.093495+0000","last_clean":"2024-04-25T01:25:25.093495+0000","last_became_active":"2024-04-25T01:22:56.837815+0000","last_became_peered":"2024-04-25T01:22:56.837815+0000","last_unstale":"2024-04-25T01:25:25.093495+0000","last_undegraded":"2024-04-25T01:25:25.093495+0000","last_fullsized":"2024-04-25T01:25:25.093495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:14.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238786+0000","last_change":"2024-04-25T01:22:57.850840+0000","last_active":"2024-04-25T01:25:25.238786+0000","last_peered":"2024-04-25T01:25:25.238786+0000","last_clean":"2024-04-25T01:25:25.238786+0000","last_became_active":"2024-04-25T01:22:57.850255+0000","last_became_peered":"2024-04-25T01:22:57.850255+0000","last_unstale":"2024-04-25T01:25:25.238786+0000","last_undegraded":"2024-04-25T01:25:25.238786+0000","last_fullsized":"2024-04-25T01:25:25.238786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:25.887235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085296+0000","last_change":"2024-04-25T01:23:01.869396+0000","last_active":"2024-04-25T01:25:25.085296+0000","last_peered":"2024-04-25T01:25:25.085296+0000","last_clean":"2024-04-25T01:25:25.085296+0000","last_became_active":"2024-04-25T01:23:01.868777+0000","last_became_peered":"2024-04-25T01:23:01.868777+0000","last_unstale":"2024-04-25T01:25:25.085296+0000","last_undegraded":"2024-04-25T01:25:25.085296+0000","last_fullsized":"2024-04-25T01:25:25.085296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:37.276122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'4","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240662+0000","last_change":"2024-04-25T01:22:56.833289+0000","last_active":"2024-04-25T01:25:25.240662+0000","last_peered":"2024-04-25T01:25:25.240662+0000","last_clean":"2024-04-25T01:25:25.240662+0000","last_became_active":"2024-04-25T01:22:56.832353+0000","last_became_peered":"2024-04-25T01:22:56.832353+0000","last_unstale":"2024-04-25T01:25:25.240662+0000","last_undegraded":"2024-04-25T01:25:25.240662+0000","last_fullsized":"2024-04-25T01:25:25.240662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:18.251018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238649+0000","last_change":"2024-04-25T01:22:57.851974+0000","last_active":"2024-04-25T01:25:25.238649+0000","last_peered":"2024-04-25T01:25:25.238649+0000","last_clean":"2024-04-25T01:25:25.238649+0000","last_became_active":"2024-04-25T01:22:57.851573+0000","last_became_peered":"2024-04-25T01:22:57.851573+0000","last_unstale":"2024-04-25T01:25:25.238649+0000","last_undegraded":"2024-04-25T01:25:25.238649+0000","last_fullsized":"2024-04-25T01:25:25.238649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:27.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117895+0000","last_change":"2024-04-25T01:23:01.876980+0000","last_active":"2024-04-25T01:25:25.117895+0000","last_peered":"2024-04-25T01:25:25.117895+0000","last_clean":"2024-04-25T01:25:25.117895+0000","last_became_active":"2024-04-25T01:23:01.876548+0000","last_became_peered":"2024-04-25T01:23:01.876548+0000","last_unstale":"2024-04-25T01:25:25.117895+0000","last_undegraded":"2024-04-25T01:25:25.117895+0000","last_fullsized":"2024-04-25T01:25:25.117895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:13.470736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087000+0000","last_change":"2024-04-25T01:22:56.839694+0000","last_active":"2024-04-25T01:25:25.087000+0000","last_peered":"2024-04-25T01:25:25.087000+0000","last_clean":"2024-04-25T01:25:25.087000+0000","last_became_active":"2024-04-25T01:22:56.839367+0000","last_became_peered":"2024-04-25T01:22:56.839367+0000","last_unstale":"2024-04-25T01:25:25.087000+0000","last_undegraded":"2024-04-25T01:25:25.087000+0000","last_fullsized":"2024-04-25T01:25:25.087000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:49.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093465+0000","last_change":"2024-04-25T01:22:57.841526+0000","last_active":"2024-04-25T01:25:25.093465+0000","last_peered":"2024-04-25T01:25:25.093465+0000","last_clean":"2024-04-25T01:25:25.093465+0000","last_became_active":"2024-04-25T01:22:57.841192+0000","last_became_peered":"2024-04-25T01:22:57.841192+0000","last_unstale":"2024-04-25T01:25:25.093465+0000","last_undegraded":"2024-04-25T01:25:25.093465+0000","last_fullsized":"2024-04-25T01:25:25.093465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:38.958901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093301+0000","last_change":"2024-04-25T01:23:01.865486+0000","last_active":"2024-04-25T01:25:25.093301+0000","last_peered":"2024-04-25T01:25:25.093301+0000","last_clean":"2024-04-25T01:25:25.093301+0000","last_became_active":"2024-04-25T01:23:01.865068+0000","last_became_peered":"2024-04-25T01:23:01.865068+0000","last_unstale":"2024-04-25T01:25:25.093301+0000","last_undegraded":"2024-04-25T01:25:25.093301+0000","last_fullsized":"2024-04-25T01:25:25.093301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:50.432724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087557+0000","last_change":"2024-04-25T01:22:56.841507+0000","last_active":"2024-04-25T01:25:25.087557+0000","last_peered":"2024-04-25T01:25:25.087557+0000","last_clean":"2024-04-25T01:25:25.087557+0000","last_became_active":"2024-04-25T01:22:56.839970+0000","last_became_peered":"2024-04-25T01:22:56.839970+0000","last_unstale":"2024-04-25T01:25:25.087557+0000","last_undegraded":"2024-04-25T01:25:25.087557+0000","last_fullsized":"2024-04-25T01:25:25.087557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:37.670052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800378+0000","last_change":"2024-04-25T01:22:57.850853+0000","last_active":"2024-04-25T01:25:25.800378+0000","last_peered":"2024-04-25T01:25:25.800378+0000","last_clean":"2024-04-25T01:25:25.800378+0000","last_became_active":"2024-04-25T01:22:57.850498+0000","last_became_peered":"2024-04-25T01:22:57.850498+0000","last_unstale":"2024-04-25T01:25:25.800378+0000","last_undegraded":"2024-04-25T01:25:25.800378+0000","last_fullsized":"2024-04-25T01:25:25.800378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:13.271048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087021+0000","last_change":"2024-04-25T01:23:01.874254+0000","last_active":"2024-04-25T01:25:25.087021+0000","last_peered":"2024-04-25T01:25:25.087021+0000","last_clean":"2024-04-25T01:25:25.087021+0000","last_became_active":"2024-04-25T01:23:01.873711+0000","last_became_peered":"2024-04-25T01:23:01.873711+0000","last_unstale":"2024-04-25T01:25:25.087021+0000","last_undegraded":"2024-04-25T01:25:25.087021+0000","last_fullsized":"2024-04-25T01:25:25.087021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:13.510061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085035+0000","last_change":"2024-04-25T01:22:56.827137+0000","last_active":"2024-04-25T01:25:25.085035+0000","last_peered":"2024-04-25T01:25:25.085035+0000","last_clean":"2024-04-25T01:25:25.085035+0000","last_became_active":"2024-04-25T01:22:56.826769+0000","last_became_peered":"2024-04-25T01:22:56.826769+0000","last_unstale":"2024-04-25T01:25:25.085035+0000","last_undegraded":"2024-04-25T01:25:25.085035+0000","last_fullsized":"2024-04-25T01:25:25.085035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:21.072643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983993+0000","last_change":"2024-04-25T01:22:57.843713+0000","last_active":"2024-04-25T01:25:24.983993+0000","last_peered":"2024-04-25T01:25:24.983993+0000","last_clean":"2024-04-25T01:25:24.983993+0000","last_became_active":"2024-04-25T01:22:57.843384+0000","last_became_peered":"2024-04-25T01:22:57.843384+0000","last_unstale":"2024-04-25T01:25:24.983993+0000","last_undegraded":"2024-04-25T01:25:24.983993+0000","last_fullsized":"2024-04-25T01:25:24.983993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:18.772379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117639+0000","last_change":"2024-04-25T01:23:01.874150+0000","last_active":"2024-04-25T01:25:25.117639+0000","last_peered":"2024-04-25T01:25:25.117639+0000","last_clean":"2024-04-25T01:25:25.117639+0000","last_became_active":"2024-04-25T01:23:01.873718+0000","last_became_peered":"2024-04-25T01:23:01.873718+0000","last_unstale":"2024-04-25T01:25:25.117639+0000","last_undegraded":"2024-04-25T01:25:25.117639+0000","last_fullsized":"2024-04-25T01:25:25.117639+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:26.317703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238669+0000","last_change":"2024-04-25T01:22:56.843605+0000","last_active":"2024-04-25T01:25:25.238669+0000","last_peered":"2024-04-25T01:25:25.238669+0000","last_clean":"2024-04-25T01:25:25.238669+0000","last_became_active":"2024-04-25T01:22:56.843313+0000","last_became_peered":"2024-04-25T01:22:56.843313+0000","last_unstale":"2024-04-25T01:25:25.238669+0000","last_undegraded":"2024-04-25T01:25:25.238669+0000","last_fullsized":"2024-04-25T01:25:25.238669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:41.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983293+0000","last_change":"2024-04-25T01:23:47.005378+0000","last_active":"2024-04-25T01:25:24.983293+0000","last_peered":"2024-04-25T01:25:24.983293+0000","last_clean":"2024-04-25T01:25:24.983293+0000","last_became_active":"2024-04-25T01:23:47.004962+0000","last_became_peered":"2024-04-25T01:23:47.004962+0000","last_unstale":"2024-04-25T01:25:24.983293+0000","last_undegraded":"2024-04-25T01:25:24.983293+0000","last_fullsized":"2024-04-25T01:25:24.983293+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:28.548285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117894+0000","last_change":"2024-04-25T01:23:01.872792+0000","last_active":"2024-04-25T01:25:25.117894+0000","last_peered":"2024-04-25T01:25:25.117894+0000","last_clean":"2024-04-25T01:25:25.117894+0000","last_became_active":"2024-04-25T01:23:01.872391+0000","last_became_peered":"2024-04-25T01:23:01.872391+0000","last_unstale":"2024-04-25T01:25:25.117894+0000","last_undegraded":"2024-04-25T01:25:25.117894+0000","last_fullsized":"2024-04-25T01:25:25.117894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:13.088996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093632+0000","last_change":"2024-04-25T01:22:57.843131+0000","last_active":"2024-04-25T01:25:25.093632+0000","last_peered":"2024-04-25T01:25:25.093632+0000","last_clean":"2024-04-25T01:25:25.093632+0000","last_became_active":"2024-04-25T01:22:57.842540+0000","last_became_peered":"2024-04-25T01:22:57.842540+0000","last_unstale":"2024-04-25T01:25:25.093632+0000","last_undegraded":"2024-04-25T01:25:25.093632+0000","last_fullsized":"2024-04-25T01:25:25.093632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:51.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117205+0000","last_change":"2024-04-25T01:22:56.844610+0000","last_active":"2024-04-25T01:25:25.117205+0000","last_peered":"2024-04-25T01:25:25.117205+0000","last_clean":"2024-04-25T01:25:25.117205+0000","last_became_active":"2024-04-25T01:22:56.844237+0000","last_became_peered":"2024-04-25T01:22:56.844237+0000","last_unstale":"2024-04-25T01:25:25.117205+0000","last_undegraded":"2024-04-25T01:25:25.117205+0000","last_fullsized":"2024-04-25T01:25:25.117205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:09.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238830+0000","last_change":"2024-04-25T01:23:01.865711+0000","last_active":"2024-04-25T01:25:25.238830+0000","last_peered":"2024-04-25T01:25:25.238830+0000","last_clean":"2024-04-25T01:25:25.238830+0000","last_became_active":"2024-04-25T01:23:01.865317+0000","last_became_peered":"2024-04-25T01:23:01.865317+0000","last_unstale":"2024-04-25T01:25:25.238830+0000","last_undegraded":"2024-04-25T01:25:25.238830+0000","last_fullsized":"2024-04-25T01:25:25.238830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:08.995490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983908+0000","last_change":"2024-04-25T01:22:57.851623+0000","last_active":"2024-04-25T01:25:24.983908+0000","last_peered":"2024-04-25T01:25:24.983908+0000","last_clean":"2024-04-25T01:25:24.983908+0000","last_became_active":"2024-04-25T01:22:57.850616+0000","last_became_peered":"2024-04-25T01:22:57.850616+0000","last_unstale":"2024-04-25T01:25:24.983908+0000","last_undegraded":"2024-04-25T01:25:24.983908+0000","last_fullsized":"2024-04-25T01:25:24.983908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:27.447996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118832+0000","last_change":"2024-04-25T01:22:56.832094+0000","last_active":"2024-04-25T01:25:25.118832+0000","last_peered":"2024-04-25T01:25:25.118832+0000","last_clean":"2024-04-25T01:25:25.118832+0000","last_became_active":"2024-04-25T01:22:56.831743+0000","last_became_peered":"2024-04-25T01:22:56.831743+0000","last_unstale":"2024-04-25T01:25:25.118832+0000","last_undegraded":"2024-04-25T01:25:25.118832+0000","last_fullsized":"2024-04-25T01:25:25.118832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:06.116081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238473+0000","last_change":"2024-04-25T01:21:40.899517+0000","last_active":"2024-04-25T01:25:25.238473+0000","last_peered":"2024-04-25T01:25:25.238473+0000","last_clean":"2024-04-25T01:25:25.238473+0000","last_became_active":"2024-04-25T01:21:40.899043+0000","last_became_peered":"2024-04-25T01:21:40.899043+0000","last_unstale":"2024-04-25T01:25:25.238473+0000","last_undegraded":"2024-04-25T01:25:25.238473+0000","last_fullsized":"2024-04-25T01:25:25.238473+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_clean_scrub_stamp":"2024-04-25T01:18:12.053452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:41.234852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093701+0000","last_change":"2024-04-25T01:23:01.878334+0000","last_active":"2024-04-25T01:25:25.093701+0000","last_peered":"2024-04-25T01:25:25.093701+0000","last_clean":"2024-04-25T01:25:25.093701+0000","last_became_active":"2024-04-25T01:23:01.877955+0000","last_became_peered":"2024-04-25T01:23:01.877955+0000","last_unstale":"2024-04-25T01:25:25.093701+0000","last_undegraded":"2024-04-25T01:25:25.093701+0000","last_fullsized":"2024-04-25T01:25:25.093701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:33.339430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116577+0000","last_change":"2024-04-25T01:22:57.843020+0000","last_active":"2024-04-25T01:25:25.116577+0000","last_peered":"2024-04-25T01:25:25.116577+0000","last_clean":"2024-04-25T01:25:25.116577+0000","last_became_active":"2024-04-25T01:22:57.842671+0000","last_became_peered":"2024-04-25T01:22:57.842671+0000","last_unstale":"2024-04-25T01:25:25.116577+0000","last_undegraded":"2024-04-25T01:25:25.116577+0000","last_fullsized":"2024-04-25T01:25:25.116577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:53.831432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238821+0000","last_change":"2024-04-25T01:22:56.836106+0000","last_active":"2024-04-25T01:25:25.238821+0000","last_peered":"2024-04-25T01:25:25.238821+0000","last_clean":"2024-04-25T01:25:25.238821+0000","last_became_active":"2024-04-25T01:22:56.835814+0000","last_became_peered":"2024-04-25T01:22:56.835814+0000","last_unstale":"2024-04-25T01:25:25.238821+0000","last_undegraded":"2024-04-25T01:25:25.238821+0000","last_fullsized":"2024-04-25T01:25:25.238821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:18.004312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800091+0000","last_change":"2024-04-25T01:23:01.870723+0000","last_active":"2024-04-25T01:25:25.800091+0000","last_peered":"2024-04-25T01:25:25.800091+0000","last_clean":"2024-04-25T01:25:25.800091+0000","last_became_active":"2024-04-25T01:23:01.870205+0000","last_became_peered":"2024-04-25T01:23:01.870205+0000","last_unstale":"2024-04-25T01:25:25.800091+0000","last_undegraded":"2024-04-25T01:25:25.800091+0000","last_fullsized":"2024-04-25T01:25:25.800091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:47.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800282+0000","last_change":"2024-04-25T01:24:47.497526+0000","last_active":"2024-04-25T01:25:25.800282+0000","last_peered":"2024-04-25T01:25:25.800282+0000","last_clean":"2024-04-25T01:25:25.800282+0000","last_became_active":"2024-04-25T01:24:47.493380+0000","last_became_peered":"2024-04-25T01:24:47.493380+0000","last_unstale":"2024-04-25T01:25:25.800282+0000","last_undegraded":"2024-04-25T01:25:25.800282+0000","last_fullsized":"2024-04-25T01:25:25.800282+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:40:22.024954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117796+0000","last_change":"2024-04-25T01:22:57.841727+0000","last_active":"2024-04-25T01:25:25.117796+0000","last_peered":"2024-04-25T01:25:25.117796+0000","last_clean":"2024-04-25T01:25:25.117796+0000","last_became_active":"2024-04-25T01:22:57.841366+0000","last_became_peered":"2024-04-25T01:22:57.841366+0000","last_unstale":"2024-04-25T01:25:25.117796+0000","last_undegraded":"2024-04-25T01:25:25.117796+0000","last_fullsized":"2024-04-25T01:25:25.117796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:20.585916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983754+0000","last_change":"2024-04-25T01:23:01.869814+0000","last_active":"2024-04-25T01:25:24.983754+0000","last_peered":"2024-04-25T01:25:24.983754+0000","last_clean":"2024-04-25T01:25:24.983754+0000","last_became_active":"2024-04-25T01:23:01.869387+0000","last_became_peered":"2024-04-25T01:23:01.869387+0000","last_unstale":"2024-04-25T01:25:24.983754+0000","last_undegraded":"2024-04-25T01:25:24.983754+0000","last_fullsized":"2024-04-25T01:25:24.983754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:18:45.098295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'45","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.199682+0000","last_change":"2024-04-25T01:22:56.838506+0000","last_active":"2024-04-25T01:25:25.199682+0000","last_peered":"2024-04-25T01:25:25.199682+0000","last_clean":"2024-04-25T01:25:25.199682+0000","last_became_active":"2024-04-25T01:22:56.838185+0000","last_became_peered":"2024-04-25T01:22:56.838185+0000","last_unstale":"2024-04-25T01:25:25.199682+0000","last_undegraded":"2024-04-25T01:25:25.199682+0000","last_fullsized":"2024-04-25T01:25:25.199682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:35.306231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984432+0000","last_change":"2024-04-25T01:22:57.846544+0000","last_active":"2024-04-25T01:25:24.984432+0000","last_peered":"2024-04-25T01:25:24.984432+0000","last_clean":"2024-04-25T01:25:24.984432+0000","last_became_active":"2024-04-25T01:22:57.843507+0000","last_became_peered":"2024-04-25T01:22:57.843507+0000","last_unstale":"2024-04-25T01:25:24.984432+0000","last_undegraded":"2024-04-25T01:25:24.984432+0000","last_fullsized":"2024-04-25T01:25:24.984432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:30.600317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983013+0000","last_change":"2024-04-25T01:23:01.863986+0000","last_active":"2024-04-25T01:25:24.983013+0000","last_peered":"2024-04-25T01:25:24.983013+0000","last_clean":"2024-04-25T01:25:24.983013+0000","last_became_active":"2024-04-25T01:23:01.863397+0000","last_became_peered":"2024-04-25T01:23:01.863397+0000","last_unstale":"2024-04-25T01:25:24.983013+0000","last_undegraded":"2024-04-25T01:25:24.983013+0000","last_fullsized":"2024-04-25T01:25:24.983013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:45:56.229349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119308+0000","last_change":"2024-04-25T01:22:56.838201+0000","last_active":"2024-04-25T01:25:25.119308+0000","last_peered":"2024-04-25T01:25:25.119308+0000","last_clean":"2024-04-25T01:25:25.119308+0000","last_became_active":"2024-04-25T01:22:56.837890+0000","last_became_peered":"2024-04-25T01:22:56.837890+0000","last_unstale":"2024-04-25T01:25:25.119308+0000","last_undegraded":"2024-04-25T01:25:25.119308+0000","last_fullsized":"2024-04-25T01:25:25.119308+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:17.418112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241186+0000","last_change":"2024-04-25T01:22:57.853923+0000","last_active":"2024-04-25T01:25:25.241186+0000","last_peered":"2024-04-25T01:25:25.241186+0000","last_clean":"2024-04-25T01:25:25.241186+0000","last_became_active":"2024-04-25T01:22:57.853566+0000","last_became_peered":"2024-04-25T01:22:57.853566+0000","last_unstale":"2024-04-25T01:25:25.241186+0000","last_undegraded":"2024-04-25T01:25:25.241186+0000","last_fullsized":"2024-04-25T01:25:25.241186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:36.509650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119340+0000","last_change":"2024-04-25T01:23:47.038657+0000","last_active":"2024-04-25T01:25:25.119340+0000","last_peered":"2024-04-25T01:25:25.119340+0000","last_clean":"2024-04-25T01:25:25.119340+0000","last_became_active":"2024-04-25T01:23:47.038168+0000","last_became_peered":"2024-04-25T01:23:47.038168+0000","last_unstale":"2024-04-25T01:25:25.119340+0000","last_undegraded":"2024-04-25T01:25:25.119340+0000","last_fullsized":"2024-04-25T01:25:25.119340+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:59.844333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239155+0000","last_change":"2024-04-25T01:22:56.828651+0000","last_active":"2024-04-25T01:25:25.239155+0000","last_peered":"2024-04-25T01:25:25.239155+0000","last_clean":"2024-04-25T01:25:25.239155+0000","last_became_active":"2024-04-25T01:22:56.828186+0000","last_became_peered":"2024-04-25T01:22:56.828186+0000","last_unstale":"2024-04-25T01:25:25.239155+0000","last_undegraded":"2024-04-25T01:25:25.239155+0000","last_fullsized":"2024-04-25T01:25:25.239155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:56.020822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.086946+0000","last_change":"2024-04-25T01:22:57.849768+0000","last_active":"2024-04-25T01:25:25.086946+0000","last_peered":"2024-04-25T01:25:25.086946+0000","last_clean":"2024-04-25T01:25:25.086946+0000","last_became_active":"2024-04-25T01:22:57.849418+0000","last_became_peered":"2024-04-25T01:22:57.849418+0000","last_unstale":"2024-04-25T01:25:25.086946+0000","last_undegraded":"2024-04-25T01:25:25.086946+0000","last_fullsized":"2024-04-25T01:25:25.086946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:41.292915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087501+0000","last_change":"2024-04-25T01:23:47.148790+0000","last_active":"2024-04-25T01:25:25.087501+0000","last_peered":"2024-04-25T01:25:25.087501+0000","last_clean":"2024-04-25T01:25:25.087501+0000","last_became_active":"2024-04-25T01:23:47.148180+0000","last_became_peered":"2024-04-25T01:23:47.148180+0000","last_unstale":"2024-04-25T01:25:25.087501+0000","last_undegraded":"2024-04-25T01:25:25.087501+0000","last_fullsized":"2024-04-25T01:25:25.087501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:08.296696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983834+0000","last_change":"2024-04-25T01:22:56.844672+0000","last_active":"2024-04-25T01:25:24.983834+0000","last_peered":"2024-04-25T01:25:24.983834+0000","last_clean":"2024-04-25T01:25:24.983834+0000","last_became_active":"2024-04-25T01:22:56.844141+0000","last_became_peered":"2024-04-25T01:22:56.844141+0000","last_unstale":"2024-04-25T01:25:24.983834+0000","last_undegraded":"2024-04-25T01:25:24.983834+0000","last_fullsized":"2024-04-25T01:25:24.983834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:14.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800019+0000","last_change":"2024-04-25T01:22:57.845006+0000","last_active":"2024-04-25T01:25:25.800019+0000","last_peered":"2024-04-25T01:25:25.800019+0000","last_clean":"2024-04-25T01:25:25.800019+0000","last_became_active":"2024-04-25T01:22:57.844672+0000","last_became_peered":"2024-04-25T01:22:57.844672+0000","last_unstale":"2024-04-25T01:25:25.800019+0000","last_undegraded":"2024-04-25T01:25:25.800019+0000","last_fullsized":"2024-04-25T01:25:25.800019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:18.238512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119458+0000","last_change":"2024-04-25T01:23:01.869438+0000","last_active":"2024-04-25T01:25:25.119458+0000","last_peered":"2024-04-25T01:25:25.119458+0000","last_clean":"2024-04-25T01:25:25.119458+0000","last_became_active":"2024-04-25T01:23:01.868985+0000","last_became_peered":"2024-04-25T01:23:01.868985+0000","last_unstale":"2024-04-25T01:25:25.119458+0000","last_undegraded":"2024-04-25T01:25:25.119458+0000","last_fullsized":"2024-04-25T01:25:25.119458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:35.509098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983953+0000","last_change":"2024-04-25T01:22:56.834475+0000","last_active":"2024-04-25T01:25:24.983953+0000","last_peered":"2024-04-25T01:25:24.983953+0000","last_clean":"2024-04-25T01:25:24.983953+0000","last_became_active":"2024-04-25T01:22:56.834076+0000","last_became_peered":"2024-04-25T01:22:56.834076+0000","last_unstale":"2024-04-25T01:25:24.983953+0000","last_undegraded":"2024-04-25T01:25:24.983953+0000","last_fullsized":"2024-04-25T01:25:24.983953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:55:47.464009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984196+0000","last_change":"2024-04-25T01:22:57.860549+0000","last_active":"2024-04-25T01:25:24.984196+0000","last_peered":"2024-04-25T01:25:24.984196+0000","last_clean":"2024-04-25T01:25:24.984196+0000","last_became_active":"2024-04-25T01:22:57.860226+0000","last_became_peered":"2024-04-25T01:22:57.860226+0000","last_unstale":"2024-04-25T01:25:24.984196+0000","last_undegraded":"2024-04-25T01:25:24.984196+0000","last_fullsized":"2024-04-25T01:25:24.984196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:10.663508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119226+0000","last_change":"2024-04-25T01:23:01.867602+0000","last_active":"2024-04-25T01:25:25.119226+0000","last_peered":"2024-04-25T01:25:25.119226+0000","last_clean":"2024-04-25T01:25:25.119226+0000","last_became_active":"2024-04-25T01:23:01.867150+0000","last_became_peered":"2024-04-25T01:23:01.867150+0000","last_unstale":"2024-04-25T01:25:25.119226+0000","last_undegraded":"2024-04-25T01:25:25.119226+0000","last_fullsized":"2024-04-25T01:25:25.119226+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:34.571031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087024+0000","last_change":"2024-04-25T01:22:56.841082+0000","last_active":"2024-04-25T01:25:25.087024+0000","last_peered":"2024-04-25T01:25:25.087024+0000","last_clean":"2024-04-25T01:25:25.087024+0000","last_became_active":"2024-04-25T01:22:56.839856+0000","last_became_peered":"2024-04-25T01:22:56.839856+0000","last_unstale":"2024-04-25T01:25:25.087024+0000","last_undegraded":"2024-04-25T01:25:25.087024+0000","last_fullsized":"2024-04-25T01:25:25.087024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:14.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238762+0000","last_change":"2024-04-25T01:22:57.852242+0000","last_active":"2024-04-25T01:25:25.238762+0000","last_peered":"2024-04-25T01:25:25.238762+0000","last_clean":"2024-04-25T01:25:25.238762+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:25:25.238762+0000","last_undegraded":"2024-04-25T01:25:25.238762+0000","last_fullsized":"2024-04-25T01:25:25.238762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:38:02.810025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238989+0000","last_change":"2024-04-25T01:23:01.871074+0000","last_active":"2024-04-25T01:25:25.238989+0000","last_peered":"2024-04-25T01:25:25.238989+0000","last_clean":"2024-04-25T01:25:25.238989+0000","last_became_active":"2024-04-25T01:23:01.870570+0000","last_became_peered":"2024-04-25T01:23:01.870570+0000","last_unstale":"2024-04-25T01:25:25.238989+0000","last_undegraded":"2024-04-25T01:25:25.238989+0000","last_fullsized":"2024-04-25T01:25:25.238989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:14.476400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118784+0000","last_change":"2024-04-25T01:23:47.011202+0000","last_active":"2024-04-25T01:25:25.118784+0000","last_peered":"2024-04-25T01:25:25.118784+0000","last_clean":"2024-04-25T01:25:25.118784+0000","last_became_active":"2024-04-25T01:23:47.006425+0000","last_became_peered":"2024-04-25T01:23:47.006425+0000","last_unstale":"2024-04-25T01:25:25.118784+0000","last_undegraded":"2024-04-25T01:25:25.118784+0000","last_fullsized":"2024-04-25T01:25:25.118784+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:51:46.791184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239193+0000","last_change":"2024-04-25T01:22:57.843173+0000","last_active":"2024-04-25T01:25:25.239193+0000","last_peered":"2024-04-25T01:25:25.239193+0000","last_clean":"2024-04-25T01:25:25.239193+0000","last_became_active":"2024-04-25T01:22:57.842734+0000","last_became_peered":"2024-04-25T01:22:57.842734+0000","last_unstale":"2024-04-25T01:25:25.239193+0000","last_undegraded":"2024-04-25T01:25:25.239193+0000","last_fullsized":"2024-04-25T01:25:25.239193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:33.197493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240612+0000","last_change":"2024-04-25T01:23:01.864068+0000","last_active":"2024-04-25T01:25:25.240612+0000","last_peered":"2024-04-25T01:25:25.240612+0000","last_clean":"2024-04-25T01:25:25.240612+0000","last_became_active":"2024-04-25T01:23:01.863640+0000","last_became_peered":"2024-04-25T01:23:01.863640+0000","last_unstale":"2024-04-25T01:25:25.240612+0000","last_undegraded":"2024-04-25T01:25:25.240612+0000","last_fullsized":"2024-04-25T01:25:25.240612+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:26.925012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.194844+0000","last_change":"2024-04-25T01:22:56.837969+0000","last_active":"2024-04-25T01:25:25.194844+0000","last_peered":"2024-04-25T01:25:25.194844+0000","last_clean":"2024-04-25T01:25:25.194844+0000","last_became_active":"2024-04-25T01:22:56.837641+0000","last_became_peered":"2024-04-25T01:22:56.837641+0000","last_unstale":"2024-04-25T01:25:25.194844+0000","last_undegraded":"2024-04-25T01:25:25.194844+0000","last_fullsized":"2024-04-25T01:25:25.194844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:04.376038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238624+0000","last_change":"2024-04-25T01:22:57.851057+0000","last_active":"2024-04-25T01:25:25.238624+0000","last_peered":"2024-04-25T01:25:25.238624+0000","last_clean":"2024-04-25T01:25:25.238624+0000","last_became_active":"2024-04-25T01:22:57.850278+0000","last_became_peered":"2024-04-25T01:22:57.850278+0000","last_unstale":"2024-04-25T01:25:25.238624+0000","last_undegraded":"2024-04-25T01:25:25.238624+0000","last_fullsized":"2024-04-25T01:25:25.238624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:15.332193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093137+0000","last_change":"2024-04-25T01:23:01.866673+0000","last_active":"2024-04-25T01:25:25.093137+0000","last_peered":"2024-04-25T01:25:25.093137+0000","last_clean":"2024-04-25T01:25:25.093137+0000","last_became_active":"2024-04-25T01:23:01.866210+0000","last_became_peered":"2024-04-25T01:23:01.866210+0000","last_unstale":"2024-04-25T01:25:25.093137+0000","last_undegraded":"2024-04-25T01:25:25.093137+0000","last_fullsized":"2024-04-25T01:25:25.093137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:20.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117961+0000","last_change":"2024-04-25T01:22:56.844590+0000","last_active":"2024-04-25T01:25:25.117961+0000","last_peered":"2024-04-25T01:25:25.117961+0000","last_clean":"2024-04-25T01:25:25.117961+0000","last_became_active":"2024-04-25T01:22:56.844279+0000","last_became_peered":"2024-04-25T01:22:56.844279+0000","last_unstale":"2024-04-25T01:25:25.117961+0000","last_undegraded":"2024-04-25T01:25:25.117961+0000","last_fullsized":"2024-04-25T01:25:25.117961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:47.155832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239487+0000","last_change":"2024-04-25T01:22:57.858850+0000","last_active":"2024-04-25T01:25:25.239487+0000","last_peered":"2024-04-25T01:25:25.239487+0000","last_clean":"2024-04-25T01:25:25.239487+0000","last_became_active":"2024-04-25T01:22:57.858498+0000","last_became_peered":"2024-04-25T01:22:57.858498+0000","last_unstale":"2024-04-25T01:25:25.239487+0000","last_undegraded":"2024-04-25T01:25:25.239487+0000","last_fullsized":"2024-04-25T01:25:25.239487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:04.836062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800109+0000","last_change":"2024-04-25T01:23:47.007111+0000","last_active":"2024-04-25T01:25:25.800109+0000","last_peered":"2024-04-25T01:25:25.800109+0000","last_clean":"2024-04-25T01:25:25.800109+0000","last_became_active":"2024-04-25T01:23:47.006699+0000","last_became_peered":"2024-04-25T01:23:47.006699+0000","last_unstale":"2024-04-25T01:25:25.800109+0000","last_undegraded":"2024-04-25T01:25:25.800109+0000","last_fullsized":"2024-04-25T01:25:25.800109+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:57:43.608183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983737+0000","last_change":"2024-04-25T01:23:47.488742+0000","last_active":"2024-04-25T01:25:24.983737+0000","last_peered":"2024-04-25T01:25:24.983737+0000","last_clean":"2024-04-25T01:25:24.983737+0000","last_became_active":"2024-04-25T01:23:47.488193+0000","last_became_peered":"2024-04-25T01:23:47.488193+0000","last_unstale":"2024-04-25T01:25:24.983737+0000","last_undegraded":"2024-04-25T01:25:24.983737+0000","last_fullsized":"2024-04-25T01:25:24.983737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:02.658395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983788+0000","last_change":"2024-04-25T01:22:57.851138+0000","last_active":"2024-04-25T01:25:24.983788+0000","last_peered":"2024-04-25T01:25:24.983788+0000","last_clean":"2024-04-25T01:25:24.983788+0000","last_became_active":"2024-04-25T01:22:57.849107+0000","last_became_peered":"2024-04-25T01:22:57.849107+0000","last_unstale":"2024-04-25T01:25:24.983788+0000","last_undegraded":"2024-04-25T01:25:24.983788+0000","last_fullsized":"2024-04-25T01:25:24.983788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:47.665356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240803+0000","last_change":"2024-04-25T01:23:47.488162+0000","last_active":"2024-04-25T01:25:25.240803+0000","last_peered":"2024-04-25T01:25:25.240803+0000","last_clean":"2024-04-25T01:25:25.240803+0000","last_became_active":"2024-04-25T01:23:47.487731+0000","last_became_peered":"2024-04-25T01:23:47.487731+0000","last_unstale":"2024-04-25T01:25:25.240803+0000","last_undegraded":"2024-04-25T01:25:25.240803+0000","last_fullsized":"2024-04-25T01:25:25.240803+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:14.716313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117755+0000","last_change":"2024-04-25T01:22:57.851715+0000","last_active":"2024-04-25T01:25:25.117755+0000","last_peered":"2024-04-25T01:25:25.117755+0000","last_clean":"2024-04-25T01:25:25.117755+0000","last_became_active":"2024-04-25T01:22:57.847287+0000","last_became_peered":"2024-04-25T01:22:57.847287+0000","last_unstale":"2024-04-25T01:25:25.117755+0000","last_undegraded":"2024-04-25T01:25:25.117755+0000","last_fullsized":"2024-04-25T01:25:25.117755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.749931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800497+0000","last_change":"2024-04-25T01:22:57.846075+0000","last_active":"2024-04-25T01:25:25.800497+0000","last_peered":"2024-04-25T01:25:25.800497+0000","last_clean":"2024-04-25T01:25:25.800497+0000","last_became_active":"2024-04-25T01:22:57.845388+0000","last_became_peered":"2024-04-25T01:22:57.845388+0000","last_unstale":"2024-04-25T01:25:25.800497+0000","last_undegraded":"2024-04-25T01:25:25.800497+0000","last_fullsized":"2024-04-25T01:25:25.800497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:19.859806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984966+0000","last_change":"2024-04-25T01:22:57.851446+0000","last_active":"2024-04-25T01:25:24.984966+0000","last_peered":"2024-04-25T01:25:24.984966+0000","last_clean":"2024-04-25T01:25:24.984966+0000","last_became_active":"2024-04-25T01:22:57.848729+0000","last_became_peered":"2024-04-25T01:22:57.848729+0000","last_unstale":"2024-04-25T01:25:24.984966+0000","last_undegraded":"2024-04-25T01:25:24.984966+0000","last_fullsized":"2024-04-25T01:25:24.984966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:49.721024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984555+0000","last_change":"2024-04-25T01:22:57.847598+0000","last_active":"2024-04-25T01:25:24.984555+0000","last_peered":"2024-04-25T01:25:24.984555+0000","last_clean":"2024-04-25T01:25:24.984555+0000","last_became_active":"2024-04-25T01:22:57.847232+0000","last_became_peered":"2024-04-25T01:22:57.847232+0000","last_unstale":"2024-04-25T01:25:24.984555+0000","last_undegraded":"2024-04-25T01:25:24.984555+0000","last_fullsized":"2024-04-25T01:25:24.984555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.227700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093300+0000","last_change":"2024-04-25T01:22:57.842881+0000","last_active":"2024-04-25T01:25:25.093300+0000","last_peered":"2024-04-25T01:25:25.093300+0000","last_clean":"2024-04-25T01:25:25.093300+0000","last_became_active":"2024-04-25T01:22:57.842108+0000","last_became_peered":"2024-04-25T01:22:57.842108+0000","last_unstale":"2024-04-25T01:25:25.093300+0000","last_undegraded":"2024-04-25T01:25:25.093300+0000","last_fullsized":"2024-04-25T01:25:25.093300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:53.208188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983828+0000","last_change":"2024-04-25T01:22:57.860047+0000","last_active":"2024-04-25T01:25:24.983828+0000","last_peered":"2024-04-25T01:25:24.983828+0000","last_clean":"2024-04-25T01:25:24.983828+0000","last_became_active":"2024-04-25T01:22:57.859702+0000","last_became_peered":"2024-04-25T01:22:57.859702+0000","last_unstale":"2024-04-25T01:25:24.983828+0000","last_undegraded":"2024-04-25T01:25:24.983828+0000","last_fullsized":"2024-04-25T01:25:24.983828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:00.684055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239212+0000","last_change":"2024-04-25T01:22:57.847996+0000","last_active":"2024-04-25T01:25:25.239212+0000","last_peered":"2024-04-25T01:25:25.239212+0000","last_clean":"2024-04-25T01:25:25.239212+0000","last_became_active":"2024-04-25T01:22:57.847561+0000","last_became_peered":"2024-04-25T01:22:57.847561+0000","last_unstale":"2024-04-25T01:25:25.239212+0000","last_undegraded":"2024-04-25T01:25:25.239212+0000","last_fullsized":"2024-04-25T01:25:25.239212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:16.327558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117555+0000","last_change":"2024-04-25T01:22:57.845216+0000","last_active":"2024-04-25T01:25:25.117555+0000","last_peered":"2024-04-25T01:25:25.117555+0000","last_clean":"2024-04-25T01:25:25.117555+0000","last_became_active":"2024-04-25T01:22:57.844883+0000","last_became_peered":"2024-04-25T01:22:57.844883+0000","last_unstale":"2024-04-25T01:25:25.117555+0000","last_undegraded":"2024-04-25T01:25:25.117555+0000","last_fullsized":"2024-04-25T01:25:25.117555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:33.782048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800342+0000","last_change":"2024-04-25T01:22:57.853251+0000","last_active":"2024-04-25T01:25:25.800342+0000","last_peered":"2024-04-25T01:25:25.800342+0000","last_clean":"2024-04-25T01:25:25.800342+0000","last_became_active":"2024-04-25T01:22:57.852878+0000","last_became_peered":"2024-04-25T01:22:57.852878+0000","last_unstale":"2024-04-25T01:25:25.800342+0000","last_undegraded":"2024-04-25T01:25:25.800342+0000","last_fullsized":"2024-04-25T01:25:25.800342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:42.487837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107295,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599417,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":850,"num_read_kb":535,"num_write":289,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":64209,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":1798251,"data_compressed":89584,"data_compressed_allocated":118784,"data_compressed_original":1626112,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":529593,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32348,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712804,"statfs":{"total":95995035648,"available":95961911296,"internally_reserved":0,"allocated":1200128,"data_stored":1318507,"data_compressed":73161,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":927119,"data_compressed":48759,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":922921,"data_compressed":48759,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1126559,"data_compressed":65378,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":515088,"data_compressed":26623,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594570,"data_compressed":31240,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526371,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":526193,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1049993,"data_compressed":54915,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":600283,"data_compressed":40037,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1126400,"data_stored":1253356,"data_compressed":71534,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69677,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":71294,"data_compressed":10424,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133999,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203246,"data_compressed":7783,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:25:30.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:25:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 -- 172.21.15.27:0/2872723310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640100620 msgr2=0x7f3640100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 --2- 172.21.15.27:0/2872723310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640100620 0x7f3640100aa0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3628009930 tx=0x7f362802a420 comp rx=0 tx=0).stop 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 -- 172.21.15.27:0/2872723310 shutdown_connections 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 --2- 172.21.15.27:0/2872723310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3640100fe0 0x7f364010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 --2- 172.21.15.27:0/2872723310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640100620 0x7f3640100aa0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 --2- 172.21.15.27:0/2872723310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3640106630 0x7f3640106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 -- 172.21.15.27:0/2872723310 >> 172.21.15.27:0/2872723310 conn(0x7f3640077740 msgr2=0x7f3640077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 -- 172.21.15.27:0/2872723310 shutdown_connections 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.218+0000 7f364515a640 1 -- 172.21.15.27:0/2872723310 wait complete. 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.219+0000 7f364515a640 1 Processor -- start 2024-04-25T01:25:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.219+0000 7f364515a640 1 -- start start 2024-04-25T01:25:30.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f364515a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3640100620 0x7f364010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f364515a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3640100fe0 0x7f364010e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f364515a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106630 0x7f364010ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f364515a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3640111b40 con 0x7f3640106630 2024-04-25T01:25:30.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f364515a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f36401119c0 con 0x7f3640100fe0 2024-04-25T01:25:30.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f364515a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3640111cc0 con 0x7f3640100620 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106630 0x7f364010ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106630 0x7f364010ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50376/0 (socket says 172.21.15.27:50376) 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 -- 172.21.15.27:0/3642347843 learned_addr learned my addr 172.21.15.27:0/3642347843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363ed9d640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3640100620 0x7f364010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363e59c640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3640100fe0 0x7f364010e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 -- 172.21.15.27:0/3642347843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3640100620 msgr2=0x7f364010df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3640100620 0x7f364010df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 -- 172.21.15.27:0/3642347843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3640100fe0 msgr2=0x7f364010e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3640100fe0 0x7f364010e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 -- 172.21.15.27:0/3642347843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36401a64a0 con 0x7f3640106630 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f363f59e640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106630 0x7f364010ea00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f363000eb10 tx=0x7f363000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f3637fff640 1 -- 172.21.15.27:0/3642347843 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630004b20 con 0x7f3640106630 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f3637fff640 1 -- 172.21.15.27:0/3642347843 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3630004cc0 con 0x7f3640106630 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f3637fff640 1 -- 172.21.15.27:0/3642347843 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630010760 con 0x7f3640106630 2024-04-25T01:25:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36401a6790 con 0x7f3640106630 2024-04-25T01:25:30.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.220+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36401a6c60 con 0x7f3640106630 2024-04-25T01:25:30.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.221+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3640068560 con 0x7f3640106630 2024-04-25T01:25:30.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.222+0000 7f3637fff640 1 -- 172.21.15.27:0/3642347843 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f3630005020 con 0x7f3640106630 2024-04-25T01:25:30.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.223+0000 7f3637fff640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f362c0a2e50 0x7f362c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.223+0000 7f363ed9d640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f362c0a2e50 0x7f362c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.223+0000 7f3637fff640 1 -- 172.21.15.27:0/3642347843 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f36300d0010 con 0x7f3640106630 2024-04-25T01:25:30.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.224+0000 7f363ed9d640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f362c0a2e50 0x7f362c0a5310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f36380058c0 tx=0x7f363800a2d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.228+0000 7f3637fff640 1 -- 172.21.15.27:0/3642347843 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f363009c650 con 0x7f3640106630 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2245360085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3281170569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3793438795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:30.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2245360085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: Updating smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3281170569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3793438795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:30.293 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:30.294 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:25:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.412+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3640061b80 con 0x7f362c0a2e50 2024-04-25T01:25:30.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.421+0000 7f3637fff640 1 -- 172.21.15.27:0/3642347843 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339699 (secure 0 0 0) 0x7f3640061b80 con 0x7f362c0a2e50 2024-04-25T01:25:30.405 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:30.407 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T01:25:30.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f362c0a2e50 msgr2=0x7f362c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f362c0a2e50 0x7f362c0a5310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f36380058c0 tx=0x7f363800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106630 msgr2=0x7f364010ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106630 0x7f364010ea00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f363000eb10 tx=0x7f363000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 shutdown_connections 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f362c0a2e50 0x7f362c0a5310 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106630 0x7f364010ea00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3640100fe0 0x7f364010e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 --2- 172.21.15.27:0/3642347843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3640100620 0x7f364010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 >> 172.21.15.27:0/3642347843 conn(0x7f3640077740 msgr2=0x7f3640102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 shutdown_connections 2024-04-25T01:25:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.425+0000 7f364515a640 1 -- 172.21.15.27:0/3642347843 wait complete. 2024-04-25T01:25:30.416 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:25:28.998037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190019,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":986,"num_read_kb":654,"num_write":472,"num_write_kb":4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1244945,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361864,"kb_used_data":12672,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579960,"statfs":{"total":1151940427776,"available":1151569879040,"internally_reserved":0,"allocated":12976128,"data_stored":9890553,"data_compressed":549245,"data_compressed_allocated":741376,"data_compressed_original":4370432,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990220"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800504+0000","last_change":"2024-04-25T01:22:57.847700+0000","last_active":"2024-04-25T01:25:25.800504+0000","last_peered":"2024-04-25T01:25:25.800504+0000","last_clean":"2024-04-25T01:25:25.800504+0000","last_became_active":"2024-04-25T01:22:57.846877+0000","last_became_peered":"2024-04-25T01:22:57.846877+0000","last_unstale":"2024-04-25T01:25:25.800504+0000","last_undegraded":"2024-04-25T01:25:25.800504+0000","last_fullsized":"2024-04-25T01:25:25.800504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:05.918261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241338+0000","last_change":"2024-04-25T01:22:57.848396+0000","last_active":"2024-04-25T01:25:25.241338+0000","last_peered":"2024-04-25T01:25:25.241338+0000","last_clean":"2024-04-25T01:25:25.241338+0000","last_became_active":"2024-04-25T01:22:57.847971+0000","last_became_peered":"2024-04-25T01:22:57.847971+0000","last_unstale":"2024-04-25T01:25:25.241338+0000","last_undegraded":"2024-04-25T01:25:25.241338+0000","last_fullsized":"2024-04-25T01:25:25.241338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:13.143828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240866+0000","last_change":"2024-04-25T01:22:57.856908+0000","last_active":"2024-04-25T01:25:25.240866+0000","last_peered":"2024-04-25T01:25:25.240866+0000","last_clean":"2024-04-25T01:25:25.240866+0000","last_became_active":"2024-04-25T01:22:57.856218+0000","last_became_peered":"2024-04-25T01:22:57.856218+0000","last_unstale":"2024-04-25T01:25:25.240866+0000","last_undegraded":"2024-04-25T01:25:25.240866+0000","last_fullsized":"2024-04-25T01:25:25.240866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:51.971222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119108+0000","last_change":"2024-04-25T01:22:57.849022+0000","last_active":"2024-04-25T01:25:25.119108+0000","last_peered":"2024-04-25T01:25:25.119108+0000","last_clean":"2024-04-25T01:25:25.119108+0000","last_became_active":"2024-04-25T01:22:57.848303+0000","last_became_peered":"2024-04-25T01:22:57.848303+0000","last_unstale":"2024-04-25T01:25:25.119108+0000","last_undegraded":"2024-04-25T01:25:25.119108+0000","last_fullsized":"2024-04-25T01:25:25.119108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:52.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087107+0000","last_change":"2024-04-25T01:23:47.488312+0000","last_active":"2024-04-25T01:25:25.087107+0000","last_peered":"2024-04-25T01:25:25.087107+0000","last_clean":"2024-04-25T01:25:25.087107+0000","last_became_active":"2024-04-25T01:23:47.487923+0000","last_became_peered":"2024-04-25T01:23:47.487923+0000","last_unstale":"2024-04-25T01:25:25.087107+0000","last_undegraded":"2024-04-25T01:25:25.087107+0000","last_fullsized":"2024-04-25T01:25:25.087107+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:30.276853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983361+0000","last_change":"2024-04-25T01:22:57.855733+0000","last_active":"2024-04-25T01:25:24.983361+0000","last_peered":"2024-04-25T01:25:24.983361+0000","last_clean":"2024-04-25T01:25:24.983361+0000","last_became_active":"2024-04-25T01:22:57.855075+0000","last_became_peered":"2024-04-25T01:22:57.855075+0000","last_unstale":"2024-04-25T01:25:24.983361+0000","last_undegraded":"2024-04-25T01:25:24.983361+0000","last_fullsized":"2024-04-25T01:25:24.983361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:24.563461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238372+0000","last_change":"2024-04-25T01:22:57.849693+0000","last_active":"2024-04-25T01:25:25.238372+0000","last_peered":"2024-04-25T01:25:25.238372+0000","last_clean":"2024-04-25T01:25:25.238372+0000","last_became_active":"2024-04-25T01:22:57.849324+0000","last_became_peered":"2024-04-25T01:22:57.849324+0000","last_unstale":"2024-04-25T01:25:25.238372+0000","last_undegraded":"2024-04-25T01:25:25.238372+0000","last_fullsized":"2024-04-25T01:25:25.238372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:09.936496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983951+0000","last_change":"2024-04-25T01:22:57.847962+0000","last_active":"2024-04-25T01:25:24.983951+0000","last_peered":"2024-04-25T01:25:24.983951+0000","last_clean":"2024-04-25T01:25:24.983951+0000","last_became_active":"2024-04-25T01:22:57.847592+0000","last_became_peered":"2024-04-25T01:22:57.847592+0000","last_unstale":"2024-04-25T01:25:24.983951+0000","last_undegraded":"2024-04-25T01:25:24.983951+0000","last_fullsized":"2024-04-25T01:25:24.983951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:45.594118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239964+0000","last_change":"2024-04-25T01:22:57.860711+0000","last_active":"2024-04-25T01:25:25.239964+0000","last_peered":"2024-04-25T01:25:25.239964+0000","last_clean":"2024-04-25T01:25:25.239964+0000","last_became_active":"2024-04-25T01:22:57.860393+0000","last_became_peered":"2024-04-25T01:22:57.860393+0000","last_unstale":"2024-04-25T01:25:25.239964+0000","last_undegraded":"2024-04-25T01:25:25.239964+0000","last_fullsized":"2024-04-25T01:25:25.239964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:31.068095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984143+0000","last_change":"2024-04-25T01:22:57.853741+0000","last_active":"2024-04-25T01:25:24.984143+0000","last_peered":"2024-04-25T01:25:24.984143+0000","last_clean":"2024-04-25T01:25:24.984143+0000","last_became_active":"2024-04-25T01:22:57.851015+0000","last_became_peered":"2024-04-25T01:22:57.851015+0000","last_unstale":"2024-04-25T01:25:24.984143+0000","last_undegraded":"2024-04-25T01:25:24.984143+0000","last_fullsized":"2024-04-25T01:25:24.984143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:45.831335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239112+0000","last_change":"2024-04-25T01:22:57.853354+0000","last_active":"2024-04-25T01:25:25.239112+0000","last_peered":"2024-04-25T01:25:25.239112+0000","last_clean":"2024-04-25T01:25:25.239112+0000","last_became_active":"2024-04-25T01:22:57.853043+0000","last_became_peered":"2024-04-25T01:22:57.853043+0000","last_unstale":"2024-04-25T01:25:25.239112+0000","last_undegraded":"2024-04-25T01:25:25.239112+0000","last_fullsized":"2024-04-25T01:25:25.239112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:54.378751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119156+0000","last_change":"2024-04-25T01:22:57.855542+0000","last_active":"2024-04-25T01:25:25.119156+0000","last_peered":"2024-04-25T01:25:25.119156+0000","last_clean":"2024-04-25T01:25:25.119156+0000","last_became_active":"2024-04-25T01:22:57.855196+0000","last_became_peered":"2024-04-25T01:22:57.855196+0000","last_unstale":"2024-04-25T01:25:25.119156+0000","last_undegraded":"2024-04-25T01:25:25.119156+0000","last_fullsized":"2024-04-25T01:25:25.119156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:56.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117568+0000","last_change":"2024-04-25T01:22:57.829353+0000","last_active":"2024-04-25T01:25:25.117568+0000","last_peered":"2024-04-25T01:25:25.117568+0000","last_clean":"2024-04-25T01:25:25.117568+0000","last_became_active":"2024-04-25T01:22:57.828937+0000","last_became_peered":"2024-04-25T01:22:57.828937+0000","last_unstale":"2024-04-25T01:25:25.117568+0000","last_undegraded":"2024-04-25T01:25:25.117568+0000","last_fullsized":"2024-04-25T01:25:25.117568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.541556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241354+0000","last_change":"2024-04-25T01:22:57.854635+0000","last_active":"2024-04-25T01:25:25.241354+0000","last_peered":"2024-04-25T01:25:25.241354+0000","last_clean":"2024-04-25T01:25:25.241354+0000","last_became_active":"2024-04-25T01:22:57.854292+0000","last_became_peered":"2024-04-25T01:22:57.854292+0000","last_unstale":"2024-04-25T01:25:25.241354+0000","last_undegraded":"2024-04-25T01:25:25.241354+0000","last_fullsized":"2024-04-25T01:25:25.241354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:48.209059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240635+0000","last_change":"2024-04-25T01:22:57.852210+0000","last_active":"2024-04-25T01:25:25.240635+0000","last_peered":"2024-04-25T01:25:25.240635+0000","last_clean":"2024-04-25T01:25:25.240635+0000","last_became_active":"2024-04-25T01:22:57.851833+0000","last_became_peered":"2024-04-25T01:22:57.851833+0000","last_unstale":"2024-04-25T01:25:25.240635+0000","last_undegraded":"2024-04-25T01:25:25.240635+0000","last_fullsized":"2024-04-25T01:25:25.240635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:14.664785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239054+0000","last_change":"2024-04-25T01:22:57.849357+0000","last_active":"2024-04-25T01:25:25.239054+0000","last_peered":"2024-04-25T01:25:25.239054+0000","last_clean":"2024-04-25T01:25:25.239054+0000","last_became_active":"2024-04-25T01:22:57.849050+0000","last_became_peered":"2024-04-25T01:22:57.849050+0000","last_unstale":"2024-04-25T01:25:25.239054+0000","last_undegraded":"2024-04-25T01:25:25.239054+0000","last_fullsized":"2024-04-25T01:25:25.239054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:37.669424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800559+0000","last_change":"2024-04-25T01:22:57.844790+0000","last_active":"2024-04-25T01:25:25.800559+0000","last_peered":"2024-04-25T01:25:25.800559+0000","last_clean":"2024-04-25T01:25:25.800559+0000","last_became_active":"2024-04-25T01:22:57.844442+0000","last_became_peered":"2024-04-25T01:22:57.844442+0000","last_unstale":"2024-04-25T01:25:25.800559+0000","last_undegraded":"2024-04-25T01:25:25.800559+0000","last_fullsized":"2024-04-25T01:25:25.800559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:33.766034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085227+0000","last_change":"2024-04-25T01:22:57.844430+0000","last_active":"2024-04-25T01:25:25.085227+0000","last_peered":"2024-04-25T01:25:25.085227+0000","last_clean":"2024-04-25T01:25:25.085227+0000","last_became_active":"2024-04-25T01:22:57.844110+0000","last_became_peered":"2024-04-25T01:22:57.844110+0000","last_unstale":"2024-04-25T01:25:25.085227+0000","last_undegraded":"2024-04-25T01:25:25.085227+0000","last_fullsized":"2024-04-25T01:25:25.085227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:50.348249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984509+0000","last_change":"2024-04-25T01:22:57.852705+0000","last_active":"2024-04-25T01:25:24.984509+0000","last_peered":"2024-04-25T01:25:24.984509+0000","last_clean":"2024-04-25T01:25:24.984509+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:25:24.984509+0000","last_undegraded":"2024-04-25T01:25:24.984509+0000","last_fullsized":"2024-04-25T01:25:24.984509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:10.320459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983615+0000","last_change":"2024-04-25T01:23:01.879341+0000","last_active":"2024-04-25T01:25:24.983615+0000","last_peered":"2024-04-25T01:25:24.983615+0000","last_clean":"2024-04-25T01:25:24.983615+0000","last_became_active":"2024-04-25T01:23:01.878881+0000","last_became_peered":"2024-04-25T01:23:01.878881+0000","last_unstale":"2024-04-25T01:25:24.983615+0000","last_undegraded":"2024-04-25T01:25:24.983615+0000","last_fullsized":"2024-04-25T01:25:24.983615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:46.036868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085363+0000","last_change":"2024-04-25T01:22:56.839867+0000","last_active":"2024-04-25T01:25:25.085363+0000","last_peered":"2024-04-25T01:25:25.085363+0000","last_clean":"2024-04-25T01:25:25.085363+0000","last_became_active":"2024-04-25T01:22:56.839521+0000","last_became_peered":"2024-04-25T01:22:56.839521+0000","last_unstale":"2024-04-25T01:25:25.085363+0000","last_undegraded":"2024-04-25T01:25:25.085363+0000","last_fullsized":"2024-04-25T01:25:25.085363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:00.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984330+0000","last_change":"2024-04-25T01:22:57.855922+0000","last_active":"2024-04-25T01:25:24.984330+0000","last_peered":"2024-04-25T01:25:24.984330+0000","last_clean":"2024-04-25T01:25:24.984330+0000","last_became_active":"2024-04-25T01:22:57.855584+0000","last_became_peered":"2024-04-25T01:22:57.855584+0000","last_unstale":"2024-04-25T01:25:24.984330+0000","last_undegraded":"2024-04-25T01:25:24.984330+0000","last_fullsized":"2024-04-25T01:25:24.984330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:02.791269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118747+0000","last_change":"2024-04-25T01:23:01.872340+0000","last_active":"2024-04-25T01:25:25.118747+0000","last_peered":"2024-04-25T01:25:25.118747+0000","last_clean":"2024-04-25T01:25:25.118747+0000","last_became_active":"2024-04-25T01:23:01.871900+0000","last_became_peered":"2024-04-25T01:23:01.871900+0000","last_unstale":"2024-04-25T01:25:25.118747+0000","last_undegraded":"2024-04-25T01:25:25.118747+0000","last_fullsized":"2024-04-25T01:25:25.118747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:11.918022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984380+0000","last_change":"2024-04-25T01:22:56.843851+0000","last_active":"2024-04-25T01:25:24.984380+0000","last_peered":"2024-04-25T01:25:24.984380+0000","last_clean":"2024-04-25T01:25:24.984380+0000","last_became_active":"2024-04-25T01:22:56.843232+0000","last_became_peered":"2024-04-25T01:22:56.843232+0000","last_unstale":"2024-04-25T01:25:24.984380+0000","last_undegraded":"2024-04-25T01:25:24.984380+0000","last_fullsized":"2024-04-25T01:25:24.984380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:13.758328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241152+0000","last_change":"2024-04-25T01:22:57.857686+0000","last_active":"2024-04-25T01:25:25.241152+0000","last_peered":"2024-04-25T01:25:25.241152+0000","last_clean":"2024-04-25T01:25:25.241152+0000","last_became_active":"2024-04-25T01:22:57.857175+0000","last_became_peered":"2024-04-25T01:22:57.857175+0000","last_unstale":"2024-04-25T01:25:25.241152+0000","last_undegraded":"2024-04-25T01:25:25.241152+0000","last_fullsized":"2024-04-25T01:25:25.241152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:24.777861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118799+0000","last_change":"2024-04-25T01:23:01.867914+0000","last_active":"2024-04-25T01:25:25.118799+0000","last_peered":"2024-04-25T01:25:25.118799+0000","last_clean":"2024-04-25T01:25:25.118799+0000","last_became_active":"2024-04-25T01:23:01.867103+0000","last_became_peered":"2024-04-25T01:23:01.867103+0000","last_unstale":"2024-04-25T01:25:25.118799+0000","last_undegraded":"2024-04-25T01:25:25.118799+0000","last_fullsized":"2024-04-25T01:25:25.118799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:26.425433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087318+0000","last_change":"2024-04-25T01:22:56.841310+0000","last_active":"2024-04-25T01:25:25.087318+0000","last_peered":"2024-04-25T01:25:25.087318+0000","last_clean":"2024-04-25T01:25:25.087318+0000","last_became_active":"2024-04-25T01:22:56.840882+0000","last_became_peered":"2024-04-25T01:22:56.840882+0000","last_unstale":"2024-04-25T01:25:25.087318+0000","last_undegraded":"2024-04-25T01:25:25.087318+0000","last_fullsized":"2024-04-25T01:25:25.087318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:29.971448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239551+0000","last_change":"2024-04-25T01:22:57.842810+0000","last_active":"2024-04-25T01:25:25.239551+0000","last_peered":"2024-04-25T01:25:25.239551+0000","last_clean":"2024-04-25T01:25:25.239551+0000","last_became_active":"2024-04-25T01:22:57.842457+0000","last_became_peered":"2024-04-25T01:22:57.842457+0000","last_unstale":"2024-04-25T01:25:25.239551+0000","last_undegraded":"2024-04-25T01:25:25.239551+0000","last_fullsized":"2024-04-25T01:25:25.239551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:45.512037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119017+0000","last_change":"2024-04-25T01:23:01.877298+0000","last_active":"2024-04-25T01:25:25.119017+0000","last_peered":"2024-04-25T01:25:25.119017+0000","last_clean":"2024-04-25T01:25:25.119017+0000","last_became_active":"2024-04-25T01:23:01.876874+0000","last_became_peered":"2024-04-25T01:23:01.876874+0000","last_unstale":"2024-04-25T01:25:25.119017+0000","last_undegraded":"2024-04-25T01:25:25.119017+0000","last_fullsized":"2024-04-25T01:25:25.119017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:35:08.221388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800229+0000","last_change":"2024-04-25T01:24:47.495071+0000","last_active":"2024-04-25T01:25:25.800229+0000","last_peered":"2024-04-25T01:25:25.800229+0000","last_clean":"2024-04-25T01:25:25.800229+0000","last_became_active":"2024-04-25T01:24:47.491300+0000","last_became_peered":"2024-04-25T01:24:47.491300+0000","last_unstale":"2024-04-25T01:25:25.800229+0000","last_undegraded":"2024-04-25T01:25:25.800229+0000","last_fullsized":"2024-04-25T01:25:25.800229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:33:00.552625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093517+0000","last_change":"2024-04-25T01:22:57.851210+0000","last_active":"2024-04-25T01:25:25.093517+0000","last_peered":"2024-04-25T01:25:25.093517+0000","last_clean":"2024-04-25T01:25:25.093517+0000","last_became_active":"2024-04-25T01:22:57.850867+0000","last_became_peered":"2024-04-25T01:22:57.850867+0000","last_unstale":"2024-04-25T01:25:25.093517+0000","last_undegraded":"2024-04-25T01:25:25.093517+0000","last_fullsized":"2024-04-25T01:25:25.093517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:15.998433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984328+0000","last_change":"2024-04-25T01:23:47.489730+0000","last_active":"2024-04-25T01:25:24.984328+0000","last_peered":"2024-04-25T01:25:24.984328+0000","last_clean":"2024-04-25T01:25:24.984328+0000","last_became_active":"2024-04-25T01:23:47.489190+0000","last_became_peered":"2024-04-25T01:23:47.489190+0000","last_unstale":"2024-04-25T01:25:24.984328+0000","last_undegraded":"2024-04-25T01:25:24.984328+0000","last_fullsized":"2024-04-25T01:25:24.984328+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.101073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.237569+0000","last_change":"2024-04-25T01:22:56.834741+0000","last_active":"2024-04-25T01:25:25.237569+0000","last_peered":"2024-04-25T01:25:25.237569+0000","last_clean":"2024-04-25T01:25:25.237569+0000","last_became_active":"2024-04-25T01:22:56.834296+0000","last_became_peered":"2024-04-25T01:22:56.834296+0000","last_unstale":"2024-04-25T01:25:25.237569+0000","last_undegraded":"2024-04-25T01:25:25.237569+0000","last_fullsized":"2024-04-25T01:25:25.237569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:38.940603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087133+0000","last_change":"2024-04-25T01:22:57.854646+0000","last_active":"2024-04-25T01:25:25.087133+0000","last_peered":"2024-04-25T01:25:25.087133+0000","last_clean":"2024-04-25T01:25:25.087133+0000","last_became_active":"2024-04-25T01:22:57.853839+0000","last_became_peered":"2024-04-25T01:22:57.853839+0000","last_unstale":"2024-04-25T01:25:25.087133+0000","last_undegraded":"2024-04-25T01:25:25.087133+0000","last_fullsized":"2024-04-25T01:25:25.087133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:36.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984172+0000","last_change":"2024-04-25T01:23:01.875478+0000","last_active":"2024-04-25T01:25:24.984172+0000","last_peered":"2024-04-25T01:25:24.984172+0000","last_clean":"2024-04-25T01:25:24.984172+0000","last_became_active":"2024-04-25T01:23:01.874643+0000","last_became_peered":"2024-04-25T01:23:01.874643+0000","last_unstale":"2024-04-25T01:25:24.984172+0000","last_undegraded":"2024-04-25T01:25:24.984172+0000","last_fullsized":"2024-04-25T01:25:24.984172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.953507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983490+0000","last_change":"2024-04-25T01:22:56.843118+0000","last_active":"2024-04-25T01:25:24.983490+0000","last_peered":"2024-04-25T01:25:24.983490+0000","last_clean":"2024-04-25T01:25:24.983490+0000","last_became_active":"2024-04-25T01:22:56.842798+0000","last_became_peered":"2024-04-25T01:22:56.842798+0000","last_unstale":"2024-04-25T01:25:24.983490+0000","last_undegraded":"2024-04-25T01:25:24.983490+0000","last_fullsized":"2024-04-25T01:25:24.983490+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:00.054913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093690+0000","last_change":"2024-04-25T01:22:57.850265+0000","last_active":"2024-04-25T01:25:25.093690+0000","last_peered":"2024-04-25T01:25:25.093690+0000","last_clean":"2024-04-25T01:25:25.093690+0000","last_became_active":"2024-04-25T01:22:57.849480+0000","last_became_peered":"2024-04-25T01:22:57.849480+0000","last_unstale":"2024-04-25T01:25:25.093690+0000","last_undegraded":"2024-04-25T01:25:25.093690+0000","last_fullsized":"2024-04-25T01:25:25.093690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:48.577252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800384+0000","last_change":"2024-04-25T01:23:01.865369+0000","last_active":"2024-04-25T01:25:25.800384+0000","last_peered":"2024-04-25T01:25:25.800384+0000","last_clean":"2024-04-25T01:25:25.800384+0000","last_became_active":"2024-04-25T01:23:01.864872+0000","last_became_peered":"2024-04-25T01:23:01.864872+0000","last_unstale":"2024-04-25T01:25:25.800384+0000","last_undegraded":"2024-04-25T01:25:25.800384+0000","last_fullsized":"2024-04-25T01:25:25.800384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:56.266514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116988+0000","last_change":"2024-04-25T01:22:56.842576+0000","last_active":"2024-04-25T01:25:25.116988+0000","last_peered":"2024-04-25T01:25:25.116988+0000","last_clean":"2024-04-25T01:25:25.116988+0000","last_became_active":"2024-04-25T01:22:56.841081+0000","last_became_peered":"2024-04-25T01:22:56.841081+0000","last_unstale":"2024-04-25T01:25:25.116988+0000","last_undegraded":"2024-04-25T01:25:25.116988+0000","last_fullsized":"2024-04-25T01:25:25.116988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:08.174426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983862+0000","last_change":"2024-04-25T01:22:57.842812+0000","last_active":"2024-04-25T01:25:24.983862+0000","last_peered":"2024-04-25T01:25:24.983862+0000","last_clean":"2024-04-25T01:25:24.983862+0000","last_became_active":"2024-04-25T01:22:57.842462+0000","last_became_peered":"2024-04-25T01:22:57.842462+0000","last_unstale":"2024-04-25T01:25:24.983862+0000","last_undegraded":"2024-04-25T01:25:24.983862+0000","last_fullsized":"2024-04-25T01:25:24.983862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:14:13.444850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241074+0000","last_change":"2024-04-25T01:23:01.870401+0000","last_active":"2024-04-25T01:25:25.241074+0000","last_peered":"2024-04-25T01:25:25.241074+0000","last_clean":"2024-04-25T01:25:25.241074+0000","last_became_active":"2024-04-25T01:23:01.869932+0000","last_became_peered":"2024-04-25T01:23:01.869932+0000","last_unstale":"2024-04-25T01:25:25.241074+0000","last_undegraded":"2024-04-25T01:25:25.241074+0000","last_fullsized":"2024-04-25T01:25:25.241074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:13.323209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093295+0000","last_change":"2024-04-25T01:22:56.835946+0000","last_active":"2024-04-25T01:25:25.093295+0000","last_peered":"2024-04-25T01:25:25.093295+0000","last_clean":"2024-04-25T01:25:25.093295+0000","last_became_active":"2024-04-25T01:22:56.835630+0000","last_became_peered":"2024-04-25T01:22:56.835630+0000","last_unstale":"2024-04-25T01:25:25.093295+0000","last_undegraded":"2024-04-25T01:25:25.093295+0000","last_fullsized":"2024-04-25T01:25:25.093295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:02.128275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.084835+0000","last_change":"2024-04-25T01:22:57.844462+0000","last_active":"2024-04-25T01:25:25.084835+0000","last_peered":"2024-04-25T01:25:25.084835+0000","last_clean":"2024-04-25T01:25:25.084835+0000","last_became_active":"2024-04-25T01:22:57.843426+0000","last_became_peered":"2024-04-25T01:22:57.843426+0000","last_unstale":"2024-04-25T01:25:25.084835+0000","last_undegraded":"2024-04-25T01:25:25.084835+0000","last_fullsized":"2024-04-25T01:25:25.084835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:42.896622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984063+0000","last_change":"2024-04-25T01:23:01.871387+0000","last_active":"2024-04-25T01:25:24.984063+0000","last_peered":"2024-04-25T01:25:24.984063+0000","last_clean":"2024-04-25T01:25:24.984063+0000","last_became_active":"2024-04-25T01:23:01.870901+0000","last_became_peered":"2024-04-25T01:23:01.870901+0000","last_unstale":"2024-04-25T01:25:24.984063+0000","last_undegraded":"2024-04-25T01:25:24.984063+0000","last_fullsized":"2024-04-25T01:25:24.984063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:49.582515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238114+0000","last_change":"2024-04-25T01:22:56.834440+0000","last_active":"2024-04-25T01:25:25.238114+0000","last_peered":"2024-04-25T01:25:25.238114+0000","last_clean":"2024-04-25T01:25:25.238114+0000","last_became_active":"2024-04-25T01:22:56.834116+0000","last_became_peered":"2024-04-25T01:22:56.834116+0000","last_unstale":"2024-04-25T01:25:25.238114+0000","last_undegraded":"2024-04-25T01:25:25.238114+0000","last_fullsized":"2024-04-25T01:25:25.238114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:01.176601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239994+0000","last_change":"2024-04-25T01:22:57.853560+0000","last_active":"2024-04-25T01:25:25.239994+0000","last_peered":"2024-04-25T01:25:25.239994+0000","last_clean":"2024-04-25T01:25:25.239994+0000","last_became_active":"2024-04-25T01:22:57.853234+0000","last_became_peered":"2024-04-25T01:22:57.853234+0000","last_unstale":"2024-04-25T01:25:25.239994+0000","last_undegraded":"2024-04-25T01:25:25.239994+0000","last_fullsized":"2024-04-25T01:25:25.239994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:51.139371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.084730+0000","last_change":"2024-04-25T01:23:01.873856+0000","last_active":"2024-04-25T01:25:25.084730+0000","last_peered":"2024-04-25T01:25:25.084730+0000","last_clean":"2024-04-25T01:25:25.084730+0000","last_became_active":"2024-04-25T01:23:01.873413+0000","last_became_peered":"2024-04-25T01:23:01.873413+0000","last_unstale":"2024-04-25T01:25:25.084730+0000","last_undegraded":"2024-04-25T01:25:25.084730+0000","last_fullsized":"2024-04-25T01:25:25.084730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:11.802479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239188+0000","last_change":"2024-04-25T01:22:56.831080+0000","last_active":"2024-04-25T01:25:25.239188+0000","last_peered":"2024-04-25T01:25:25.239188+0000","last_clean":"2024-04-25T01:25:25.239188+0000","last_became_active":"2024-04-25T01:22:56.830735+0000","last_became_peered":"2024-04-25T01:22:56.830735+0000","last_unstale":"2024-04-25T01:25:25.239188+0000","last_undegraded":"2024-04-25T01:25:25.239188+0000","last_fullsized":"2024-04-25T01:25:25.239188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:38.418439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241002+0000","last_change":"2024-04-25T01:22:57.853838+0000","last_active":"2024-04-25T01:25:25.241002+0000","last_peered":"2024-04-25T01:25:25.241002+0000","last_clean":"2024-04-25T01:25:25.241002+0000","last_became_active":"2024-04-25T01:22:57.853023+0000","last_became_peered":"2024-04-25T01:22:57.853023+0000","last_unstale":"2024-04-25T01:25:25.241002+0000","last_undegraded":"2024-04-25T01:25:25.241002+0000","last_fullsized":"2024-04-25T01:25:25.241002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:18.515831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118964+0000","last_change":"2024-04-25T01:23:01.865446+0000","last_active":"2024-04-25T01:25:25.118964+0000","last_peered":"2024-04-25T01:25:25.118964+0000","last_clean":"2024-04-25T01:25:25.118964+0000","last_became_active":"2024-04-25T01:23:01.864937+0000","last_became_peered":"2024-04-25T01:23:01.864937+0000","last_unstale":"2024-04-25T01:25:25.118964+0000","last_undegraded":"2024-04-25T01:25:25.118964+0000","last_fullsized":"2024-04-25T01:25:25.118964+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:18.041430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093597+0000","last_change":"2024-04-25T01:22:56.836055+0000","last_active":"2024-04-25T01:25:25.093597+0000","last_peered":"2024-04-25T01:25:25.093597+0000","last_clean":"2024-04-25T01:25:25.093597+0000","last_became_active":"2024-04-25T01:22:56.835760+0000","last_became_peered":"2024-04-25T01:22:56.835760+0000","last_unstale":"2024-04-25T01:25:25.093597+0000","last_undegraded":"2024-04-25T01:25:25.093597+0000","last_fullsized":"2024-04-25T01:25:25.093597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:59.249996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239754+0000","last_change":"2024-04-25T01:22:57.852444+0000","last_active":"2024-04-25T01:25:25.239754+0000","last_peered":"2024-04-25T01:25:25.239754+0000","last_clean":"2024-04-25T01:25:25.239754+0000","last_became_active":"2024-04-25T01:22:57.852127+0000","last_became_peered":"2024-04-25T01:22:57.852127+0000","last_unstale":"2024-04-25T01:25:25.239754+0000","last_undegraded":"2024-04-25T01:25:25.239754+0000","last_fullsized":"2024-04-25T01:25:25.239754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:11.718512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983143+0000","last_change":"2024-04-25T01:23:47.039057+0000","last_active":"2024-04-25T01:25:24.983143+0000","last_peered":"2024-04-25T01:25:24.983143+0000","last_clean":"2024-04-25T01:25:24.983143+0000","last_became_active":"2024-04-25T01:23:47.038568+0000","last_became_peered":"2024-04-25T01:23:47.038568+0000","last_unstale":"2024-04-25T01:25:24.983143+0000","last_undegraded":"2024-04-25T01:25:24.983143+0000","last_fullsized":"2024-04-25T01:25:24.983143+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:23.828975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"94'39","reported_seq":346,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.549084+0000","last_change":"2024-04-25T01:24:47.501745+0000","last_active":"2024-04-25T01:25:25.549084+0000","last_peered":"2024-04-25T01:25:25.549084+0000","last_clean":"2024-04-25T01:25:25.549084+0000","last_became_active":"2024-04-25T01:24:47.494829+0000","last_became_peered":"2024-04-25T01:24:47.494829+0000","last_unstale":"2024-04-25T01:25:25.549084+0000","last_undegraded":"2024-04-25T01:25:25.549084+0000","last_fullsized":"2024-04-25T01:25:25.549084+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:54.482539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69677,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":252,"num_read_kb":194,"num_write":40,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":63605,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087512+0000","last_change":"2024-04-25T01:22:57.854554+0000","last_active":"2024-04-25T01:25:25.087512+0000","last_peered":"2024-04-25T01:25:25.087512+0000","last_clean":"2024-04-25T01:25:25.087512+0000","last_became_active":"2024-04-25T01:22:57.853328+0000","last_became_peered":"2024-04-25T01:22:57.853328+0000","last_unstale":"2024-04-25T01:25:25.087512+0000","last_undegraded":"2024-04-25T01:25:25.087512+0000","last_fullsized":"2024-04-25T01:25:25.087512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:21.774148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093976+0000","last_change":"2024-04-25T01:23:01.870571+0000","last_active":"2024-04-25T01:25:25.093976+0000","last_peered":"2024-04-25T01:25:25.093976+0000","last_clean":"2024-04-25T01:25:25.093976+0000","last_became_active":"2024-04-25T01:23:01.869982+0000","last_became_peered":"2024-04-25T01:23:01.869982+0000","last_unstale":"2024-04-25T01:25:25.093976+0000","last_undegraded":"2024-04-25T01:25:25.093976+0000","last_fullsized":"2024-04-25T01:25:25.093976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:17.294254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984043+0000","last_change":"2024-04-25T01:22:56.835134+0000","last_active":"2024-04-25T01:25:24.984043+0000","last_peered":"2024-04-25T01:25:24.984043+0000","last_clean":"2024-04-25T01:25:24.984043+0000","last_became_active":"2024-04-25T01:22:56.834207+0000","last_became_peered":"2024-04-25T01:22:56.834207+0000","last_unstale":"2024-04-25T01:25:24.984043+0000","last_undegraded":"2024-04-25T01:25:24.984043+0000","last_fullsized":"2024-04-25T01:25:24.984043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:07.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116949+0000","last_change":"2024-04-25T01:22:57.847693+0000","last_active":"2024-04-25T01:25:25.116949+0000","last_peered":"2024-04-25T01:25:25.116949+0000","last_clean":"2024-04-25T01:25:25.116949+0000","last_became_active":"2024-04-25T01:22:57.847211+0000","last_became_peered":"2024-04-25T01:22:57.847211+0000","last_unstale":"2024-04-25T01:25:25.116949+0000","last_undegraded":"2024-04-25T01:25:25.116949+0000","last_fullsized":"2024-04-25T01:25:25.116949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:48.295608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800279+0000","last_change":"2024-04-25T01:23:01.870340+0000","last_active":"2024-04-25T01:25:25.800279+0000","last_peered":"2024-04-25T01:25:25.800279+0000","last_clean":"2024-04-25T01:25:25.800279+0000","last_became_active":"2024-04-25T01:23:01.869662+0000","last_became_peered":"2024-04-25T01:23:01.869662+0000","last_unstale":"2024-04-25T01:25:25.800279+0000","last_undegraded":"2024-04-25T01:25:25.800279+0000","last_fullsized":"2024-04-25T01:25:25.800279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:05:57.313991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093495+0000","last_change":"2024-04-25T01:22:56.838146+0000","last_active":"2024-04-25T01:25:25.093495+0000","last_peered":"2024-04-25T01:25:25.093495+0000","last_clean":"2024-04-25T01:25:25.093495+0000","last_became_active":"2024-04-25T01:22:56.837815+0000","last_became_peered":"2024-04-25T01:22:56.837815+0000","last_unstale":"2024-04-25T01:25:25.093495+0000","last_undegraded":"2024-04-25T01:25:25.093495+0000","last_fullsized":"2024-04-25T01:25:25.093495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:14.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238786+0000","last_change":"2024-04-25T01:22:57.850840+0000","last_active":"2024-04-25T01:25:25.238786+0000","last_peered":"2024-04-25T01:25:25.238786+0000","last_clean":"2024-04-25T01:25:25.238786+0000","last_became_active":"2024-04-25T01:22:57.850255+0000","last_became_peered":"2024-04-25T01:22:57.850255+0000","last_unstale":"2024-04-25T01:25:25.238786+0000","last_undegraded":"2024-04-25T01:25:25.238786+0000","last_fullsized":"2024-04-25T01:25:25.238786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:25.887235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085296+0000","last_change":"2024-04-25T01:23:01.869396+0000","last_active":"2024-04-25T01:25:25.085296+0000","last_peered":"2024-04-25T01:25:25.085296+0000","last_clean":"2024-04-25T01:25:25.085296+0000","last_became_active":"2024-04-25T01:23:01.868777+0000","last_became_peered":"2024-04-25T01:23:01.868777+0000","last_unstale":"2024-04-25T01:25:25.085296+0000","last_undegraded":"2024-04-25T01:25:25.085296+0000","last_fullsized":"2024-04-25T01:25:25.085296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:37.276122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'4","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240662+0000","last_change":"2024-04-25T01:22:56.833289+0000","last_active":"2024-04-25T01:25:25.240662+0000","last_peered":"2024-04-25T01:25:25.240662+0000","last_clean":"2024-04-25T01:25:25.240662+0000","last_became_active":"2024-04-25T01:22:56.832353+0000","last_became_peered":"2024-04-25T01:22:56.832353+0000","last_unstale":"2024-04-25T01:25:25.240662+0000","last_undegraded":"2024-04-25T01:25:25.240662+0000","last_fullsized":"2024-04-25T01:25:25.240662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:18.251018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238649+0000","last_change":"2024-04-25T01:22:57.851974+0000","last_active":"2024-04-25T01:25:25.238649+0000","last_peered":"2024-04-25T01:25:25.238649+0000","last_clean":"2024-04-25T01:25:25.238649+0000","last_became_active":"2024-04-25T01:22:57.851573+0000","last_became_peered":"2024-04-25T01:22:57.851573+0000","last_unstale":"2024-04-25T01:25:25.238649+0000","last_undegraded":"2024-04-25T01:25:25.238649+0000","last_fullsized":"2024-04-25T01:25:25.238649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:27.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117895+0000","last_change":"2024-04-25T01:23:01.876980+0000","last_active":"2024-04-25T01:25:25.117895+0000","last_peered":"2024-04-25T01:25:25.117895+0000","last_clean":"2024-04-25T01:25:25.117895+0000","last_became_active":"2024-04-25T01:23:01.876548+0000","last_became_peered":"2024-04-25T01:23:01.876548+0000","last_unstale":"2024-04-25T01:25:25.117895+0000","last_undegraded":"2024-04-25T01:25:25.117895+0000","last_fullsized":"2024-04-25T01:25:25.117895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:13.470736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087000+0000","last_change":"2024-04-25T01:22:56.839694+0000","last_active":"2024-04-25T01:25:25.087000+0000","last_peered":"2024-04-25T01:25:25.087000+0000","last_clean":"2024-04-25T01:25:25.087000+0000","last_became_active":"2024-04-25T01:22:56.839367+0000","last_became_peered":"2024-04-25T01:22:56.839367+0000","last_unstale":"2024-04-25T01:25:25.087000+0000","last_undegraded":"2024-04-25T01:25:25.087000+0000","last_fullsized":"2024-04-25T01:25:25.087000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:49.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093465+0000","last_change":"2024-04-25T01:22:57.841526+0000","last_active":"2024-04-25T01:25:25.093465+0000","last_peered":"2024-04-25T01:25:25.093465+0000","last_clean":"2024-04-25T01:25:25.093465+0000","last_became_active":"2024-04-25T01:22:57.841192+0000","last_became_peered":"2024-04-25T01:22:57.841192+0000","last_unstale":"2024-04-25T01:25:25.093465+0000","last_undegraded":"2024-04-25T01:25:25.093465+0000","last_fullsized":"2024-04-25T01:25:25.093465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:38.958901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093301+0000","last_change":"2024-04-25T01:23:01.865486+0000","last_active":"2024-04-25T01:25:25.093301+0000","last_peered":"2024-04-25T01:25:25.093301+0000","last_clean":"2024-04-25T01:25:25.093301+0000","last_became_active":"2024-04-25T01:23:01.865068+0000","last_became_peered":"2024-04-25T01:23:01.865068+0000","last_unstale":"2024-04-25T01:25:25.093301+0000","last_undegraded":"2024-04-25T01:25:25.093301+0000","last_fullsized":"2024-04-25T01:25:25.093301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:50.432724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087557+0000","last_change":"2024-04-25T01:22:56.841507+0000","last_active":"2024-04-25T01:25:25.087557+0000","last_peered":"2024-04-25T01:25:25.087557+0000","last_clean":"2024-04-25T01:25:25.087557+0000","last_became_active":"2024-04-25T01:22:56.839970+0000","last_became_peered":"2024-04-25T01:22:56.839970+0000","last_unstale":"2024-04-25T01:25:25.087557+0000","last_undegraded":"2024-04-25T01:25:25.087557+0000","last_fullsized":"2024-04-25T01:25:25.087557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:37.670052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800378+0000","last_change":"2024-04-25T01:22:57.850853+0000","last_active":"2024-04-25T01:25:25.800378+0000","last_peered":"2024-04-25T01:25:25.800378+0000","last_clean":"2024-04-25T01:25:25.800378+0000","last_became_active":"2024-04-25T01:22:57.850498+0000","last_became_peered":"2024-04-25T01:22:57.850498+0000","last_unstale":"2024-04-25T01:25:25.800378+0000","last_undegraded":"2024-04-25T01:25:25.800378+0000","last_fullsized":"2024-04-25T01:25:25.800378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:13.271048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087021+0000","last_change":"2024-04-25T01:23:01.874254+0000","last_active":"2024-04-25T01:25:25.087021+0000","last_peered":"2024-04-25T01:25:25.087021+0000","last_clean":"2024-04-25T01:25:25.087021+0000","last_became_active":"2024-04-25T01:23:01.873711+0000","last_became_peered":"2024-04-25T01:23:01.873711+0000","last_unstale":"2024-04-25T01:25:25.087021+0000","last_undegraded":"2024-04-25T01:25:25.087021+0000","last_fullsized":"2024-04-25T01:25:25.087021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:13.510061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085035+0000","last_change":"2024-04-25T01:22:56.827137+0000","last_active":"2024-04-25T01:25:25.085035+0000","last_peered":"2024-04-25T01:25:25.085035+0000","last_clean":"2024-04-25T01:25:25.085035+0000","last_became_active":"2024-04-25T01:22:56.826769+0000","last_became_peered":"2024-04-25T01:22:56.826769+0000","last_unstale":"2024-04-25T01:25:25.085035+0000","last_undegraded":"2024-04-25T01:25:25.085035+0000","last_fullsized":"2024-04-25T01:25:25.085035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:21.072643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983993+0000","last_change":"2024-04-25T01:22:57.843713+0000","last_active":"2024-04-25T01:25:24.983993+0000","last_peered":"2024-04-25T01:25:24.983993+0000","last_clean":"2024-04-25T01:25:24.983993+0000","last_became_active":"2024-04-25T01:22:57.843384+0000","last_became_peered":"2024-04-25T01:22:57.843384+0000","last_unstale":"2024-04-25T01:25:24.983993+0000","last_undegraded":"2024-04-25T01:25:24.983993+0000","last_fullsized":"2024-04-25T01:25:24.983993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:18.772379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117639+0000","last_change":"2024-04-25T01:23:01.874150+0000","last_active":"2024-04-25T01:25:25.117639+0000","last_peered":"2024-04-25T01:25:25.117639+0000","last_clean":"2024-04-25T01:25:25.117639+0000","last_became_active":"2024-04-25T01:23:01.873718+0000","last_became_peered":"2024-04-25T01:23:01.873718+0000","last_unstale":"2024-04-25T01:25:25.117639+0000","last_undegraded":"2024-04-25T01:25:25.117639+0000","last_fullsized":"2024-04-25T01:25:25.117639+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:26.317703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238669+0000","last_change":"2024-04-25T01:22:56.843605+0000","last_active":"2024-04-25T01:25:25.238669+0000","last_peered":"2024-04-25T01:25:25.238669+0000","last_clean":"2024-04-25T01:25:25.238669+0000","last_became_active":"2024-04-25T01:22:56.843313+0000","last_became_peered":"2024-04-25T01:22:56.843313+0000","last_unstale":"2024-04-25T01:25:25.238669+0000","last_undegraded":"2024-04-25T01:25:25.238669+0000","last_fullsized":"2024-04-25T01:25:25.238669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:41.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983293+0000","last_change":"2024-04-25T01:23:47.005378+0000","last_active":"2024-04-25T01:25:24.983293+0000","last_peered":"2024-04-25T01:25:24.983293+0000","last_clean":"2024-04-25T01:25:24.983293+0000","last_became_active":"2024-04-25T01:23:47.004962+0000","last_became_peered":"2024-04-25T01:23:47.004962+0000","last_unstale":"2024-04-25T01:25:24.983293+0000","last_undegraded":"2024-04-25T01:25:24.983293+0000","last_fullsized":"2024-04-25T01:25:24.983293+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:28.548285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117894+0000","last_change":"2024-04-25T01:23:01.872792+0000","last_active":"2024-04-25T01:25:25.117894+0000","last_peered":"2024-04-25T01:25:25.117894+0000","last_clean":"2024-04-25T01:25:25.117894+0000","last_became_active":"2024-04-25T01:23:01.872391+0000","last_became_peered":"2024-04-25T01:23:01.872391+0000","last_unstale":"2024-04-25T01:25:25.117894+0000","last_undegraded":"2024-04-25T01:25:25.117894+0000","last_fullsized":"2024-04-25T01:25:25.117894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:13.088996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093632+0000","last_change":"2024-04-25T01:22:57.843131+0000","last_active":"2024-04-25T01:25:25.093632+0000","last_peered":"2024-04-25T01:25:25.093632+0000","last_clean":"2024-04-25T01:25:25.093632+0000","last_became_active":"2024-04-25T01:22:57.842540+0000","last_became_peered":"2024-04-25T01:22:57.842540+0000","last_unstale":"2024-04-25T01:25:25.093632+0000","last_undegraded":"2024-04-25T01:25:25.093632+0000","last_fullsized":"2024-04-25T01:25:25.093632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:51.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117205+0000","last_change":"2024-04-25T01:22:56.844610+0000","last_active":"2024-04-25T01:25:25.117205+0000","last_peered":"2024-04-25T01:25:25.117205+0000","last_clean":"2024-04-25T01:25:25.117205+0000","last_became_active":"2024-04-25T01:22:56.844237+0000","last_became_peered":"2024-04-25T01:22:56.844237+0000","last_unstale":"2024-04-25T01:25:25.117205+0000","last_undegraded":"2024-04-25T01:25:25.117205+0000","last_fullsized":"2024-04-25T01:25:25.117205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:09.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238830+0000","last_change":"2024-04-25T01:23:01.865711+0000","last_active":"2024-04-25T01:25:25.238830+0000","last_peered":"2024-04-25T01:25:25.238830+0000","last_clean":"2024-04-25T01:25:25.238830+0000","last_became_active":"2024-04-25T01:23:01.865317+0000","last_became_peered":"2024-04-25T01:23:01.865317+0000","last_unstale":"2024-04-25T01:25:25.238830+0000","last_undegraded":"2024-04-25T01:25:25.238830+0000","last_fullsized":"2024-04-25T01:25:25.238830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:08.995490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983908+0000","last_change":"2024-04-25T01:22:57.851623+0000","last_active":"2024-04-25T01:25:24.983908+0000","last_peered":"2024-04-25T01:25:24.983908+0000","last_clean":"2024-04-25T01:25:24.983908+0000","last_became_active":"2024-04-25T01:22:57.850616+0000","last_became_peered":"2024-04-25T01:22:57.850616+0000","last_unstale":"2024-04-25T01:25:24.983908+0000","last_undegraded":"2024-04-25T01:25:24.983908+0000","last_fullsized":"2024-04-25T01:25:24.983908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:27.447996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118832+0000","last_change":"2024-04-25T01:22:56.832094+0000","last_active":"2024-04-25T01:25:25.118832+0000","last_peered":"2024-04-25T01:25:25.118832+0000","last_clean":"2024-04-25T01:25:25.118832+0000","last_became_active":"2024-04-25T01:22:56.831743+0000","last_became_peered":"2024-04-25T01:22:56.831743+0000","last_unstale":"2024-04-25T01:25:25.118832+0000","last_undegraded":"2024-04-25T01:25:25.118832+0000","last_fullsized":"2024-04-25T01:25:25.118832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:06.116081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238473+0000","last_change":"2024-04-25T01:21:40.899517+0000","last_active":"2024-04-25T01:25:25.238473+0000","last_peered":"2024-04-25T01:25:25.238473+0000","last_clean":"2024-04-25T01:25:25.238473+0000","last_became_active":"2024-04-25T01:21:40.899043+0000","last_became_peered":"2024-04-25T01:21:40.899043+0000","last_unstale":"2024-04-25T01:25:25.238473+0000","last_undegraded":"2024-04-25T01:25:25.238473+0000","last_fullsized":"2024-04-25T01:25:25.238473+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_clean_scrub_stamp":"2024-04-25T01:18:12.053452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:41.234852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093701+0000","last_change":"2024-04-25T01:23:01.878334+0000","last_active":"2024-04-25T01:25:25.093701+0000","last_peered":"2024-04-25T01:25:25.093701+0000","last_clean":"2024-04-25T01:25:25.093701+0000","last_became_active":"2024-04-25T01:23:01.877955+0000","last_became_peered":"2024-04-25T01:23:01.877955+0000","last_unstale":"2024-04-25T01:25:25.093701+0000","last_undegraded":"2024-04-25T01:25:25.093701+0000","last_fullsized":"2024-04-25T01:25:25.093701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:33.339430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116577+0000","last_change":"2024-04-25T01:22:57.843020+0000","last_active":"2024-04-25T01:25:25.116577+0000","last_peered":"2024-04-25T01:25:25.116577+0000","last_clean":"2024-04-25T01:25:25.116577+0000","last_became_active":"2024-04-25T01:22:57.842671+0000","last_became_peered":"2024-04-25T01:22:57.842671+0000","last_unstale":"2024-04-25T01:25:25.116577+0000","last_undegraded":"2024-04-25T01:25:25.116577+0000","last_fullsized":"2024-04-25T01:25:25.116577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:53.831432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238821+0000","last_change":"2024-04-25T01:22:56.836106+0000","last_active":"2024-04-25T01:25:25.238821+0000","last_peered":"2024-04-25T01:25:25.238821+0000","last_clean":"2024-04-25T01:25:25.238821+0000","last_became_active":"2024-04-25T01:22:56.835814+0000","last_became_peered":"2024-04-25T01:22:56.835814+0000","last_unstale":"2024-04-25T01:25:25.238821+0000","last_undegraded":"2024-04-25T01:25:25.238821+0000","last_fullsized":"2024-04-25T01:25:25.238821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:18.004312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800091+0000","last_change":"2024-04-25T01:23:01.870723+0000","last_active":"2024-04-25T01:25:25.800091+0000","last_peered":"2024-04-25T01:25:25.800091+0000","last_clean":"2024-04-25T01:25:25.800091+0000","last_became_active":"2024-04-25T01:23:01.870205+0000","last_became_peered":"2024-04-25T01:23:01.870205+0000","last_unstale":"2024-04-25T01:25:25.800091+0000","last_undegraded":"2024-04-25T01:25:25.800091+0000","last_fullsized":"2024-04-25T01:25:25.800091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:47.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800282+0000","last_change":"2024-04-25T01:24:47.497526+0000","last_active":"2024-04-25T01:25:25.800282+0000","last_peered":"2024-04-25T01:25:25.800282+0000","last_clean":"2024-04-25T01:25:25.800282+0000","last_became_active":"2024-04-25T01:24:47.493380+0000","last_became_peered":"2024-04-25T01:24:47.493380+0000","last_unstale":"2024-04-25T01:25:25.800282+0000","last_undegraded":"2024-04-25T01:25:25.800282+0000","last_fullsized":"2024-04-25T01:25:25.800282+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:40:22.024954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117796+0000","last_change":"2024-04-25T01:22:57.841727+0000","last_active":"2024-04-25T01:25:25.117796+0000","last_peered":"2024-04-25T01:25:25.117796+0000","last_clean":"2024-04-25T01:25:25.117796+0000","last_became_active":"2024-04-25T01:22:57.841366+0000","last_became_peered":"2024-04-25T01:22:57.841366+0000","last_unstale":"2024-04-25T01:25:25.117796+0000","last_undegraded":"2024-04-25T01:25:25.117796+0000","last_fullsized":"2024-04-25T01:25:25.117796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:20.585916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983754+0000","last_change":"2024-04-25T01:23:01.869814+0000","last_active":"2024-04-25T01:25:24.983754+0000","last_peered":"2024-04-25T01:25:24.983754+0000","last_clean":"2024-04-25T01:25:24.983754+0000","last_became_active":"2024-04-25T01:23:01.869387+0000","last_became_peered":"2024-04-25T01:23:01.869387+0000","last_unstale":"2024-04-25T01:25:24.983754+0000","last_undegraded":"2024-04-25T01:25:24.983754+0000","last_fullsized":"2024-04-25T01:25:24.983754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:18:45.098295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'45","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.199682+0000","last_change":"2024-04-25T01:22:56.838506+0000","last_active":"2024-04-25T01:25:25.199682+0000","last_peered":"2024-04-25T01:25:25.199682+0000","last_clean":"2024-04-25T01:25:25.199682+0000","last_became_active":"2024-04-25T01:22:56.838185+0000","last_became_peered":"2024-04-25T01:22:56.838185+0000","last_unstale":"2024-04-25T01:25:25.199682+0000","last_undegraded":"2024-04-25T01:25:25.199682+0000","last_fullsized":"2024-04-25T01:25:25.199682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:35.306231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984432+0000","last_change":"2024-04-25T01:22:57.846544+0000","last_active":"2024-04-25T01:25:24.984432+0000","last_peered":"2024-04-25T01:25:24.984432+0000","last_clean":"2024-04-25T01:25:24.984432+0000","last_became_active":"2024-04-25T01:22:57.843507+0000","last_became_peered":"2024-04-25T01:22:57.843507+0000","last_unstale":"2024-04-25T01:25:24.984432+0000","last_undegraded":"2024-04-25T01:25:24.984432+0000","last_fullsized":"2024-04-25T01:25:24.984432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:30.600317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983013+0000","last_change":"2024-04-25T01:23:01.863986+0000","last_active":"2024-04-25T01:25:24.983013+0000","last_peered":"2024-04-25T01:25:24.983013+0000","last_clean":"2024-04-25T01:25:24.983013+0000","last_became_active":"2024-04-25T01:23:01.863397+0000","last_became_peered":"2024-04-25T01:23:01.863397+0000","last_unstale":"2024-04-25T01:25:24.983013+0000","last_undegraded":"2024-04-25T01:25:24.983013+0000","last_fullsized":"2024-04-25T01:25:24.983013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:45:56.229349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119308+0000","last_change":"2024-04-25T01:22:56.838201+0000","last_active":"2024-04-25T01:25:25.119308+0000","last_peered":"2024-04-25T01:25:25.119308+0000","last_clean":"2024-04-25T01:25:25.119308+0000","last_became_active":"2024-04-25T01:22:56.837890+0000","last_became_peered":"2024-04-25T01:22:56.837890+0000","last_unstale":"2024-04-25T01:25:25.119308+0000","last_undegraded":"2024-04-25T01:25:25.119308+0000","last_fullsized":"2024-04-25T01:25:25.119308+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:17.418112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241186+0000","last_change":"2024-04-25T01:22:57.853923+0000","last_active":"2024-04-25T01:25:25.241186+0000","last_peered":"2024-04-25T01:25:25.241186+0000","last_clean":"2024-04-25T01:25:25.241186+0000","last_became_active":"2024-04-25T01:22:57.853566+0000","last_became_peered":"2024-04-25T01:22:57.853566+0000","last_unstale":"2024-04-25T01:25:25.241186+0000","last_undegraded":"2024-04-25T01:25:25.241186+0000","last_fullsized":"2024-04-25T01:25:25.241186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:36.509650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119340+0000","last_change":"2024-04-25T01:23:47.038657+0000","last_active":"2024-04-25T01:25:25.119340+0000","last_peered":"2024-04-25T01:25:25.119340+0000","last_clean":"2024-04-25T01:25:25.119340+0000","last_became_active":"2024-04-25T01:23:47.038168+0000","last_became_peered":"2024-04-25T01:23:47.038168+0000","last_unstale":"2024-04-25T01:25:25.119340+0000","last_undegraded":"2024-04-25T01:25:25.119340+0000","last_fullsized":"2024-04-25T01:25:25.119340+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:59.844333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239155+0000","last_change":"2024-04-25T01:22:56.828651+0000","last_active":"2024-04-25T01:25:25.239155+0000","last_peered":"2024-04-25T01:25:25.239155+0000","last_clean":"2024-04-25T01:25:25.239155+0000","last_became_active":"2024-04-25T01:22:56.828186+0000","last_became_peered":"2024-04-25T01:22:56.828186+0000","last_unstale":"2024-04-25T01:25:25.239155+0000","last_undegraded":"2024-04-25T01:25:25.239155+0000","last_fullsized":"2024-04-25T01:25:25.239155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:56.020822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.086946+0000","last_change":"2024-04-25T01:22:57.849768+0000","last_active":"2024-04-25T01:25:25.086946+0000","last_peered":"2024-04-25T01:25:25.086946+0000","last_clean":"2024-04-25T01:25:25.086946+0000","last_became_active":"2024-04-25T01:22:57.849418+0000","last_became_peered":"2024-04-25T01:22:57.849418+0000","last_unstale":"2024-04-25T01:25:25.086946+0000","last_undegraded":"2024-04-25T01:25:25.086946+0000","last_fullsized":"2024-04-25T01:25:25.086946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:41.292915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087501+0000","last_change":"2024-04-25T01:23:47.148790+0000","last_active":"2024-04-25T01:25:25.087501+0000","last_peered":"2024-04-25T01:25:25.087501+0000","last_clean":"2024-04-25T01:25:25.087501+0000","last_became_active":"2024-04-25T01:23:47.148180+0000","last_became_peered":"2024-04-25T01:23:47.148180+0000","last_unstale":"2024-04-25T01:25:25.087501+0000","last_undegraded":"2024-04-25T01:25:25.087501+0000","last_fullsized":"2024-04-25T01:25:25.087501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:08.296696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983834+0000","last_change":"2024-04-25T01:22:56.844672+0000","last_active":"2024-04-25T01:25:24.983834+0000","last_peered":"2024-04-25T01:25:24.983834+0000","last_clean":"2024-04-25T01:25:24.983834+0000","last_became_active":"2024-04-25T01:22:56.844141+0000","last_became_peered":"2024-04-25T01:22:56.844141+0000","last_unstale":"2024-04-25T01:25:24.983834+0000","last_undegraded":"2024-04-25T01:25:24.983834+0000","last_fullsized":"2024-04-25T01:25:24.983834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:14.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800019+0000","last_change":"2024-04-25T01:22:57.845006+0000","last_active":"2024-04-25T01:25:25.800019+0000","last_peered":"2024-04-25T01:25:25.800019+0000","last_clean":"2024-04-25T01:25:25.800019+0000","last_became_active":"2024-04-25T01:22:57.844672+0000","last_became_peered":"2024-04-25T01:22:57.844672+0000","last_unstale":"2024-04-25T01:25:25.800019+0000","last_undegraded":"2024-04-25T01:25:25.800019+0000","last_fullsized":"2024-04-25T01:25:25.800019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:18.238512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119458+0000","last_change":"2024-04-25T01:23:01.869438+0000","last_active":"2024-04-25T01:25:25.119458+0000","last_peered":"2024-04-25T01:25:25.119458+0000","last_clean":"2024-04-25T01:25:25.119458+0000","last_became_active":"2024-04-25T01:23:01.868985+0000","last_became_peered":"2024-04-25T01:23:01.868985+0000","last_unstale":"2024-04-25T01:25:25.119458+0000","last_undegraded":"2024-04-25T01:25:25.119458+0000","last_fullsized":"2024-04-25T01:25:25.119458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:35.509098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983953+0000","last_change":"2024-04-25T01:22:56.834475+0000","last_active":"2024-04-25T01:25:24.983953+0000","last_peered":"2024-04-25T01:25:24.983953+0000","last_clean":"2024-04-25T01:25:24.983953+0000","last_became_active":"2024-04-25T01:22:56.834076+0000","last_became_peered":"2024-04-25T01:22:56.834076+0000","last_unstale":"2024-04-25T01:25:24.983953+0000","last_undegraded":"2024-04-25T01:25:24.983953+0000","last_fullsized":"2024-04-25T01:25:24.983953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:55:47.464009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984196+0000","last_change":"2024-04-25T01:22:57.860549+0000","last_active":"2024-04-25T01:25:24.984196+0000","last_peered":"2024-04-25T01:25:24.984196+0000","last_clean":"2024-04-25T01:25:24.984196+0000","last_became_active":"2024-04-25T01:22:57.860226+0000","last_became_peered":"2024-04-25T01:22:57.860226+0000","last_unstale":"2024-04-25T01:25:24.984196+0000","last_undegraded":"2024-04-25T01:25:24.984196+0000","last_fullsized":"2024-04-25T01:25:24.984196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:10.663508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119226+0000","last_change":"2024-04-25T01:23:01.867602+0000","last_active":"2024-04-25T01:25:25.119226+0000","last_peered":"2024-04-25T01:25:25.119226+0000","last_clean":"2024-04-25T01:25:25.119226+0000","last_became_active":"2024-04-25T01:23:01.867150+0000","last_became_peered":"2024-04-25T01:23:01.867150+0000","last_unstale":"2024-04-25T01:25:25.119226+0000","last_undegraded":"2024-04-25T01:25:25.119226+0000","last_fullsized":"2024-04-25T01:25:25.119226+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:34.571031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087024+0000","last_change":"2024-04-25T01:22:56.841082+0000","last_active":"2024-04-25T01:25:25.087024+0000","last_peered":"2024-04-25T01:25:25.087024+0000","last_clean":"2024-04-25T01:25:25.087024+0000","last_became_active":"2024-04-25T01:22:56.839856+0000","last_became_peered":"2024-04-25T01:22:56.839856+0000","last_unstale":"2024-04-25T01:25:25.087024+0000","last_undegraded":"2024-04-25T01:25:25.087024+0000","last_fullsized":"2024-04-25T01:25:25.087024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:14.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238762+0000","last_change":"2024-04-25T01:22:57.852242+0000","last_active":"2024-04-25T01:25:25.238762+0000","last_peered":"2024-04-25T01:25:25.238762+0000","last_clean":"2024-04-25T01:25:25.238762+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:25:25.238762+0000","last_undegraded":"2024-04-25T01:25:25.238762+0000","last_fullsized":"2024-04-25T01:25:25.238762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:38:02.810025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238989+0000","last_change":"2024-04-25T01:23:01.871074+0000","last_active":"2024-04-25T01:25:25.238989+0000","last_peered":"2024-04-25T01:25:25.238989+0000","last_clean":"2024-04-25T01:25:25.238989+0000","last_became_active":"2024-04-25T01:23:01.870570+0000","last_became_peered":"2024-04-25T01:23:01.870570+0000","last_unstale":"2024-04-25T01:25:25.238989+0000","last_undegraded":"2024-04-25T01:25:25.238989+0000","last_fullsized":"2024-04-25T01:25:25.238989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:14.476400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118784+0000","last_change":"2024-04-25T01:23:47.011202+0000","last_active":"2024-04-25T01:25:25.118784+0000","last_peered":"2024-04-25T01:25:25.118784+0000","last_clean":"2024-04-25T01:25:25.118784+0000","last_became_active":"2024-04-25T01:23:47.006425+0000","last_became_peered":"2024-04-25T01:23:47.006425+0000","last_unstale":"2024-04-25T01:25:25.118784+0000","last_undegraded":"2024-04-25T01:25:25.118784+0000","last_fullsized":"2024-04-25T01:25:25.118784+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:51:46.791184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239193+0000","last_change":"2024-04-25T01:22:57.843173+0000","last_active":"2024-04-25T01:25:25.239193+0000","last_peered":"2024-04-25T01:25:25.239193+0000","last_clean":"2024-04-25T01:25:25.239193+0000","last_became_active":"2024-04-25T01:22:57.842734+0000","last_became_peered":"2024-04-25T01:22:57.842734+0000","last_unstale":"2024-04-25T01:25:25.239193+0000","last_undegraded":"2024-04-25T01:25:25.239193+0000","last_fullsized":"2024-04-25T01:25:25.239193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:33.197493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240612+0000","last_change":"2024-04-25T01:23:01.864068+0000","last_active":"2024-04-25T01:25:25.240612+0000","last_peered":"2024-04-25T01:25:25.240612+0000","last_clean":"2024-04-25T01:25:25.240612+0000","last_became_active":"2024-04-25T01:23:01.863640+0000","last_became_peered":"2024-04-25T01:23:01.863640+0000","last_unstale":"2024-04-25T01:25:25.240612+0000","last_undegraded":"2024-04-25T01:25:25.240612+0000","last_fullsized":"2024-04-25T01:25:25.240612+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:26.925012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.194844+0000","last_change":"2024-04-25T01:22:56.837969+0000","last_active":"2024-04-25T01:25:25.194844+0000","last_peered":"2024-04-25T01:25:25.194844+0000","last_clean":"2024-04-25T01:25:25.194844+0000","last_became_active":"2024-04-25T01:22:56.837641+0000","last_became_peered":"2024-04-25T01:22:56.837641+0000","last_unstale":"2024-04-25T01:25:25.194844+0000","last_undegraded":"2024-04-25T01:25:25.194844+0000","last_fullsized":"2024-04-25T01:25:25.194844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:04.376038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238624+0000","last_change":"2024-04-25T01:22:57.851057+0000","last_active":"2024-04-25T01:25:25.238624+0000","last_peered":"2024-04-25T01:25:25.238624+0000","last_clean":"2024-04-25T01:25:25.238624+0000","last_became_active":"2024-04-25T01:22:57.850278+0000","last_became_peered":"2024-04-25T01:22:57.850278+0000","last_unstale":"2024-04-25T01:25:25.238624+0000","last_undegraded":"2024-04-25T01:25:25.238624+0000","last_fullsized":"2024-04-25T01:25:25.238624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:15.332193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093137+0000","last_change":"2024-04-25T01:23:01.866673+0000","last_active":"2024-04-25T01:25:25.093137+0000","last_peered":"2024-04-25T01:25:25.093137+0000","last_clean":"2024-04-25T01:25:25.093137+0000","last_became_active":"2024-04-25T01:23:01.866210+0000","last_became_peered":"2024-04-25T01:23:01.866210+0000","last_unstale":"2024-04-25T01:25:25.093137+0000","last_undegraded":"2024-04-25T01:25:25.093137+0000","last_fullsized":"2024-04-25T01:25:25.093137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:20.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117961+0000","last_change":"2024-04-25T01:22:56.844590+0000","last_active":"2024-04-25T01:25:25.117961+0000","last_peered":"2024-04-25T01:25:25.117961+0000","last_clean":"2024-04-25T01:25:25.117961+0000","last_became_active":"2024-04-25T01:22:56.844279+0000","last_became_peered":"2024-04-25T01:22:56.844279+0000","last_unstale":"2024-04-25T01:25:25.117961+0000","last_undegraded":"2024-04-25T01:25:25.117961+0000","last_fullsized":"2024-04-25T01:25:25.117961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:47.155832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239487+0000","last_change":"2024-04-25T01:22:57.858850+0000","last_active":"2024-04-25T01:25:25.239487+0000","last_peered":"2024-04-25T01:25:25.239487+0000","last_clean":"2024-04-25T01:25:25.239487+0000","last_became_active":"2024-04-25T01:22:57.858498+0000","last_became_peered":"2024-04-25T01:22:57.858498+0000","last_unstale":"2024-04-25T01:25:25.239487+0000","last_undegraded":"2024-04-25T01:25:25.239487+0000","last_fullsized":"2024-04-25T01:25:25.239487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:04.836062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800109+0000","last_change":"2024-04-25T01:23:47.007111+0000","last_active":"2024-04-25T01:25:25.800109+0000","last_peered":"2024-04-25T01:25:25.800109+0000","last_clean":"2024-04-25T01:25:25.800109+0000","last_became_active":"2024-04-25T01:23:47.006699+0000","last_became_peered":"2024-04-25T01:23:47.006699+0000","last_unstale":"2024-04-25T01:25:25.800109+0000","last_undegraded":"2024-04-25T01:25:25.800109+0000","last_fullsized":"2024-04-25T01:25:25.800109+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:57:43.608183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983737+0000","last_change":"2024-04-25T01:23:47.488742+0000","last_active":"2024-04-25T01:25:24.983737+0000","last_peered":"2024-04-25T01:25:24.983737+0000","last_clean":"2024-04-25T01:25:24.983737+0000","last_became_active":"2024-04-25T01:23:47.488193+0000","last_became_peered":"2024-04-25T01:23:47.488193+0000","last_unstale":"2024-04-25T01:25:24.983737+0000","last_undegraded":"2024-04-25T01:25:24.983737+0000","last_fullsized":"2024-04-25T01:25:24.983737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:02.658395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983788+0000","last_change":"2024-04-25T01:22:57.851138+0000","last_active":"2024-04-25T01:25:24.983788+0000","last_peered":"2024-04-25T01:25:24.983788+0000","last_clean":"2024-04-25T01:25:24.983788+0000","last_became_active":"2024-04-25T01:22:57.849107+0000","last_became_peered":"2024-04-25T01:22:57.849107+0000","last_unstale":"2024-04-25T01:25:24.983788+0000","last_undegraded":"2024-04-25T01:25:24.983788+0000","last_fullsized":"2024-04-25T01:25:24.983788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:47.665356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240803+0000","last_change":"2024-04-25T01:23:47.488162+0000","last_active":"2024-04-25T01:25:25.240803+0000","last_peered":"2024-04-25T01:25:25.240803+0000","last_clean":"2024-04-25T01:25:25.240803+0000","last_became_active":"2024-04-25T01:23:47.487731+0000","last_became_peered":"2024-04-25T01:23:47.487731+0000","last_unstale":"2024-04-25T01:25:25.240803+0000","last_undegraded":"2024-04-25T01:25:25.240803+0000","last_fullsized":"2024-04-25T01:25:25.240803+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:14.716313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117755+0000","last_change":"2024-04-25T01:22:57.851715+0000","last_active":"2024-04-25T01:25:25.117755+0000","last_peered":"2024-04-25T01:25:25.117755+0000","last_clean":"2024-04-25T01:25:25.117755+0000","last_became_active":"2024-04-25T01:22:57.847287+0000","last_became_peered":"2024-04-25T01:22:57.847287+0000","last_unstale":"2024-04-25T01:25:25.117755+0000","last_undegraded":"2024-04-25T01:25:25.117755+0000","last_fullsized":"2024-04-25T01:25:25.117755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.749931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800497+0000","last_change":"2024-04-25T01:22:57.846075+0000","last_active":"2024-04-25T01:25:25.800497+0000","last_peered":"2024-04-25T01:25:25.800497+0000","last_clean":"2024-04-25T01:25:25.800497+0000","last_became_active":"2024-04-25T01:22:57.845388+0000","last_became_peered":"2024-04-25T01:22:57.845388+0000","last_unstale":"2024-04-25T01:25:25.800497+0000","last_undegraded":"2024-04-25T01:25:25.800497+0000","last_fullsized":"2024-04-25T01:25:25.800497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:19.859806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984966+0000","last_change":"2024-04-25T01:22:57.851446+0000","last_active":"2024-04-25T01:25:24.984966+0000","last_peered":"2024-04-25T01:25:24.984966+0000","last_clean":"2024-04-25T01:25:24.984966+0000","last_became_active":"2024-04-25T01:22:57.848729+0000","last_became_peered":"2024-04-25T01:22:57.848729+0000","last_unstale":"2024-04-25T01:25:24.984966+0000","last_undegraded":"2024-04-25T01:25:24.984966+0000","last_fullsized":"2024-04-25T01:25:24.984966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:49.721024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984555+0000","last_change":"2024-04-25T01:22:57.847598+0000","last_active":"2024-04-25T01:25:24.984555+0000","last_peered":"2024-04-25T01:25:24.984555+0000","last_clean":"2024-04-25T01:25:24.984555+0000","last_became_active":"2024-04-25T01:22:57.847232+0000","last_became_peered":"2024-04-25T01:22:57.847232+0000","last_unstale":"2024-04-25T01:25:24.984555+0000","last_undegraded":"2024-04-25T01:25:24.984555+0000","last_fullsized":"2024-04-25T01:25:24.984555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.227700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093300+0000","last_change":"2024-04-25T01:22:57.842881+0000","last_active":"2024-04-25T01:25:25.093300+0000","last_peered":"2024-04-25T01:25:25.093300+0000","last_clean":"2024-04-25T01:25:25.093300+0000","last_became_active":"2024-04-25T01:22:57.842108+0000","last_became_peered":"2024-04-25T01:22:57.842108+0000","last_unstale":"2024-04-25T01:25:25.093300+0000","last_undegraded":"2024-04-25T01:25:25.093300+0000","last_fullsized":"2024-04-25T01:25:25.093300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:53.208188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983828+0000","last_change":"2024-04-25T01:22:57.860047+0000","last_active":"2024-04-25T01:25:24.983828+0000","last_peered":"2024-04-25T01:25:24.983828+0000","last_clean":"2024-04-25T01:25:24.983828+0000","last_became_active":"2024-04-25T01:22:57.859702+0000","last_became_peered":"2024-04-25T01:22:57.859702+0000","last_unstale":"2024-04-25T01:25:24.983828+0000","last_undegraded":"2024-04-25T01:25:24.983828+0000","last_fullsized":"2024-04-25T01:25:24.983828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:00.684055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239212+0000","last_change":"2024-04-25T01:22:57.847996+0000","last_active":"2024-04-25T01:25:25.239212+0000","last_peered":"2024-04-25T01:25:25.239212+0000","last_clean":"2024-04-25T01:25:25.239212+0000","last_became_active":"2024-04-25T01:22:57.847561+0000","last_became_peered":"2024-04-25T01:22:57.847561+0000","last_unstale":"2024-04-25T01:25:25.239212+0000","last_undegraded":"2024-04-25T01:25:25.239212+0000","last_fullsized":"2024-04-25T01:25:25.239212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:16.327558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117555+0000","last_change":"2024-04-25T01:22:57.845216+0000","last_active":"2024-04-25T01:25:25.117555+0000","last_peered":"2024-04-25T01:25:25.117555+0000","last_clean":"2024-04-25T01:25:25.117555+0000","last_became_active":"2024-04-25T01:22:57.844883+0000","last_became_peered":"2024-04-25T01:22:57.844883+0000","last_unstale":"2024-04-25T01:25:25.117555+0000","last_undegraded":"2024-04-25T01:25:25.117555+0000","last_fullsized":"2024-04-25T01:25:25.117555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:33.782048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800342+0000","last_change":"2024-04-25T01:22:57.853251+0000","last_active":"2024-04-25T01:25:25.800342+0000","last_peered":"2024-04-25T01:25:25.800342+0000","last_clean":"2024-04-25T01:25:25.800342+0000","last_became_active":"2024-04-25T01:22:57.852878+0000","last_became_peered":"2024-04-25T01:22:57.852878+0000","last_unstale":"2024-04-25T01:25:25.800342+0000","last_undegraded":"2024-04-25T01:25:25.800342+0000","last_fullsized":"2024-04-25T01:25:25.800342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:42.487837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107295,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599417,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":850,"num_read_kb":535,"num_write":289,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":64209,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":1798251,"data_compressed":89584,"data_compressed_allocated":118784,"data_compressed_original":1626112,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":529593,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32348,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712804,"statfs":{"total":95995035648,"available":95961911296,"internally_reserved":0,"allocated":1200128,"data_stored":1318507,"data_compressed":73161,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":927119,"data_compressed":48759,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":922921,"data_compressed":48759,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1126559,"data_compressed":65378,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":515088,"data_compressed":26623,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594570,"data_compressed":31240,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526371,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":526193,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1049993,"data_compressed":54915,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":600283,"data_compressed":40037,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1126400,"data_stored":1253356,"data_compressed":71534,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69677,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":71294,"data_compressed":10424,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133999,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203246,"data_compressed":7783,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:25:30.421 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T01:25:30.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:25:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.578+0000 7fdc47f5a640 1 -- 172.21.15.27:0/4120045760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 msgr2=0x7fdc4010b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.578+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/4120045760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 0x7fdc4010b820 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009900 tx=0x7fdc3002a480 comp rx=0 tx=0).stop 2024-04-25T01:25:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 -- 172.21.15.27:0/4120045760 shutdown_connections 2024-04-25T01:25:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/4120045760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc4010bd60 0x7fdc4010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/4120045760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 0x7fdc4010b820 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/4120045760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc40102a20 0x7fdc40102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 -- 172.21.15.27:0/4120045760 >> 172.21.15.27:0/4120045760 conn(0x7fdc400777a0 msgr2=0x7fdc40077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 -- 172.21.15.27:0/4120045760 shutdown_connections 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 -- 172.21.15.27:0/4120045760 wait complete. 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 Processor -- start 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 -- start start 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc40102a20 0x7fdc401980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 0x7fdc40198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc4010bd60 0x7fdc4019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdc40111b30 con 0x7fdc40103340 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdc401119b0 con 0x7fdc4010bd60 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.579+0000 7fdc47f5a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc40111cb0 con 0x7fdc40102a20 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 0x7fdc40198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 0x7fdc40198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50400/0 (socket says 172.21.15.27:50400) 2024-04-25T01:25:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 -- 172.21.15.27:0/3599750768 learned_addr learned my addr 172.21.15.27:0/3599750768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc45cf6640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc40102a20 0x7fdc401980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc464f7640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc4010bd60 0x7fdc4019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 -- 172.21.15.27:0/3599750768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc40102a20 msgr2=0x7fdc401980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc40102a20 0x7fdc401980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 -- 172.21.15.27:0/3599750768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc4010bd60 msgr2=0x7fdc4019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc4010bd60 0x7fdc4019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 -- 172.21.15.27:0/3599750768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4019d070 con 0x7fdc40103340 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc454f5640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 0x7fdc40198610 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdc30006010 tx=0x7fdc300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc36ffd640 1 -- 172.21.15.27:0/3599750768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3003f030 con 0x7fdc40103340 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc36ffd640 1 -- 172.21.15.27:0/3599750768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc30003ce0 con 0x7fdc40103340 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4019d300 con 0x7fdc40103340 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc36ffd640 1 -- 172.21.15.27:0/3599750768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc30035070 con 0x7fdc40103340 2024-04-25T01:25:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.580+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc401a4bf0 con 0x7fdc40103340 2024-04-25T01:25:30.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.581+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc400689b0 con 0x7fdc40103340 2024-04-25T01:25:30.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.582+0000 7fdc36ffd640 1 -- 172.21.15.27:0/3599750768 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7fdc30042030 con 0x7fdc40103340 2024-04-25T01:25:30.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.583+0000 7fdc36ffd640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc2c0a2e20 0x7fdc2c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.583+0000 7fdc45cf6640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc2c0a2e20 0x7fdc2c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.583+0000 7fdc36ffd640 1 -- 172.21.15.27:0/3599750768 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdc300efe20 con 0x7fdc40103340 2024-04-25T01:25:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.583+0000 7fdc45cf6640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc2c0a2e20 0x7fdc2c0a52e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00bbe0 tx=0x7fdc3c00b390 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.587+0000 7fdc36ffd640 1 -- 172.21.15.27:0/3599750768 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdc300bbfa0 con 0x7fdc40103340 2024-04-25T01:25:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.768+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc40061b80 con 0x7fdc2c0a2e20 2024-04-25T01:25:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.777+0000 7fdc36ffd640 1 -- 172.21.15.27:0/3599750768 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339699 (secure 0 0 0) 0x7fdc40061b80 con 0x7fdc2c0a2e20 2024-04-25T01:25:30.761 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:30.763 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T01:25:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.780+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc2c0a2e20 msgr2=0x7fdc2c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.780+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc2c0a2e20 0x7fdc2c0a52e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00bbe0 tx=0x7fdc3c00b390 comp rx=0 tx=0).stop 2024-04-25T01:25:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 msgr2=0x7fdc40198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 0x7fdc40198610 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdc30006010 tx=0x7fdc300042f0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 shutdown_connections 2024-04-25T01:25:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc4010bd60 0x7fdc4019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc2c0a2e20 0x7fdc2c0a52e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40103340 0x7fdc40198610 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 --2- 172.21.15.27:0/3599750768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc40102a20 0x7fdc401980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 >> 172.21.15.27:0/3599750768 conn(0x7fdc400777a0 msgr2=0x7fdc400fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 shutdown_connections 2024-04-25T01:25:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.781+0000 7fdc47f5a640 1 -- 172.21.15.27:0/3599750768 wait complete. 2024-04-25T01:25:30.772 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:25:28.998037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190019,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":986,"num_read_kb":654,"num_write":472,"num_write_kb":4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1244945,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361864,"kb_used_data":12672,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579960,"statfs":{"total":1151940427776,"available":1151569879040,"internally_reserved":0,"allocated":12976128,"data_stored":9890553,"data_compressed":549245,"data_compressed_allocated":741376,"data_compressed_original":4370432,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990220"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800504+0000","last_change":"2024-04-25T01:22:57.847700+0000","last_active":"2024-04-25T01:25:25.800504+0000","last_peered":"2024-04-25T01:25:25.800504+0000","last_clean":"2024-04-25T01:25:25.800504+0000","last_became_active":"2024-04-25T01:22:57.846877+0000","last_became_peered":"2024-04-25T01:22:57.846877+0000","last_unstale":"2024-04-25T01:25:25.800504+0000","last_undegraded":"2024-04-25T01:25:25.800504+0000","last_fullsized":"2024-04-25T01:25:25.800504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:05.918261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241338+0000","last_change":"2024-04-25T01:22:57.848396+0000","last_active":"2024-04-25T01:25:25.241338+0000","last_peered":"2024-04-25T01:25:25.241338+0000","last_clean":"2024-04-25T01:25:25.241338+0000","last_became_active":"2024-04-25T01:22:57.847971+0000","last_became_peered":"2024-04-25T01:22:57.847971+0000","last_unstale":"2024-04-25T01:25:25.241338+0000","last_undegraded":"2024-04-25T01:25:25.241338+0000","last_fullsized":"2024-04-25T01:25:25.241338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:13.143828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240866+0000","last_change":"2024-04-25T01:22:57.856908+0000","last_active":"2024-04-25T01:25:25.240866+0000","last_peered":"2024-04-25T01:25:25.240866+0000","last_clean":"2024-04-25T01:25:25.240866+0000","last_became_active":"2024-04-25T01:22:57.856218+0000","last_became_peered":"2024-04-25T01:22:57.856218+0000","last_unstale":"2024-04-25T01:25:25.240866+0000","last_undegraded":"2024-04-25T01:25:25.240866+0000","last_fullsized":"2024-04-25T01:25:25.240866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:51.971222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119108+0000","last_change":"2024-04-25T01:22:57.849022+0000","last_active":"2024-04-25T01:25:25.119108+0000","last_peered":"2024-04-25T01:25:25.119108+0000","last_clean":"2024-04-25T01:25:25.119108+0000","last_became_active":"2024-04-25T01:22:57.848303+0000","last_became_peered":"2024-04-25T01:22:57.848303+0000","last_unstale":"2024-04-25T01:25:25.119108+0000","last_undegraded":"2024-04-25T01:25:25.119108+0000","last_fullsized":"2024-04-25T01:25:25.119108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:52.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087107+0000","last_change":"2024-04-25T01:23:47.488312+0000","last_active":"2024-04-25T01:25:25.087107+0000","last_peered":"2024-04-25T01:25:25.087107+0000","last_clean":"2024-04-25T01:25:25.087107+0000","last_became_active":"2024-04-25T01:23:47.487923+0000","last_became_peered":"2024-04-25T01:23:47.487923+0000","last_unstale":"2024-04-25T01:25:25.087107+0000","last_undegraded":"2024-04-25T01:25:25.087107+0000","last_fullsized":"2024-04-25T01:25:25.087107+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:30.276853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983361+0000","last_change":"2024-04-25T01:22:57.855733+0000","last_active":"2024-04-25T01:25:24.983361+0000","last_peered":"2024-04-25T01:25:24.983361+0000","last_clean":"2024-04-25T01:25:24.983361+0000","last_became_active":"2024-04-25T01:22:57.855075+0000","last_became_peered":"2024-04-25T01:22:57.855075+0000","last_unstale":"2024-04-25T01:25:24.983361+0000","last_undegraded":"2024-04-25T01:25:24.983361+0000","last_fullsized":"2024-04-25T01:25:24.983361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:24.563461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238372+0000","last_change":"2024-04-25T01:22:57.849693+0000","last_active":"2024-04-25T01:25:25.238372+0000","last_peered":"2024-04-25T01:25:25.238372+0000","last_clean":"2024-04-25T01:25:25.238372+0000","last_became_active":"2024-04-25T01:22:57.849324+0000","last_became_peered":"2024-04-25T01:22:57.849324+0000","last_unstale":"2024-04-25T01:25:25.238372+0000","last_undegraded":"2024-04-25T01:25:25.238372+0000","last_fullsized":"2024-04-25T01:25:25.238372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:09.936496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983951+0000","last_change":"2024-04-25T01:22:57.847962+0000","last_active":"2024-04-25T01:25:24.983951+0000","last_peered":"2024-04-25T01:25:24.983951+0000","last_clean":"2024-04-25T01:25:24.983951+0000","last_became_active":"2024-04-25T01:22:57.847592+0000","last_became_peered":"2024-04-25T01:22:57.847592+0000","last_unstale":"2024-04-25T01:25:24.983951+0000","last_undegraded":"2024-04-25T01:25:24.983951+0000","last_fullsized":"2024-04-25T01:25:24.983951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:45.594118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239964+0000","last_change":"2024-04-25T01:22:57.860711+0000","last_active":"2024-04-25T01:25:25.239964+0000","last_peered":"2024-04-25T01:25:25.239964+0000","last_clean":"2024-04-25T01:25:25.239964+0000","last_became_active":"2024-04-25T01:22:57.860393+0000","last_became_peered":"2024-04-25T01:22:57.860393+0000","last_unstale":"2024-04-25T01:25:25.239964+0000","last_undegraded":"2024-04-25T01:25:25.239964+0000","last_fullsized":"2024-04-25T01:25:25.239964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:31.068095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984143+0000","last_change":"2024-04-25T01:22:57.853741+0000","last_active":"2024-04-25T01:25:24.984143+0000","last_peered":"2024-04-25T01:25:24.984143+0000","last_clean":"2024-04-25T01:25:24.984143+0000","last_became_active":"2024-04-25T01:22:57.851015+0000","last_became_peered":"2024-04-25T01:22:57.851015+0000","last_unstale":"2024-04-25T01:25:24.984143+0000","last_undegraded":"2024-04-25T01:25:24.984143+0000","last_fullsized":"2024-04-25T01:25:24.984143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:45.831335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239112+0000","last_change":"2024-04-25T01:22:57.853354+0000","last_active":"2024-04-25T01:25:25.239112+0000","last_peered":"2024-04-25T01:25:25.239112+0000","last_clean":"2024-04-25T01:25:25.239112+0000","last_became_active":"2024-04-25T01:22:57.853043+0000","last_became_peered":"2024-04-25T01:22:57.853043+0000","last_unstale":"2024-04-25T01:25:25.239112+0000","last_undegraded":"2024-04-25T01:25:25.239112+0000","last_fullsized":"2024-04-25T01:25:25.239112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:54.378751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119156+0000","last_change":"2024-04-25T01:22:57.855542+0000","last_active":"2024-04-25T01:25:25.119156+0000","last_peered":"2024-04-25T01:25:25.119156+0000","last_clean":"2024-04-25T01:25:25.119156+0000","last_became_active":"2024-04-25T01:22:57.855196+0000","last_became_peered":"2024-04-25T01:22:57.855196+0000","last_unstale":"2024-04-25T01:25:25.119156+0000","last_undegraded":"2024-04-25T01:25:25.119156+0000","last_fullsized":"2024-04-25T01:25:25.119156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:56.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117568+0000","last_change":"2024-04-25T01:22:57.829353+0000","last_active":"2024-04-25T01:25:25.117568+0000","last_peered":"2024-04-25T01:25:25.117568+0000","last_clean":"2024-04-25T01:25:25.117568+0000","last_became_active":"2024-04-25T01:22:57.828937+0000","last_became_peered":"2024-04-25T01:22:57.828937+0000","last_unstale":"2024-04-25T01:25:25.117568+0000","last_undegraded":"2024-04-25T01:25:25.117568+0000","last_fullsized":"2024-04-25T01:25:25.117568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:38.541556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241354+0000","last_change":"2024-04-25T01:22:57.854635+0000","last_active":"2024-04-25T01:25:25.241354+0000","last_peered":"2024-04-25T01:25:25.241354+0000","last_clean":"2024-04-25T01:25:25.241354+0000","last_became_active":"2024-04-25T01:22:57.854292+0000","last_became_peered":"2024-04-25T01:22:57.854292+0000","last_unstale":"2024-04-25T01:25:25.241354+0000","last_undegraded":"2024-04-25T01:25:25.241354+0000","last_fullsized":"2024-04-25T01:25:25.241354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:48.209059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240635+0000","last_change":"2024-04-25T01:22:57.852210+0000","last_active":"2024-04-25T01:25:25.240635+0000","last_peered":"2024-04-25T01:25:25.240635+0000","last_clean":"2024-04-25T01:25:25.240635+0000","last_became_active":"2024-04-25T01:22:57.851833+0000","last_became_peered":"2024-04-25T01:22:57.851833+0000","last_unstale":"2024-04-25T01:25:25.240635+0000","last_undegraded":"2024-04-25T01:25:25.240635+0000","last_fullsized":"2024-04-25T01:25:25.240635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:14.664785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239054+0000","last_change":"2024-04-25T01:22:57.849357+0000","last_active":"2024-04-25T01:25:25.239054+0000","last_peered":"2024-04-25T01:25:25.239054+0000","last_clean":"2024-04-25T01:25:25.239054+0000","last_became_active":"2024-04-25T01:22:57.849050+0000","last_became_peered":"2024-04-25T01:22:57.849050+0000","last_unstale":"2024-04-25T01:25:25.239054+0000","last_undegraded":"2024-04-25T01:25:25.239054+0000","last_fullsized":"2024-04-25T01:25:25.239054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:37.669424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800559+0000","last_change":"2024-04-25T01:22:57.844790+0000","last_active":"2024-04-25T01:25:25.800559+0000","last_peered":"2024-04-25T01:25:25.800559+0000","last_clean":"2024-04-25T01:25:25.800559+0000","last_became_active":"2024-04-25T01:22:57.844442+0000","last_became_peered":"2024-04-25T01:22:57.844442+0000","last_unstale":"2024-04-25T01:25:25.800559+0000","last_undegraded":"2024-04-25T01:25:25.800559+0000","last_fullsized":"2024-04-25T01:25:25.800559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:33.766034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085227+0000","last_change":"2024-04-25T01:22:57.844430+0000","last_active":"2024-04-25T01:25:25.085227+0000","last_peered":"2024-04-25T01:25:25.085227+0000","last_clean":"2024-04-25T01:25:25.085227+0000","last_became_active":"2024-04-25T01:22:57.844110+0000","last_became_peered":"2024-04-25T01:22:57.844110+0000","last_unstale":"2024-04-25T01:25:25.085227+0000","last_undegraded":"2024-04-25T01:25:25.085227+0000","last_fullsized":"2024-04-25T01:25:25.085227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:50.348249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984509+0000","last_change":"2024-04-25T01:22:57.852705+0000","last_active":"2024-04-25T01:25:24.984509+0000","last_peered":"2024-04-25T01:25:24.984509+0000","last_clean":"2024-04-25T01:25:24.984509+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:25:24.984509+0000","last_undegraded":"2024-04-25T01:25:24.984509+0000","last_fullsized":"2024-04-25T01:25:24.984509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:10.320459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983615+0000","last_change":"2024-04-25T01:23:01.879341+0000","last_active":"2024-04-25T01:25:24.983615+0000","last_peered":"2024-04-25T01:25:24.983615+0000","last_clean":"2024-04-25T01:25:24.983615+0000","last_became_active":"2024-04-25T01:23:01.878881+0000","last_became_peered":"2024-04-25T01:23:01.878881+0000","last_unstale":"2024-04-25T01:25:24.983615+0000","last_undegraded":"2024-04-25T01:25:24.983615+0000","last_fullsized":"2024-04-25T01:25:24.983615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:46.036868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085363+0000","last_change":"2024-04-25T01:22:56.839867+0000","last_active":"2024-04-25T01:25:25.085363+0000","last_peered":"2024-04-25T01:25:25.085363+0000","last_clean":"2024-04-25T01:25:25.085363+0000","last_became_active":"2024-04-25T01:22:56.839521+0000","last_became_peered":"2024-04-25T01:22:56.839521+0000","last_unstale":"2024-04-25T01:25:25.085363+0000","last_undegraded":"2024-04-25T01:25:25.085363+0000","last_fullsized":"2024-04-25T01:25:25.085363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:00.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984330+0000","last_change":"2024-04-25T01:22:57.855922+0000","last_active":"2024-04-25T01:25:24.984330+0000","last_peered":"2024-04-25T01:25:24.984330+0000","last_clean":"2024-04-25T01:25:24.984330+0000","last_became_active":"2024-04-25T01:22:57.855584+0000","last_became_peered":"2024-04-25T01:22:57.855584+0000","last_unstale":"2024-04-25T01:25:24.984330+0000","last_undegraded":"2024-04-25T01:25:24.984330+0000","last_fullsized":"2024-04-25T01:25:24.984330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:02.791269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118747+0000","last_change":"2024-04-25T01:23:01.872340+0000","last_active":"2024-04-25T01:25:25.118747+0000","last_peered":"2024-04-25T01:25:25.118747+0000","last_clean":"2024-04-25T01:25:25.118747+0000","last_became_active":"2024-04-25T01:23:01.871900+0000","last_became_peered":"2024-04-25T01:23:01.871900+0000","last_unstale":"2024-04-25T01:25:25.118747+0000","last_undegraded":"2024-04-25T01:25:25.118747+0000","last_fullsized":"2024-04-25T01:25:25.118747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:11.918022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984380+0000","last_change":"2024-04-25T01:22:56.843851+0000","last_active":"2024-04-25T01:25:24.984380+0000","last_peered":"2024-04-25T01:25:24.984380+0000","last_clean":"2024-04-25T01:25:24.984380+0000","last_became_active":"2024-04-25T01:22:56.843232+0000","last_became_peered":"2024-04-25T01:22:56.843232+0000","last_unstale":"2024-04-25T01:25:24.984380+0000","last_undegraded":"2024-04-25T01:25:24.984380+0000","last_fullsized":"2024-04-25T01:25:24.984380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:13.758328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241152+0000","last_change":"2024-04-25T01:22:57.857686+0000","last_active":"2024-04-25T01:25:25.241152+0000","last_peered":"2024-04-25T01:25:25.241152+0000","last_clean":"2024-04-25T01:25:25.241152+0000","last_became_active":"2024-04-25T01:22:57.857175+0000","last_became_peered":"2024-04-25T01:22:57.857175+0000","last_unstale":"2024-04-25T01:25:25.241152+0000","last_undegraded":"2024-04-25T01:25:25.241152+0000","last_fullsized":"2024-04-25T01:25:25.241152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:24.777861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118799+0000","last_change":"2024-04-25T01:23:01.867914+0000","last_active":"2024-04-25T01:25:25.118799+0000","last_peered":"2024-04-25T01:25:25.118799+0000","last_clean":"2024-04-25T01:25:25.118799+0000","last_became_active":"2024-04-25T01:23:01.867103+0000","last_became_peered":"2024-04-25T01:23:01.867103+0000","last_unstale":"2024-04-25T01:25:25.118799+0000","last_undegraded":"2024-04-25T01:25:25.118799+0000","last_fullsized":"2024-04-25T01:25:25.118799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:26.425433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087318+0000","last_change":"2024-04-25T01:22:56.841310+0000","last_active":"2024-04-25T01:25:25.087318+0000","last_peered":"2024-04-25T01:25:25.087318+0000","last_clean":"2024-04-25T01:25:25.087318+0000","last_became_active":"2024-04-25T01:22:56.840882+0000","last_became_peered":"2024-04-25T01:22:56.840882+0000","last_unstale":"2024-04-25T01:25:25.087318+0000","last_undegraded":"2024-04-25T01:25:25.087318+0000","last_fullsized":"2024-04-25T01:25:25.087318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:29.971448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239551+0000","last_change":"2024-04-25T01:22:57.842810+0000","last_active":"2024-04-25T01:25:25.239551+0000","last_peered":"2024-04-25T01:25:25.239551+0000","last_clean":"2024-04-25T01:25:25.239551+0000","last_became_active":"2024-04-25T01:22:57.842457+0000","last_became_peered":"2024-04-25T01:22:57.842457+0000","last_unstale":"2024-04-25T01:25:25.239551+0000","last_undegraded":"2024-04-25T01:25:25.239551+0000","last_fullsized":"2024-04-25T01:25:25.239551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:45.512037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119017+0000","last_change":"2024-04-25T01:23:01.877298+0000","last_active":"2024-04-25T01:25:25.119017+0000","last_peered":"2024-04-25T01:25:25.119017+0000","last_clean":"2024-04-25T01:25:25.119017+0000","last_became_active":"2024-04-25T01:23:01.876874+0000","last_became_peered":"2024-04-25T01:23:01.876874+0000","last_unstale":"2024-04-25T01:25:25.119017+0000","last_undegraded":"2024-04-25T01:25:25.119017+0000","last_fullsized":"2024-04-25T01:25:25.119017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:35:08.221388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800229+0000","last_change":"2024-04-25T01:24:47.495071+0000","last_active":"2024-04-25T01:25:25.800229+0000","last_peered":"2024-04-25T01:25:25.800229+0000","last_clean":"2024-04-25T01:25:25.800229+0000","last_became_active":"2024-04-25T01:24:47.491300+0000","last_became_peered":"2024-04-25T01:24:47.491300+0000","last_unstale":"2024-04-25T01:25:25.800229+0000","last_undegraded":"2024-04-25T01:25:25.800229+0000","last_fullsized":"2024-04-25T01:25:25.800229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:33:00.552625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093517+0000","last_change":"2024-04-25T01:22:57.851210+0000","last_active":"2024-04-25T01:25:25.093517+0000","last_peered":"2024-04-25T01:25:25.093517+0000","last_clean":"2024-04-25T01:25:25.093517+0000","last_became_active":"2024-04-25T01:22:57.850867+0000","last_became_peered":"2024-04-25T01:22:57.850867+0000","last_unstale":"2024-04-25T01:25:25.093517+0000","last_undegraded":"2024-04-25T01:25:25.093517+0000","last_fullsized":"2024-04-25T01:25:25.093517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:15.998433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984328+0000","last_change":"2024-04-25T01:23:47.489730+0000","last_active":"2024-04-25T01:25:24.984328+0000","last_peered":"2024-04-25T01:25:24.984328+0000","last_clean":"2024-04-25T01:25:24.984328+0000","last_became_active":"2024-04-25T01:23:47.489190+0000","last_became_peered":"2024-04-25T01:23:47.489190+0000","last_unstale":"2024-04-25T01:25:24.984328+0000","last_undegraded":"2024-04-25T01:25:24.984328+0000","last_fullsized":"2024-04-25T01:25:24.984328+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.101073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.237569+0000","last_change":"2024-04-25T01:22:56.834741+0000","last_active":"2024-04-25T01:25:25.237569+0000","last_peered":"2024-04-25T01:25:25.237569+0000","last_clean":"2024-04-25T01:25:25.237569+0000","last_became_active":"2024-04-25T01:22:56.834296+0000","last_became_peered":"2024-04-25T01:22:56.834296+0000","last_unstale":"2024-04-25T01:25:25.237569+0000","last_undegraded":"2024-04-25T01:25:25.237569+0000","last_fullsized":"2024-04-25T01:25:25.237569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:38.940603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087133+0000","last_change":"2024-04-25T01:22:57.854646+0000","last_active":"2024-04-25T01:25:25.087133+0000","last_peered":"2024-04-25T01:25:25.087133+0000","last_clean":"2024-04-25T01:25:25.087133+0000","last_became_active":"2024-04-25T01:22:57.853839+0000","last_became_peered":"2024-04-25T01:22:57.853839+0000","last_unstale":"2024-04-25T01:25:25.087133+0000","last_undegraded":"2024-04-25T01:25:25.087133+0000","last_fullsized":"2024-04-25T01:25:25.087133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:36.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984172+0000","last_change":"2024-04-25T01:23:01.875478+0000","last_active":"2024-04-25T01:25:24.984172+0000","last_peered":"2024-04-25T01:25:24.984172+0000","last_clean":"2024-04-25T01:25:24.984172+0000","last_became_active":"2024-04-25T01:23:01.874643+0000","last_became_peered":"2024-04-25T01:23:01.874643+0000","last_unstale":"2024-04-25T01:25:24.984172+0000","last_undegraded":"2024-04-25T01:25:24.984172+0000","last_fullsized":"2024-04-25T01:25:24.984172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.953507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983490+0000","last_change":"2024-04-25T01:22:56.843118+0000","last_active":"2024-04-25T01:25:24.983490+0000","last_peered":"2024-04-25T01:25:24.983490+0000","last_clean":"2024-04-25T01:25:24.983490+0000","last_became_active":"2024-04-25T01:22:56.842798+0000","last_became_peered":"2024-04-25T01:22:56.842798+0000","last_unstale":"2024-04-25T01:25:24.983490+0000","last_undegraded":"2024-04-25T01:25:24.983490+0000","last_fullsized":"2024-04-25T01:25:24.983490+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:00.054913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093690+0000","last_change":"2024-04-25T01:22:57.850265+0000","last_active":"2024-04-25T01:25:25.093690+0000","last_peered":"2024-04-25T01:25:25.093690+0000","last_clean":"2024-04-25T01:25:25.093690+0000","last_became_active":"2024-04-25T01:22:57.849480+0000","last_became_peered":"2024-04-25T01:22:57.849480+0000","last_unstale":"2024-04-25T01:25:25.093690+0000","last_undegraded":"2024-04-25T01:25:25.093690+0000","last_fullsized":"2024-04-25T01:25:25.093690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:48.577252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800384+0000","last_change":"2024-04-25T01:23:01.865369+0000","last_active":"2024-04-25T01:25:25.800384+0000","last_peered":"2024-04-25T01:25:25.800384+0000","last_clean":"2024-04-25T01:25:25.800384+0000","last_became_active":"2024-04-25T01:23:01.864872+0000","last_became_peered":"2024-04-25T01:23:01.864872+0000","last_unstale":"2024-04-25T01:25:25.800384+0000","last_undegraded":"2024-04-25T01:25:25.800384+0000","last_fullsized":"2024-04-25T01:25:25.800384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:56.266514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116988+0000","last_change":"2024-04-25T01:22:56.842576+0000","last_active":"2024-04-25T01:25:25.116988+0000","last_peered":"2024-04-25T01:25:25.116988+0000","last_clean":"2024-04-25T01:25:25.116988+0000","last_became_active":"2024-04-25T01:22:56.841081+0000","last_became_peered":"2024-04-25T01:22:56.841081+0000","last_unstale":"2024-04-25T01:25:25.116988+0000","last_undegraded":"2024-04-25T01:25:25.116988+0000","last_fullsized":"2024-04-25T01:25:25.116988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:08.174426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983862+0000","last_change":"2024-04-25T01:22:57.842812+0000","last_active":"2024-04-25T01:25:24.983862+0000","last_peered":"2024-04-25T01:25:24.983862+0000","last_clean":"2024-04-25T01:25:24.983862+0000","last_became_active":"2024-04-25T01:22:57.842462+0000","last_became_peered":"2024-04-25T01:22:57.842462+0000","last_unstale":"2024-04-25T01:25:24.983862+0000","last_undegraded":"2024-04-25T01:25:24.983862+0000","last_fullsized":"2024-04-25T01:25:24.983862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:14:13.444850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241074+0000","last_change":"2024-04-25T01:23:01.870401+0000","last_active":"2024-04-25T01:25:25.241074+0000","last_peered":"2024-04-25T01:25:25.241074+0000","last_clean":"2024-04-25T01:25:25.241074+0000","last_became_active":"2024-04-25T01:23:01.869932+0000","last_became_peered":"2024-04-25T01:23:01.869932+0000","last_unstale":"2024-04-25T01:25:25.241074+0000","last_undegraded":"2024-04-25T01:25:25.241074+0000","last_fullsized":"2024-04-25T01:25:25.241074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:13.323209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093295+0000","last_change":"2024-04-25T01:22:56.835946+0000","last_active":"2024-04-25T01:25:25.093295+0000","last_peered":"2024-04-25T01:25:25.093295+0000","last_clean":"2024-04-25T01:25:25.093295+0000","last_became_active":"2024-04-25T01:22:56.835630+0000","last_became_peered":"2024-04-25T01:22:56.835630+0000","last_unstale":"2024-04-25T01:25:25.093295+0000","last_undegraded":"2024-04-25T01:25:25.093295+0000","last_fullsized":"2024-04-25T01:25:25.093295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:02.128275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.084835+0000","last_change":"2024-04-25T01:22:57.844462+0000","last_active":"2024-04-25T01:25:25.084835+0000","last_peered":"2024-04-25T01:25:25.084835+0000","last_clean":"2024-04-25T01:25:25.084835+0000","last_became_active":"2024-04-25T01:22:57.843426+0000","last_became_peered":"2024-04-25T01:22:57.843426+0000","last_unstale":"2024-04-25T01:25:25.084835+0000","last_undegraded":"2024-04-25T01:25:25.084835+0000","last_fullsized":"2024-04-25T01:25:25.084835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:42.896622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984063+0000","last_change":"2024-04-25T01:23:01.871387+0000","last_active":"2024-04-25T01:25:24.984063+0000","last_peered":"2024-04-25T01:25:24.984063+0000","last_clean":"2024-04-25T01:25:24.984063+0000","last_became_active":"2024-04-25T01:23:01.870901+0000","last_became_peered":"2024-04-25T01:23:01.870901+0000","last_unstale":"2024-04-25T01:25:24.984063+0000","last_undegraded":"2024-04-25T01:25:24.984063+0000","last_fullsized":"2024-04-25T01:25:24.984063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:49.582515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238114+0000","last_change":"2024-04-25T01:22:56.834440+0000","last_active":"2024-04-25T01:25:25.238114+0000","last_peered":"2024-04-25T01:25:25.238114+0000","last_clean":"2024-04-25T01:25:25.238114+0000","last_became_active":"2024-04-25T01:22:56.834116+0000","last_became_peered":"2024-04-25T01:22:56.834116+0000","last_unstale":"2024-04-25T01:25:25.238114+0000","last_undegraded":"2024-04-25T01:25:25.238114+0000","last_fullsized":"2024-04-25T01:25:25.238114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:01.176601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239994+0000","last_change":"2024-04-25T01:22:57.853560+0000","last_active":"2024-04-25T01:25:25.239994+0000","last_peered":"2024-04-25T01:25:25.239994+0000","last_clean":"2024-04-25T01:25:25.239994+0000","last_became_active":"2024-04-25T01:22:57.853234+0000","last_became_peered":"2024-04-25T01:22:57.853234+0000","last_unstale":"2024-04-25T01:25:25.239994+0000","last_undegraded":"2024-04-25T01:25:25.239994+0000","last_fullsized":"2024-04-25T01:25:25.239994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:07:51.139371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.084730+0000","last_change":"2024-04-25T01:23:01.873856+0000","last_active":"2024-04-25T01:25:25.084730+0000","last_peered":"2024-04-25T01:25:25.084730+0000","last_clean":"2024-04-25T01:25:25.084730+0000","last_became_active":"2024-04-25T01:23:01.873413+0000","last_became_peered":"2024-04-25T01:23:01.873413+0000","last_unstale":"2024-04-25T01:25:25.084730+0000","last_undegraded":"2024-04-25T01:25:25.084730+0000","last_fullsized":"2024-04-25T01:25:25.084730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:11.802479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239188+0000","last_change":"2024-04-25T01:22:56.831080+0000","last_active":"2024-04-25T01:25:25.239188+0000","last_peered":"2024-04-25T01:25:25.239188+0000","last_clean":"2024-04-25T01:25:25.239188+0000","last_became_active":"2024-04-25T01:22:56.830735+0000","last_became_peered":"2024-04-25T01:22:56.830735+0000","last_unstale":"2024-04-25T01:25:25.239188+0000","last_undegraded":"2024-04-25T01:25:25.239188+0000","last_fullsized":"2024-04-25T01:25:25.239188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:38.418439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241002+0000","last_change":"2024-04-25T01:22:57.853838+0000","last_active":"2024-04-25T01:25:25.241002+0000","last_peered":"2024-04-25T01:25:25.241002+0000","last_clean":"2024-04-25T01:25:25.241002+0000","last_became_active":"2024-04-25T01:22:57.853023+0000","last_became_peered":"2024-04-25T01:22:57.853023+0000","last_unstale":"2024-04-25T01:25:25.241002+0000","last_undegraded":"2024-04-25T01:25:25.241002+0000","last_fullsized":"2024-04-25T01:25:25.241002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:18.515831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118964+0000","last_change":"2024-04-25T01:23:01.865446+0000","last_active":"2024-04-25T01:25:25.118964+0000","last_peered":"2024-04-25T01:25:25.118964+0000","last_clean":"2024-04-25T01:25:25.118964+0000","last_became_active":"2024-04-25T01:23:01.864937+0000","last_became_peered":"2024-04-25T01:23:01.864937+0000","last_unstale":"2024-04-25T01:25:25.118964+0000","last_undegraded":"2024-04-25T01:25:25.118964+0000","last_fullsized":"2024-04-25T01:25:25.118964+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:18.041430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093597+0000","last_change":"2024-04-25T01:22:56.836055+0000","last_active":"2024-04-25T01:25:25.093597+0000","last_peered":"2024-04-25T01:25:25.093597+0000","last_clean":"2024-04-25T01:25:25.093597+0000","last_became_active":"2024-04-25T01:22:56.835760+0000","last_became_peered":"2024-04-25T01:22:56.835760+0000","last_unstale":"2024-04-25T01:25:25.093597+0000","last_undegraded":"2024-04-25T01:25:25.093597+0000","last_fullsized":"2024-04-25T01:25:25.093597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:59.249996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239754+0000","last_change":"2024-04-25T01:22:57.852444+0000","last_active":"2024-04-25T01:25:25.239754+0000","last_peered":"2024-04-25T01:25:25.239754+0000","last_clean":"2024-04-25T01:25:25.239754+0000","last_became_active":"2024-04-25T01:22:57.852127+0000","last_became_peered":"2024-04-25T01:22:57.852127+0000","last_unstale":"2024-04-25T01:25:25.239754+0000","last_undegraded":"2024-04-25T01:25:25.239754+0000","last_fullsized":"2024-04-25T01:25:25.239754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:11.718512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983143+0000","last_change":"2024-04-25T01:23:47.039057+0000","last_active":"2024-04-25T01:25:24.983143+0000","last_peered":"2024-04-25T01:25:24.983143+0000","last_clean":"2024-04-25T01:25:24.983143+0000","last_became_active":"2024-04-25T01:23:47.038568+0000","last_became_peered":"2024-04-25T01:23:47.038568+0000","last_unstale":"2024-04-25T01:25:24.983143+0000","last_undegraded":"2024-04-25T01:25:24.983143+0000","last_fullsized":"2024-04-25T01:25:24.983143+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:23.828975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"94'39","reported_seq":346,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.549084+0000","last_change":"2024-04-25T01:24:47.501745+0000","last_active":"2024-04-25T01:25:25.549084+0000","last_peered":"2024-04-25T01:25:25.549084+0000","last_clean":"2024-04-25T01:25:25.549084+0000","last_became_active":"2024-04-25T01:24:47.494829+0000","last_became_peered":"2024-04-25T01:24:47.494829+0000","last_unstale":"2024-04-25T01:25:25.549084+0000","last_undegraded":"2024-04-25T01:25:25.549084+0000","last_fullsized":"2024-04-25T01:25:25.549084+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:54.482539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69677,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":252,"num_read_kb":194,"num_write":40,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":63605,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087512+0000","last_change":"2024-04-25T01:22:57.854554+0000","last_active":"2024-04-25T01:25:25.087512+0000","last_peered":"2024-04-25T01:25:25.087512+0000","last_clean":"2024-04-25T01:25:25.087512+0000","last_became_active":"2024-04-25T01:22:57.853328+0000","last_became_peered":"2024-04-25T01:22:57.853328+0000","last_unstale":"2024-04-25T01:25:25.087512+0000","last_undegraded":"2024-04-25T01:25:25.087512+0000","last_fullsized":"2024-04-25T01:25:25.087512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:21.774148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093976+0000","last_change":"2024-04-25T01:23:01.870571+0000","last_active":"2024-04-25T01:25:25.093976+0000","last_peered":"2024-04-25T01:25:25.093976+0000","last_clean":"2024-04-25T01:25:25.093976+0000","last_became_active":"2024-04-25T01:23:01.869982+0000","last_became_peered":"2024-04-25T01:23:01.869982+0000","last_unstale":"2024-04-25T01:25:25.093976+0000","last_undegraded":"2024-04-25T01:25:25.093976+0000","last_fullsized":"2024-04-25T01:25:25.093976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:17.294254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984043+0000","last_change":"2024-04-25T01:22:56.835134+0000","last_active":"2024-04-25T01:25:24.984043+0000","last_peered":"2024-04-25T01:25:24.984043+0000","last_clean":"2024-04-25T01:25:24.984043+0000","last_became_active":"2024-04-25T01:22:56.834207+0000","last_became_peered":"2024-04-25T01:22:56.834207+0000","last_unstale":"2024-04-25T01:25:24.984043+0000","last_undegraded":"2024-04-25T01:25:24.984043+0000","last_fullsized":"2024-04-25T01:25:24.984043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:07.979045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116949+0000","last_change":"2024-04-25T01:22:57.847693+0000","last_active":"2024-04-25T01:25:25.116949+0000","last_peered":"2024-04-25T01:25:25.116949+0000","last_clean":"2024-04-25T01:25:25.116949+0000","last_became_active":"2024-04-25T01:22:57.847211+0000","last_became_peered":"2024-04-25T01:22:57.847211+0000","last_unstale":"2024-04-25T01:25:25.116949+0000","last_undegraded":"2024-04-25T01:25:25.116949+0000","last_fullsized":"2024-04-25T01:25:25.116949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:48.295608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800279+0000","last_change":"2024-04-25T01:23:01.870340+0000","last_active":"2024-04-25T01:25:25.800279+0000","last_peered":"2024-04-25T01:25:25.800279+0000","last_clean":"2024-04-25T01:25:25.800279+0000","last_became_active":"2024-04-25T01:23:01.869662+0000","last_became_peered":"2024-04-25T01:23:01.869662+0000","last_unstale":"2024-04-25T01:25:25.800279+0000","last_undegraded":"2024-04-25T01:25:25.800279+0000","last_fullsized":"2024-04-25T01:25:25.800279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:05:57.313991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093495+0000","last_change":"2024-04-25T01:22:56.838146+0000","last_active":"2024-04-25T01:25:25.093495+0000","last_peered":"2024-04-25T01:25:25.093495+0000","last_clean":"2024-04-25T01:25:25.093495+0000","last_became_active":"2024-04-25T01:22:56.837815+0000","last_became_peered":"2024-04-25T01:22:56.837815+0000","last_unstale":"2024-04-25T01:25:25.093495+0000","last_undegraded":"2024-04-25T01:25:25.093495+0000","last_fullsized":"2024-04-25T01:25:25.093495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:14.942724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238786+0000","last_change":"2024-04-25T01:22:57.850840+0000","last_active":"2024-04-25T01:25:25.238786+0000","last_peered":"2024-04-25T01:25:25.238786+0000","last_clean":"2024-04-25T01:25:25.238786+0000","last_became_active":"2024-04-25T01:22:57.850255+0000","last_became_peered":"2024-04-25T01:22:57.850255+0000","last_unstale":"2024-04-25T01:25:25.238786+0000","last_undegraded":"2024-04-25T01:25:25.238786+0000","last_fullsized":"2024-04-25T01:25:25.238786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:25.887235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085296+0000","last_change":"2024-04-25T01:23:01.869396+0000","last_active":"2024-04-25T01:25:25.085296+0000","last_peered":"2024-04-25T01:25:25.085296+0000","last_clean":"2024-04-25T01:25:25.085296+0000","last_became_active":"2024-04-25T01:23:01.868777+0000","last_became_peered":"2024-04-25T01:23:01.868777+0000","last_unstale":"2024-04-25T01:25:25.085296+0000","last_undegraded":"2024-04-25T01:25:25.085296+0000","last_fullsized":"2024-04-25T01:25:25.085296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:37.276122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'4","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240662+0000","last_change":"2024-04-25T01:22:56.833289+0000","last_active":"2024-04-25T01:25:25.240662+0000","last_peered":"2024-04-25T01:25:25.240662+0000","last_clean":"2024-04-25T01:25:25.240662+0000","last_became_active":"2024-04-25T01:22:56.832353+0000","last_became_peered":"2024-04-25T01:22:56.832353+0000","last_unstale":"2024-04-25T01:25:25.240662+0000","last_undegraded":"2024-04-25T01:25:25.240662+0000","last_fullsized":"2024-04-25T01:25:25.240662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:18.251018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238649+0000","last_change":"2024-04-25T01:22:57.851974+0000","last_active":"2024-04-25T01:25:25.238649+0000","last_peered":"2024-04-25T01:25:25.238649+0000","last_clean":"2024-04-25T01:25:25.238649+0000","last_became_active":"2024-04-25T01:22:57.851573+0000","last_became_peered":"2024-04-25T01:22:57.851573+0000","last_unstale":"2024-04-25T01:25:25.238649+0000","last_undegraded":"2024-04-25T01:25:25.238649+0000","last_fullsized":"2024-04-25T01:25:25.238649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:27.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117895+0000","last_change":"2024-04-25T01:23:01.876980+0000","last_active":"2024-04-25T01:25:25.117895+0000","last_peered":"2024-04-25T01:25:25.117895+0000","last_clean":"2024-04-25T01:25:25.117895+0000","last_became_active":"2024-04-25T01:23:01.876548+0000","last_became_peered":"2024-04-25T01:23:01.876548+0000","last_unstale":"2024-04-25T01:25:25.117895+0000","last_undegraded":"2024-04-25T01:25:25.117895+0000","last_fullsized":"2024-04-25T01:25:25.117895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:13.470736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087000+0000","last_change":"2024-04-25T01:22:56.839694+0000","last_active":"2024-04-25T01:25:25.087000+0000","last_peered":"2024-04-25T01:25:25.087000+0000","last_clean":"2024-04-25T01:25:25.087000+0000","last_became_active":"2024-04-25T01:22:56.839367+0000","last_became_peered":"2024-04-25T01:22:56.839367+0000","last_unstale":"2024-04-25T01:25:25.087000+0000","last_undegraded":"2024-04-25T01:25:25.087000+0000","last_fullsized":"2024-04-25T01:25:25.087000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:28:49.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093465+0000","last_change":"2024-04-25T01:22:57.841526+0000","last_active":"2024-04-25T01:25:25.093465+0000","last_peered":"2024-04-25T01:25:25.093465+0000","last_clean":"2024-04-25T01:25:25.093465+0000","last_became_active":"2024-04-25T01:22:57.841192+0000","last_became_peered":"2024-04-25T01:22:57.841192+0000","last_unstale":"2024-04-25T01:25:25.093465+0000","last_undegraded":"2024-04-25T01:25:25.093465+0000","last_fullsized":"2024-04-25T01:25:25.093465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:38.958901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093301+0000","last_change":"2024-04-25T01:23:01.865486+0000","last_active":"2024-04-25T01:25:25.093301+0000","last_peered":"2024-04-25T01:25:25.093301+0000","last_clean":"2024-04-25T01:25:25.093301+0000","last_became_active":"2024-04-25T01:23:01.865068+0000","last_became_peered":"2024-04-25T01:23:01.865068+0000","last_unstale":"2024-04-25T01:25:25.093301+0000","last_undegraded":"2024-04-25T01:25:25.093301+0000","last_fullsized":"2024-04-25T01:25:25.093301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:50.432724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087557+0000","last_change":"2024-04-25T01:22:56.841507+0000","last_active":"2024-04-25T01:25:25.087557+0000","last_peered":"2024-04-25T01:25:25.087557+0000","last_clean":"2024-04-25T01:25:25.087557+0000","last_became_active":"2024-04-25T01:22:56.839970+0000","last_became_peered":"2024-04-25T01:22:56.839970+0000","last_unstale":"2024-04-25T01:25:25.087557+0000","last_undegraded":"2024-04-25T01:25:25.087557+0000","last_fullsized":"2024-04-25T01:25:25.087557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:37.670052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800378+0000","last_change":"2024-04-25T01:22:57.850853+0000","last_active":"2024-04-25T01:25:25.800378+0000","last_peered":"2024-04-25T01:25:25.800378+0000","last_clean":"2024-04-25T01:25:25.800378+0000","last_became_active":"2024-04-25T01:22:57.850498+0000","last_became_peered":"2024-04-25T01:22:57.850498+0000","last_unstale":"2024-04-25T01:25:25.800378+0000","last_undegraded":"2024-04-25T01:25:25.800378+0000","last_fullsized":"2024-04-25T01:25:25.800378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:13.271048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087021+0000","last_change":"2024-04-25T01:23:01.874254+0000","last_active":"2024-04-25T01:25:25.087021+0000","last_peered":"2024-04-25T01:25:25.087021+0000","last_clean":"2024-04-25T01:25:25.087021+0000","last_became_active":"2024-04-25T01:23:01.873711+0000","last_became_peered":"2024-04-25T01:23:01.873711+0000","last_unstale":"2024-04-25T01:25:25.087021+0000","last_undegraded":"2024-04-25T01:25:25.087021+0000","last_fullsized":"2024-04-25T01:25:25.087021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:13.510061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.085035+0000","last_change":"2024-04-25T01:22:56.827137+0000","last_active":"2024-04-25T01:25:25.085035+0000","last_peered":"2024-04-25T01:25:25.085035+0000","last_clean":"2024-04-25T01:25:25.085035+0000","last_became_active":"2024-04-25T01:22:56.826769+0000","last_became_peered":"2024-04-25T01:22:56.826769+0000","last_unstale":"2024-04-25T01:25:25.085035+0000","last_undegraded":"2024-04-25T01:25:25.085035+0000","last_fullsized":"2024-04-25T01:25:25.085035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:21.072643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983993+0000","last_change":"2024-04-25T01:22:57.843713+0000","last_active":"2024-04-25T01:25:24.983993+0000","last_peered":"2024-04-25T01:25:24.983993+0000","last_clean":"2024-04-25T01:25:24.983993+0000","last_became_active":"2024-04-25T01:22:57.843384+0000","last_became_peered":"2024-04-25T01:22:57.843384+0000","last_unstale":"2024-04-25T01:25:24.983993+0000","last_undegraded":"2024-04-25T01:25:24.983993+0000","last_fullsized":"2024-04-25T01:25:24.983993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:18.772379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117639+0000","last_change":"2024-04-25T01:23:01.874150+0000","last_active":"2024-04-25T01:25:25.117639+0000","last_peered":"2024-04-25T01:25:25.117639+0000","last_clean":"2024-04-25T01:25:25.117639+0000","last_became_active":"2024-04-25T01:23:01.873718+0000","last_became_peered":"2024-04-25T01:23:01.873718+0000","last_unstale":"2024-04-25T01:25:25.117639+0000","last_undegraded":"2024-04-25T01:25:25.117639+0000","last_fullsized":"2024-04-25T01:25:25.117639+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:26.317703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238669+0000","last_change":"2024-04-25T01:22:56.843605+0000","last_active":"2024-04-25T01:25:25.238669+0000","last_peered":"2024-04-25T01:25:25.238669+0000","last_clean":"2024-04-25T01:25:25.238669+0000","last_became_active":"2024-04-25T01:22:56.843313+0000","last_became_peered":"2024-04-25T01:22:56.843313+0000","last_unstale":"2024-04-25T01:25:25.238669+0000","last_undegraded":"2024-04-25T01:25:25.238669+0000","last_fullsized":"2024-04-25T01:25:25.238669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:41.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983293+0000","last_change":"2024-04-25T01:23:47.005378+0000","last_active":"2024-04-25T01:25:24.983293+0000","last_peered":"2024-04-25T01:25:24.983293+0000","last_clean":"2024-04-25T01:25:24.983293+0000","last_became_active":"2024-04-25T01:23:47.004962+0000","last_became_peered":"2024-04-25T01:23:47.004962+0000","last_unstale":"2024-04-25T01:25:24.983293+0000","last_undegraded":"2024-04-25T01:25:24.983293+0000","last_fullsized":"2024-04-25T01:25:24.983293+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:28.548285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117894+0000","last_change":"2024-04-25T01:23:01.872792+0000","last_active":"2024-04-25T01:25:25.117894+0000","last_peered":"2024-04-25T01:25:25.117894+0000","last_clean":"2024-04-25T01:25:25.117894+0000","last_became_active":"2024-04-25T01:23:01.872391+0000","last_became_peered":"2024-04-25T01:23:01.872391+0000","last_unstale":"2024-04-25T01:25:25.117894+0000","last_undegraded":"2024-04-25T01:25:25.117894+0000","last_fullsized":"2024-04-25T01:25:25.117894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:13.088996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093632+0000","last_change":"2024-04-25T01:22:57.843131+0000","last_active":"2024-04-25T01:25:25.093632+0000","last_peered":"2024-04-25T01:25:25.093632+0000","last_clean":"2024-04-25T01:25:25.093632+0000","last_became_active":"2024-04-25T01:22:57.842540+0000","last_became_peered":"2024-04-25T01:22:57.842540+0000","last_unstale":"2024-04-25T01:25:25.093632+0000","last_undegraded":"2024-04-25T01:25:25.093632+0000","last_fullsized":"2024-04-25T01:25:25.093632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:08:51.486718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117205+0000","last_change":"2024-04-25T01:22:56.844610+0000","last_active":"2024-04-25T01:25:25.117205+0000","last_peered":"2024-04-25T01:25:25.117205+0000","last_clean":"2024-04-25T01:25:25.117205+0000","last_became_active":"2024-04-25T01:22:56.844237+0000","last_became_peered":"2024-04-25T01:22:56.844237+0000","last_unstale":"2024-04-25T01:25:25.117205+0000","last_undegraded":"2024-04-25T01:25:25.117205+0000","last_fullsized":"2024-04-25T01:25:25.117205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:09.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238830+0000","last_change":"2024-04-25T01:23:01.865711+0000","last_active":"2024-04-25T01:25:25.238830+0000","last_peered":"2024-04-25T01:25:25.238830+0000","last_clean":"2024-04-25T01:25:25.238830+0000","last_became_active":"2024-04-25T01:23:01.865317+0000","last_became_peered":"2024-04-25T01:23:01.865317+0000","last_unstale":"2024-04-25T01:25:25.238830+0000","last_undegraded":"2024-04-25T01:25:25.238830+0000","last_fullsized":"2024-04-25T01:25:25.238830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:08.995490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983908+0000","last_change":"2024-04-25T01:22:57.851623+0000","last_active":"2024-04-25T01:25:24.983908+0000","last_peered":"2024-04-25T01:25:24.983908+0000","last_clean":"2024-04-25T01:25:24.983908+0000","last_became_active":"2024-04-25T01:22:57.850616+0000","last_became_peered":"2024-04-25T01:22:57.850616+0000","last_unstale":"2024-04-25T01:25:24.983908+0000","last_undegraded":"2024-04-25T01:25:24.983908+0000","last_fullsized":"2024-04-25T01:25:24.983908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:27.447996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118832+0000","last_change":"2024-04-25T01:22:56.832094+0000","last_active":"2024-04-25T01:25:25.118832+0000","last_peered":"2024-04-25T01:25:25.118832+0000","last_clean":"2024-04-25T01:25:25.118832+0000","last_became_active":"2024-04-25T01:22:56.831743+0000","last_became_peered":"2024-04-25T01:22:56.831743+0000","last_unstale":"2024-04-25T01:25:25.118832+0000","last_undegraded":"2024-04-25T01:25:25.118832+0000","last_fullsized":"2024-04-25T01:25:25.118832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:10:06.116081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238473+0000","last_change":"2024-04-25T01:21:40.899517+0000","last_active":"2024-04-25T01:25:25.238473+0000","last_peered":"2024-04-25T01:25:25.238473+0000","last_clean":"2024-04-25T01:25:25.238473+0000","last_became_active":"2024-04-25T01:21:40.899043+0000","last_became_peered":"2024-04-25T01:21:40.899043+0000","last_unstale":"2024-04-25T01:25:25.238473+0000","last_undegraded":"2024-04-25T01:25:25.238473+0000","last_fullsized":"2024-04-25T01:25:25.238473+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:12.053452+0000","last_clean_scrub_stamp":"2024-04-25T01:18:12.053452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:41.234852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093701+0000","last_change":"2024-04-25T01:23:01.878334+0000","last_active":"2024-04-25T01:25:25.093701+0000","last_peered":"2024-04-25T01:25:25.093701+0000","last_clean":"2024-04-25T01:25:25.093701+0000","last_became_active":"2024-04-25T01:23:01.877955+0000","last_became_peered":"2024-04-25T01:23:01.877955+0000","last_unstale":"2024-04-25T01:25:25.093701+0000","last_undegraded":"2024-04-25T01:25:25.093701+0000","last_fullsized":"2024-04-25T01:25:25.093701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:33.339430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.116577+0000","last_change":"2024-04-25T01:22:57.843020+0000","last_active":"2024-04-25T01:25:25.116577+0000","last_peered":"2024-04-25T01:25:25.116577+0000","last_clean":"2024-04-25T01:25:25.116577+0000","last_became_active":"2024-04-25T01:22:57.842671+0000","last_became_peered":"2024-04-25T01:22:57.842671+0000","last_unstale":"2024-04-25T01:25:25.116577+0000","last_undegraded":"2024-04-25T01:25:25.116577+0000","last_fullsized":"2024-04-25T01:25:25.116577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:53.831432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238821+0000","last_change":"2024-04-25T01:22:56.836106+0000","last_active":"2024-04-25T01:25:25.238821+0000","last_peered":"2024-04-25T01:25:25.238821+0000","last_clean":"2024-04-25T01:25:25.238821+0000","last_became_active":"2024-04-25T01:22:56.835814+0000","last_became_peered":"2024-04-25T01:22:56.835814+0000","last_unstale":"2024-04-25T01:25:25.238821+0000","last_undegraded":"2024-04-25T01:25:25.238821+0000","last_fullsized":"2024-04-25T01:25:25.238821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:18.004312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800091+0000","last_change":"2024-04-25T01:23:01.870723+0000","last_active":"2024-04-25T01:25:25.800091+0000","last_peered":"2024-04-25T01:25:25.800091+0000","last_clean":"2024-04-25T01:25:25.800091+0000","last_became_active":"2024-04-25T01:23:01.870205+0000","last_became_peered":"2024-04-25T01:23:01.870205+0000","last_unstale":"2024-04-25T01:25:25.800091+0000","last_undegraded":"2024-04-25T01:25:25.800091+0000","last_fullsized":"2024-04-25T01:25:25.800091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:47.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800282+0000","last_change":"2024-04-25T01:24:47.497526+0000","last_active":"2024-04-25T01:25:25.800282+0000","last_peered":"2024-04-25T01:25:25.800282+0000","last_clean":"2024-04-25T01:25:25.800282+0000","last_became_active":"2024-04-25T01:24:47.493380+0000","last_became_peered":"2024-04-25T01:24:47.493380+0000","last_unstale":"2024-04-25T01:25:25.800282+0000","last_undegraded":"2024-04-25T01:25:25.800282+0000","last_fullsized":"2024-04-25T01:25:25.800282+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:40:22.024954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117796+0000","last_change":"2024-04-25T01:22:57.841727+0000","last_active":"2024-04-25T01:25:25.117796+0000","last_peered":"2024-04-25T01:25:25.117796+0000","last_clean":"2024-04-25T01:25:25.117796+0000","last_became_active":"2024-04-25T01:22:57.841366+0000","last_became_peered":"2024-04-25T01:22:57.841366+0000","last_unstale":"2024-04-25T01:25:25.117796+0000","last_undegraded":"2024-04-25T01:25:25.117796+0000","last_fullsized":"2024-04-25T01:25:25.117796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:49:20.585916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983754+0000","last_change":"2024-04-25T01:23:01.869814+0000","last_active":"2024-04-25T01:25:24.983754+0000","last_peered":"2024-04-25T01:25:24.983754+0000","last_clean":"2024-04-25T01:25:24.983754+0000","last_became_active":"2024-04-25T01:23:01.869387+0000","last_became_peered":"2024-04-25T01:23:01.869387+0000","last_unstale":"2024-04-25T01:25:24.983754+0000","last_undegraded":"2024-04-25T01:25:24.983754+0000","last_fullsized":"2024-04-25T01:25:24.983754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:18:45.098295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'45","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.199682+0000","last_change":"2024-04-25T01:22:56.838506+0000","last_active":"2024-04-25T01:25:25.199682+0000","last_peered":"2024-04-25T01:25:25.199682+0000","last_clean":"2024-04-25T01:25:25.199682+0000","last_became_active":"2024-04-25T01:22:56.838185+0000","last_became_peered":"2024-04-25T01:22:56.838185+0000","last_unstale":"2024-04-25T01:25:25.199682+0000","last_undegraded":"2024-04-25T01:25:25.199682+0000","last_fullsized":"2024-04-25T01:25:25.199682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:35.306231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984432+0000","last_change":"2024-04-25T01:22:57.846544+0000","last_active":"2024-04-25T01:25:24.984432+0000","last_peered":"2024-04-25T01:25:24.984432+0000","last_clean":"2024-04-25T01:25:24.984432+0000","last_became_active":"2024-04-25T01:22:57.843507+0000","last_became_peered":"2024-04-25T01:22:57.843507+0000","last_unstale":"2024-04-25T01:25:24.984432+0000","last_undegraded":"2024-04-25T01:25:24.984432+0000","last_fullsized":"2024-04-25T01:25:24.984432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:30.600317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983013+0000","last_change":"2024-04-25T01:23:01.863986+0000","last_active":"2024-04-25T01:25:24.983013+0000","last_peered":"2024-04-25T01:25:24.983013+0000","last_clean":"2024-04-25T01:25:24.983013+0000","last_became_active":"2024-04-25T01:23:01.863397+0000","last_became_peered":"2024-04-25T01:23:01.863397+0000","last_unstale":"2024-04-25T01:25:24.983013+0000","last_undegraded":"2024-04-25T01:25:24.983013+0000","last_fullsized":"2024-04-25T01:25:24.983013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:45:56.229349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119308+0000","last_change":"2024-04-25T01:22:56.838201+0000","last_active":"2024-04-25T01:25:25.119308+0000","last_peered":"2024-04-25T01:25:25.119308+0000","last_clean":"2024-04-25T01:25:25.119308+0000","last_became_active":"2024-04-25T01:22:56.837890+0000","last_became_peered":"2024-04-25T01:22:56.837890+0000","last_unstale":"2024-04-25T01:25:25.119308+0000","last_undegraded":"2024-04-25T01:25:25.119308+0000","last_fullsized":"2024-04-25T01:25:25.119308+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:17.418112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.241186+0000","last_change":"2024-04-25T01:22:57.853923+0000","last_active":"2024-04-25T01:25:25.241186+0000","last_peered":"2024-04-25T01:25:25.241186+0000","last_clean":"2024-04-25T01:25:25.241186+0000","last_became_active":"2024-04-25T01:22:57.853566+0000","last_became_peered":"2024-04-25T01:22:57.853566+0000","last_unstale":"2024-04-25T01:25:25.241186+0000","last_undegraded":"2024-04-25T01:25:25.241186+0000","last_fullsized":"2024-04-25T01:25:25.241186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:36.509650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119340+0000","last_change":"2024-04-25T01:23:47.038657+0000","last_active":"2024-04-25T01:25:25.119340+0000","last_peered":"2024-04-25T01:25:25.119340+0000","last_clean":"2024-04-25T01:25:25.119340+0000","last_became_active":"2024-04-25T01:23:47.038168+0000","last_became_peered":"2024-04-25T01:23:47.038168+0000","last_unstale":"2024-04-25T01:25:25.119340+0000","last_undegraded":"2024-04-25T01:25:25.119340+0000","last_fullsized":"2024-04-25T01:25:25.119340+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:28:59.844333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239155+0000","last_change":"2024-04-25T01:22:56.828651+0000","last_active":"2024-04-25T01:25:25.239155+0000","last_peered":"2024-04-25T01:25:25.239155+0000","last_clean":"2024-04-25T01:25:25.239155+0000","last_became_active":"2024-04-25T01:22:56.828186+0000","last_became_peered":"2024-04-25T01:22:56.828186+0000","last_unstale":"2024-04-25T01:25:25.239155+0000","last_undegraded":"2024-04-25T01:25:25.239155+0000","last_fullsized":"2024-04-25T01:25:25.239155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:56.020822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.086946+0000","last_change":"2024-04-25T01:22:57.849768+0000","last_active":"2024-04-25T01:25:25.086946+0000","last_peered":"2024-04-25T01:25:25.086946+0000","last_clean":"2024-04-25T01:25:25.086946+0000","last_became_active":"2024-04-25T01:22:57.849418+0000","last_became_peered":"2024-04-25T01:22:57.849418+0000","last_unstale":"2024-04-25T01:25:25.086946+0000","last_undegraded":"2024-04-25T01:25:25.086946+0000","last_fullsized":"2024-04-25T01:25:25.086946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:41.292915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087501+0000","last_change":"2024-04-25T01:23:47.148790+0000","last_active":"2024-04-25T01:25:25.087501+0000","last_peered":"2024-04-25T01:25:25.087501+0000","last_clean":"2024-04-25T01:25:25.087501+0000","last_became_active":"2024-04-25T01:23:47.148180+0000","last_became_peered":"2024-04-25T01:23:47.148180+0000","last_unstale":"2024-04-25T01:25:25.087501+0000","last_undegraded":"2024-04-25T01:25:25.087501+0000","last_fullsized":"2024-04-25T01:25:25.087501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:08.296696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983834+0000","last_change":"2024-04-25T01:22:56.844672+0000","last_active":"2024-04-25T01:25:24.983834+0000","last_peered":"2024-04-25T01:25:24.983834+0000","last_clean":"2024-04-25T01:25:24.983834+0000","last_became_active":"2024-04-25T01:22:56.844141+0000","last_became_peered":"2024-04-25T01:22:56.844141+0000","last_unstale":"2024-04-25T01:25:24.983834+0000","last_undegraded":"2024-04-25T01:25:24.983834+0000","last_fullsized":"2024-04-25T01:25:24.983834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:14.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800019+0000","last_change":"2024-04-25T01:22:57.845006+0000","last_active":"2024-04-25T01:25:25.800019+0000","last_peered":"2024-04-25T01:25:25.800019+0000","last_clean":"2024-04-25T01:25:25.800019+0000","last_became_active":"2024-04-25T01:22:57.844672+0000","last_became_peered":"2024-04-25T01:22:57.844672+0000","last_unstale":"2024-04-25T01:25:25.800019+0000","last_undegraded":"2024-04-25T01:25:25.800019+0000","last_fullsized":"2024-04-25T01:25:25.800019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:18.238512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119458+0000","last_change":"2024-04-25T01:23:01.869438+0000","last_active":"2024-04-25T01:25:25.119458+0000","last_peered":"2024-04-25T01:25:25.119458+0000","last_clean":"2024-04-25T01:25:25.119458+0000","last_became_active":"2024-04-25T01:23:01.868985+0000","last_became_peered":"2024-04-25T01:23:01.868985+0000","last_unstale":"2024-04-25T01:25:25.119458+0000","last_undegraded":"2024-04-25T01:25:25.119458+0000","last_fullsized":"2024-04-25T01:25:25.119458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:35.509098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983953+0000","last_change":"2024-04-25T01:22:56.834475+0000","last_active":"2024-04-25T01:25:24.983953+0000","last_peered":"2024-04-25T01:25:24.983953+0000","last_clean":"2024-04-25T01:25:24.983953+0000","last_became_active":"2024-04-25T01:22:56.834076+0000","last_became_peered":"2024-04-25T01:22:56.834076+0000","last_unstale":"2024-04-25T01:25:24.983953+0000","last_undegraded":"2024-04-25T01:25:24.983953+0000","last_fullsized":"2024-04-25T01:25:24.983953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:55:47.464009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984196+0000","last_change":"2024-04-25T01:22:57.860549+0000","last_active":"2024-04-25T01:25:24.984196+0000","last_peered":"2024-04-25T01:25:24.984196+0000","last_clean":"2024-04-25T01:25:24.984196+0000","last_became_active":"2024-04-25T01:22:57.860226+0000","last_became_peered":"2024-04-25T01:22:57.860226+0000","last_unstale":"2024-04-25T01:25:24.984196+0000","last_undegraded":"2024-04-25T01:25:24.984196+0000","last_fullsized":"2024-04-25T01:25:24.984196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:10.663508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.119226+0000","last_change":"2024-04-25T01:23:01.867602+0000","last_active":"2024-04-25T01:25:25.119226+0000","last_peered":"2024-04-25T01:25:25.119226+0000","last_clean":"2024-04-25T01:25:25.119226+0000","last_became_active":"2024-04-25T01:23:01.867150+0000","last_became_peered":"2024-04-25T01:23:01.867150+0000","last_unstale":"2024-04-25T01:25:25.119226+0000","last_undegraded":"2024-04-25T01:25:25.119226+0000","last_fullsized":"2024-04-25T01:25:25.119226+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:34.571031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.087024+0000","last_change":"2024-04-25T01:22:56.841082+0000","last_active":"2024-04-25T01:25:25.087024+0000","last_peered":"2024-04-25T01:25:25.087024+0000","last_clean":"2024-04-25T01:25:25.087024+0000","last_became_active":"2024-04-25T01:22:56.839856+0000","last_became_peered":"2024-04-25T01:22:56.839856+0000","last_unstale":"2024-04-25T01:25:25.087024+0000","last_undegraded":"2024-04-25T01:25:25.087024+0000","last_fullsized":"2024-04-25T01:25:25.087024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:14.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238762+0000","last_change":"2024-04-25T01:22:57.852242+0000","last_active":"2024-04-25T01:25:25.238762+0000","last_peered":"2024-04-25T01:25:25.238762+0000","last_clean":"2024-04-25T01:25:25.238762+0000","last_became_active":"2024-04-25T01:22:57.851727+0000","last_became_peered":"2024-04-25T01:22:57.851727+0000","last_unstale":"2024-04-25T01:25:25.238762+0000","last_undegraded":"2024-04-25T01:25:25.238762+0000","last_fullsized":"2024-04-25T01:25:25.238762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:38:02.810025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238989+0000","last_change":"2024-04-25T01:23:01.871074+0000","last_active":"2024-04-25T01:25:25.238989+0000","last_peered":"2024-04-25T01:25:25.238989+0000","last_clean":"2024-04-25T01:25:25.238989+0000","last_became_active":"2024-04-25T01:23:01.870570+0000","last_became_peered":"2024-04-25T01:23:01.870570+0000","last_unstale":"2024-04-25T01:25:25.238989+0000","last_undegraded":"2024-04-25T01:25:25.238989+0000","last_fullsized":"2024-04-25T01:25:25.238989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:14.476400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.118784+0000","last_change":"2024-04-25T01:23:47.011202+0000","last_active":"2024-04-25T01:25:25.118784+0000","last_peered":"2024-04-25T01:25:25.118784+0000","last_clean":"2024-04-25T01:25:25.118784+0000","last_became_active":"2024-04-25T01:23:47.006425+0000","last_became_peered":"2024-04-25T01:23:47.006425+0000","last_unstale":"2024-04-25T01:25:25.118784+0000","last_undegraded":"2024-04-25T01:25:25.118784+0000","last_fullsized":"2024-04-25T01:25:25.118784+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:51:46.791184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239193+0000","last_change":"2024-04-25T01:22:57.843173+0000","last_active":"2024-04-25T01:25:25.239193+0000","last_peered":"2024-04-25T01:25:25.239193+0000","last_clean":"2024-04-25T01:25:25.239193+0000","last_became_active":"2024-04-25T01:22:57.842734+0000","last_became_peered":"2024-04-25T01:22:57.842734+0000","last_unstale":"2024-04-25T01:25:25.239193+0000","last_undegraded":"2024-04-25T01:25:25.239193+0000","last_fullsized":"2024-04-25T01:25:25.239193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:33.197493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240612+0000","last_change":"2024-04-25T01:23:01.864068+0000","last_active":"2024-04-25T01:25:25.240612+0000","last_peered":"2024-04-25T01:25:25.240612+0000","last_clean":"2024-04-25T01:25:25.240612+0000","last_became_active":"2024-04-25T01:23:01.863640+0000","last_became_peered":"2024-04-25T01:23:01.863640+0000","last_unstale":"2024-04-25T01:25:25.240612+0000","last_undegraded":"2024-04-25T01:25:25.240612+0000","last_fullsized":"2024-04-25T01:25:25.240612+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:26.925012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.194844+0000","last_change":"2024-04-25T01:22:56.837969+0000","last_active":"2024-04-25T01:25:25.194844+0000","last_peered":"2024-04-25T01:25:25.194844+0000","last_clean":"2024-04-25T01:25:25.194844+0000","last_became_active":"2024-04-25T01:22:56.837641+0000","last_became_peered":"2024-04-25T01:22:56.837641+0000","last_unstale":"2024-04-25T01:25:25.194844+0000","last_undegraded":"2024-04-25T01:25:25.194844+0000","last_fullsized":"2024-04-25T01:25:25.194844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:04.376038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.238624+0000","last_change":"2024-04-25T01:22:57.851057+0000","last_active":"2024-04-25T01:25:25.238624+0000","last_peered":"2024-04-25T01:25:25.238624+0000","last_clean":"2024-04-25T01:25:25.238624+0000","last_became_active":"2024-04-25T01:22:57.850278+0000","last_became_peered":"2024-04-25T01:22:57.850278+0000","last_unstale":"2024-04-25T01:25:25.238624+0000","last_undegraded":"2024-04-25T01:25:25.238624+0000","last_fullsized":"2024-04-25T01:25:25.238624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:15.332193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093137+0000","last_change":"2024-04-25T01:23:01.866673+0000","last_active":"2024-04-25T01:25:25.093137+0000","last_peered":"2024-04-25T01:25:25.093137+0000","last_clean":"2024-04-25T01:25:25.093137+0000","last_became_active":"2024-04-25T01:23:01.866210+0000","last_became_peered":"2024-04-25T01:23:01.866210+0000","last_unstale":"2024-04-25T01:25:25.093137+0000","last_undegraded":"2024-04-25T01:25:25.093137+0000","last_fullsized":"2024-04-25T01:25:25.093137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:23:00.844545+0000","last_clean_scrub_stamp":"2024-04-25T01:23:00.844545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:37:20.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117961+0000","last_change":"2024-04-25T01:22:56.844590+0000","last_active":"2024-04-25T01:25:25.117961+0000","last_peered":"2024-04-25T01:25:25.117961+0000","last_clean":"2024-04-25T01:25:25.117961+0000","last_became_active":"2024-04-25T01:22:56.844279+0000","last_became_peered":"2024-04-25T01:22:56.844279+0000","last_unstale":"2024-04-25T01:25:25.117961+0000","last_undegraded":"2024-04-25T01:25:25.117961+0000","last_fullsized":"2024-04-25T01:25:25.117961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:55.815799+0000","last_clean_scrub_stamp":"2024-04-25T01:22:55.815799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:47.155832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239487+0000","last_change":"2024-04-25T01:22:57.858850+0000","last_active":"2024-04-25T01:25:25.239487+0000","last_peered":"2024-04-25T01:25:25.239487+0000","last_clean":"2024-04-25T01:25:25.239487+0000","last_became_active":"2024-04-25T01:22:57.858498+0000","last_became_peered":"2024-04-25T01:22:57.858498+0000","last_unstale":"2024-04-25T01:25:25.239487+0000","last_undegraded":"2024-04-25T01:25:25.239487+0000","last_fullsized":"2024-04-25T01:25:25.239487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:04.836062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800109+0000","last_change":"2024-04-25T01:23:47.007111+0000","last_active":"2024-04-25T01:25:25.800109+0000","last_peered":"2024-04-25T01:25:25.800109+0000","last_clean":"2024-04-25T01:25:25.800109+0000","last_became_active":"2024-04-25T01:23:47.006699+0000","last_became_peered":"2024-04-25T01:23:47.006699+0000","last_unstale":"2024-04-25T01:25:25.800109+0000","last_undegraded":"2024-04-25T01:25:25.800109+0000","last_fullsized":"2024-04-25T01:25:25.800109+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:57:43.608183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983737+0000","last_change":"2024-04-25T01:23:47.488742+0000","last_active":"2024-04-25T01:25:24.983737+0000","last_peered":"2024-04-25T01:25:24.983737+0000","last_clean":"2024-04-25T01:25:24.983737+0000","last_became_active":"2024-04-25T01:23:47.488193+0000","last_became_peered":"2024-04-25T01:23:47.488193+0000","last_unstale":"2024-04-25T01:25:24.983737+0000","last_undegraded":"2024-04-25T01:25:24.983737+0000","last_fullsized":"2024-04-25T01:25:24.983737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:02.658395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983788+0000","last_change":"2024-04-25T01:22:57.851138+0000","last_active":"2024-04-25T01:25:24.983788+0000","last_peered":"2024-04-25T01:25:24.983788+0000","last_clean":"2024-04-25T01:25:24.983788+0000","last_became_active":"2024-04-25T01:22:57.849107+0000","last_became_peered":"2024-04-25T01:22:57.849107+0000","last_unstale":"2024-04-25T01:25:24.983788+0000","last_undegraded":"2024-04-25T01:25:24.983788+0000","last_fullsized":"2024-04-25T01:25:24.983788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:47.665356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.240803+0000","last_change":"2024-04-25T01:23:47.488162+0000","last_active":"2024-04-25T01:25:25.240803+0000","last_peered":"2024-04-25T01:25:25.240803+0000","last_clean":"2024-04-25T01:25:25.240803+0000","last_became_active":"2024-04-25T01:23:47.487731+0000","last_became_peered":"2024-04-25T01:23:47.487731+0000","last_unstale":"2024-04-25T01:25:25.240803+0000","last_undegraded":"2024-04-25T01:25:25.240803+0000","last_fullsized":"2024-04-25T01:25:25.240803+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:14.716313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117755+0000","last_change":"2024-04-25T01:22:57.851715+0000","last_active":"2024-04-25T01:25:25.117755+0000","last_peered":"2024-04-25T01:25:25.117755+0000","last_clean":"2024-04-25T01:25:25.117755+0000","last_became_active":"2024-04-25T01:22:57.847287+0000","last_became_peered":"2024-04-25T01:22:57.847287+0000","last_unstale":"2024-04-25T01:25:25.117755+0000","last_undegraded":"2024-04-25T01:25:25.117755+0000","last_fullsized":"2024-04-25T01:25:25.117755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.749931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800497+0000","last_change":"2024-04-25T01:22:57.846075+0000","last_active":"2024-04-25T01:25:25.800497+0000","last_peered":"2024-04-25T01:25:25.800497+0000","last_clean":"2024-04-25T01:25:25.800497+0000","last_became_active":"2024-04-25T01:22:57.845388+0000","last_became_peered":"2024-04-25T01:22:57.845388+0000","last_unstale":"2024-04-25T01:25:25.800497+0000","last_undegraded":"2024-04-25T01:25:25.800497+0000","last_fullsized":"2024-04-25T01:25:25.800497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:19.859806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984966+0000","last_change":"2024-04-25T01:22:57.851446+0000","last_active":"2024-04-25T01:25:24.984966+0000","last_peered":"2024-04-25T01:25:24.984966+0000","last_clean":"2024-04-25T01:25:24.984966+0000","last_became_active":"2024-04-25T01:22:57.848729+0000","last_became_peered":"2024-04-25T01:22:57.848729+0000","last_unstale":"2024-04-25T01:25:24.984966+0000","last_undegraded":"2024-04-25T01:25:24.984966+0000","last_fullsized":"2024-04-25T01:25:24.984966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:49.721024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.984555+0000","last_change":"2024-04-25T01:22:57.847598+0000","last_active":"2024-04-25T01:25:24.984555+0000","last_peered":"2024-04-25T01:25:24.984555+0000","last_clean":"2024-04-25T01:25:24.984555+0000","last_became_active":"2024-04-25T01:22:57.847232+0000","last_became_peered":"2024-04-25T01:22:57.847232+0000","last_unstale":"2024-04-25T01:25:24.984555+0000","last_undegraded":"2024-04-25T01:25:24.984555+0000","last_fullsized":"2024-04-25T01:25:24.984555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.227700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.093300+0000","last_change":"2024-04-25T01:22:57.842881+0000","last_active":"2024-04-25T01:25:25.093300+0000","last_peered":"2024-04-25T01:25:25.093300+0000","last_clean":"2024-04-25T01:25:25.093300+0000","last_became_active":"2024-04-25T01:22:57.842108+0000","last_became_peered":"2024-04-25T01:22:57.842108+0000","last_unstale":"2024-04-25T01:25:25.093300+0000","last_undegraded":"2024-04-25T01:25:25.093300+0000","last_fullsized":"2024-04-25T01:25:25.093300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:53.208188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:24.983828+0000","last_change":"2024-04-25T01:22:57.860047+0000","last_active":"2024-04-25T01:25:24.983828+0000","last_peered":"2024-04-25T01:25:24.983828+0000","last_clean":"2024-04-25T01:25:24.983828+0000","last_became_active":"2024-04-25T01:22:57.859702+0000","last_became_peered":"2024-04-25T01:22:57.859702+0000","last_unstale":"2024-04-25T01:25:24.983828+0000","last_undegraded":"2024-04-25T01:25:24.983828+0000","last_fullsized":"2024-04-25T01:25:24.983828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:00.684055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.239212+0000","last_change":"2024-04-25T01:22:57.847996+0000","last_active":"2024-04-25T01:25:25.239212+0000","last_peered":"2024-04-25T01:25:25.239212+0000","last_clean":"2024-04-25T01:25:25.239212+0000","last_became_active":"2024-04-25T01:22:57.847561+0000","last_became_peered":"2024-04-25T01:22:57.847561+0000","last_unstale":"2024-04-25T01:25:25.239212+0000","last_undegraded":"2024-04-25T01:25:25.239212+0000","last_fullsized":"2024-04-25T01:25:25.239212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:59:16.327558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.117555+0000","last_change":"2024-04-25T01:22:57.845216+0000","last_active":"2024-04-25T01:25:25.117555+0000","last_peered":"2024-04-25T01:25:25.117555+0000","last_clean":"2024-04-25T01:25:25.117555+0000","last_became_active":"2024-04-25T01:22:57.844883+0000","last_became_peered":"2024-04-25T01:22:57.844883+0000","last_unstale":"2024-04-25T01:25:25.117555+0000","last_undegraded":"2024-04-25T01:25:25.117555+0000","last_fullsized":"2024-04-25T01:25:25.117555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:33.782048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:25:25.800342+0000","last_change":"2024-04-25T01:22:57.853251+0000","last_active":"2024-04-25T01:25:25.800342+0000","last_peered":"2024-04-25T01:25:25.800342+0000","last_clean":"2024-04-25T01:25:25.800342+0000","last_became_active":"2024-04-25T01:22:57.852878+0000","last_became_peered":"2024-04-25T01:22:57.852878+0000","last_unstale":"2024-04-25T01:25:25.800342+0000","last_undegraded":"2024-04-25T01:25:25.800342+0000","last_fullsized":"2024-04-25T01:25:25.800342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:22:56.817527+0000","last_clean_scrub_stamp":"2024-04-25T01:22:56.817527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:42.487837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107295,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599417,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":850,"num_read_kb":535,"num_write":289,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":64209,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":1798251,"data_compressed":89584,"data_compressed_allocated":118784,"data_compressed_original":1626112,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":529593,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32348,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712804,"statfs":{"total":95995035648,"available":95961911296,"internally_reserved":0,"allocated":1200128,"data_stored":1318507,"data_compressed":73161,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":927119,"data_compressed":48759,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":922921,"data_compressed":48759,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":1126559,"data_compressed":65378,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":515088,"data_compressed":26623,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594570,"data_compressed":31240,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526371,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":526193,"data_compressed":29613,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1049993,"data_compressed":54915,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":600283,"data_compressed":40037,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1126400,"data_stored":1253356,"data_compressed":71534,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35765,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69677,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":71294,"data_compressed":10424,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133999,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203246,"data_compressed":7783,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:25:30.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 -- 172.21.15.27:0/797168334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e01017c0 msgr2=0x7f12e0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 --2- 172.21.15.27:0/797168334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e01017c0 0x7f12e0101c20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f12d0009900 tx=0x7f12d002a480 comp rx=0 tx=0).stop 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 -- 172.21.15.27:0/797168334 shutdown_connections 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 --2- 172.21.15.27:0/797168334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e0102160 0x7f12e010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 --2- 172.21.15.27:0/797168334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e01017c0 0x7f12e0101c20 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 --2- 172.21.15.27:0/797168334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12e0107750 0x7f12e0107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 -- 172.21.15.27:0/797168334 >> 172.21.15.27:0/797168334 conn(0x7f12e00788d0 msgr2=0x7f12e0078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 -- 172.21.15.27:0/797168334 shutdown_connections 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.934+0000 7f12e6c66640 1 -- 172.21.15.27:0/797168334 wait complete. 2024-04-25T01:25:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e6c66640 1 Processor -- start 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e6c66640 1 -- start start 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e6c66640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01017c0 0x7f12e0073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e6c66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12e0102160 0x7f12e0073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e6c66640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e0107750 0x7f12e00744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e6c66640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0111b10 con 0x7f12e0107750 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e6c66640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0111990 con 0x7f12e01017c0 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e6c66640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0111c90 con 0x7f12e0102160 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e5203640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e0107750 0x7f12e00744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e5203640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e0107750 0x7f12e00744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50418/0 (socket says 172.21.15.27:50418) 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e5203640 1 -- 172.21.15.27:0/2430237075 learned_addr learned my addr 172.21.15.27:0/2430237075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12d7fff640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12e0102160 0x7f12e0073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e4a02640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01017c0 0x7f12e0073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.935+0000 7f12e5203640 1 -- 172.21.15.27:0/2430237075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12e0102160 msgr2=0x7f12e0073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12e5203640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12e0102160 0x7f12e0073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12e5203640 1 -- 172.21.15.27:0/2430237075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01017c0 msgr2=0x7f12e0073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12e5203640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01017c0 0x7f12e0073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12e5203640 1 -- 172.21.15.27:0/2430237075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e01a64f0 con 0x7f12e0107750 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12e4a02640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01017c0 0x7f12e0073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12e5203640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e0107750 0x7f12e00744c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f12d800eaa0 tx=0x7f12d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12d5ffb640 1 -- 172.21.15.27:0/2430237075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d800cdb0 con 0x7f12e0107750 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12d5ffb640 1 -- 172.21.15.27:0/2430237075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12d8005020 con 0x7f12e0107750 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12d5ffb640 1 -- 172.21.15.27:0/2430237075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d80116b0 con 0x7f12e0107750 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e01a67e0 con 0x7f12e0107750 2024-04-25T01:25:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.936+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e01a6cb0 con 0x7f12e0107750 2024-04-25T01:25:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.937+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e00689b0 con 0x7f12e0107750 2024-04-25T01:25:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.938+0000 7f12d5ffb640 1 -- 172.21.15.27:0/2430237075 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f12d8005560 con 0x7f12e0107750 2024-04-25T01:25:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.939+0000 7f12d5ffb640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12c80a2e20 0x7f12c80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.939+0000 7f12e4a02640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12c80a2e20 0x7f12c80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.939+0000 7f12d5ffb640 1 -- 172.21.15.27:0/2430237075 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f12d80cfd10 con 0x7f12e0107750 2024-04-25T01:25:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.939+0000 7f12e4a02640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12c80a2e20 0x7f12c80a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12dc005fd0 tx=0x7f12dc00b5d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:30.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:30.943+0000 7f12d5ffb640 1 -- 172.21.15.27:0/2430237075 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f12d8016030 con 0x7f12e0107750 2024-04-25T01:25:31.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.133+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12e0109bf0 con 0x7f12e0107750 2024-04-25T01:25:31.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.135+0000 7f12d5ffb640 1 -- 172.21.15.27:0/2430237075 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25170 (secure 0 0 0) 0x7f12d809c350 con 0x7f12e0107750 2024-04-25T01:25:31.115 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:31.116 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:25:24.976745+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2238094164":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/3292898694":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/4288305036":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6800/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2504114047":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.136+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12c80a2e20 msgr2=0x7f12c80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.136+0000 7f12e6c66640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12c80a2e20 0x7f12c80a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12dc005fd0 tx=0x7f12dc00b5d0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.136+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e0107750 msgr2=0x7f12e00744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.136+0000 7f12e6c66640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e0107750 0x7f12e00744c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f12d800eaa0 tx=0x7f12d800ef60 comp rx=0 tx=0).stop 2024-04-25T01:25:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.137+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 shutdown_connections 2024-04-25T01:25:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.137+0000 7f12e6c66640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12c80a2e20 0x7f12c80a52e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.137+0000 7f12e6c66640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e0107750 0x7f12e00744c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.137+0000 7f12e6c66640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12e0102160 0x7f12e0073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.137+0000 7f12e6c66640 1 --2- 172.21.15.27:0/2430237075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01017c0 0x7f12e0073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.137+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 >> 172.21.15.27:0/2430237075 conn(0x7f12e00788d0 msgr2=0x7f12e0100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:31.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.137+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 shutdown_connections 2024-04-25T01:25:31.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.137+0000 7f12e6c66640 1 -- 172.21.15.27:0/2430237075 wait complete. 2024-04-25T01:25:31.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:31 smithi027 ceph-mon[29197]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:31.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:31 smithi027 ceph-mon[29197]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:31.128 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:25:31.129 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:25:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/949040422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381029c0 msgr2=0x7fcc38102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/949040422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381029c0 0x7fcc38102da0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009900 tx=0x7fcc2c02a510 comp rx=0 tx=0).stop 2024-04-25T01:25:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/949040422 shutdown_connections 2024-04-25T01:25:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/949040422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3810bd00 0x7fcc3810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/949040422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc381032e0 0x7fcc3810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/949040422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381029c0 0x7fcc38102da0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/949040422 >> 172.21.15.27:0/949040422 conn(0x7fcc38077740 msgr2=0x7fcc38077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/949040422 shutdown_connections 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/949040422 wait complete. 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.254+0000 7fcc3d2cd640 1 Processor -- start 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc3d2cd640 1 -- start start 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc3d2cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc381029c0 0x7fcc38198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc3d2cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381032e0 0x7fcc38198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc3d2cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3810bd00 0x7fcc3819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc3d2cd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcc38111b00 con 0x7fcc381032e0 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc3d2cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcc38111980 con 0x7fcc381029c0 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc3d2cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc38111c80 con 0x7fcc3810bd00 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381032e0 0x7fcc38198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381032e0 0x7fcc38198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50434/0 (socket says 172.21.15.27:50434) 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 -- 172.21.15.27:0/2712155445 learned_addr learned my addr 172.21.15.27:0/2712155445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 -- 172.21.15.27:0/2712155445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3810bd00 msgr2=0x7fcc3819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc377fe640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3810bd00 0x7fcc3819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc36ffd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc381029c0 0x7fcc38198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3810bd00 0x7fcc3819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 -- 172.21.15.27:0/2712155445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc381029c0 msgr2=0x7fcc38198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc381029c0 0x7fcc38198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 -- 172.21.15.27:0/2712155445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3819cff0 con 0x7fcc381032e0 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc367fc640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381032e0 0x7fcc38198540 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800e9f0 tx=0x7fcc2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc1ffff640 1 -- 172.21.15.27:0/2712155445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800cd80 con 0x7fcc381032e0 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3819d2e0 con 0x7fcc381032e0 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc1ffff640 1 -- 172.21.15.27:0/2712155445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc28010910 con 0x7fcc381032e0 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.255+0000 7fcc1ffff640 1 -- 172.21.15.27:0/2712155445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc280188f0 con 0x7fcc381032e0 2024-04-25T01:25:31.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.256+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc381a4b70 con 0x7fcc381032e0 2024-04-25T01:25:31.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.256+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc38068560 con 0x7fcc381032e0 2024-04-25T01:25:31.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.257+0000 7fcc1ffff640 1 -- 172.21.15.27:0/2712155445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7fcc28010ac0 con 0x7fcc381032e0 2024-04-25T01:25:31.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.258+0000 7fcc1ffff640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcc200a2e20 0x7fcc200a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.258+0000 7fcc36ffd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcc200a2e20 0x7fcc200a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.258+0000 7fcc1ffff640 1 -- 172.21.15.27:0/2712155445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fcc280cff60 con 0x7fcc381032e0 2024-04-25T01:25:31.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.259+0000 7fcc36ffd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcc200a2e20 0x7fcc200a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0096c0 tx=0x7fcc2c000bc0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.262+0000 7fcc1ffff640 1 -- 172.21.15.27:0/2712155445 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc2809c5a0 con 0x7fcc381032e0 2024-04-25T01:25:31.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:31 smithi153 ceph-mon[32621]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:31.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:31 smithi153 ceph-mon[32621]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:31.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:31 smithi139 ceph-mon[32760]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:31.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:31 smithi139 ceph-mon[32760]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.446+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc38106e80 con 0x7fcc381032e0 2024-04-25T01:25:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.447+0000 7fcc1ffff640 1 -- 172.21.15.27:0/2712155445 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fcc2809bea0 con 0x7fcc381032e0 2024-04-25T01:25:31.427 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:25:31.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcc200a2e20 msgr2=0x7fcc200a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcc200a2e20 0x7fcc200a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0096c0 tx=0x7fcc2c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381032e0 msgr2=0x7fcc38198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381032e0 0x7fcc38198540 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800e9f0 tx=0x7fcc2800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 shutdown_connections 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3810bd00 0x7fcc3819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcc200a2e20 0x7fcc200a52e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc381032e0 0x7fcc38198540 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 --2- 172.21.15.27:0/2712155445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcc381029c0 0x7fcc38198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 >> 172.21.15.27:0/2712155445 conn(0x7fcc38077740 msgr2=0x7fcc380fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 shutdown_connections 2024-04-25T01:25:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.449+0000 7fcc3d2cd640 1 -- 172.21.15.27:0/2712155445 wait complete. 2024-04-25T01:25:31.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 -- 172.21.15.27:0/1497824654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd41077f0 msgr2=0x7fecd4107bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/1497824654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd41077f0 0x7fecd4107bd0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009900 tx=0x7fecd002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 -- 172.21.15.27:0/1497824654 shutdown_connections 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/1497824654 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fecd406a660 0x7fecd410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/1497824654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd4069cc0 0x7fecd406a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/1497824654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd41077f0 0x7fecd4107bd0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 -- 172.21.15.27:0/1497824654 >> 172.21.15.27:0/1497824654 conn(0x7fecd4077410 msgr2=0x7fecd4077820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 -- 172.21.15.27:0/1497824654 shutdown_connections 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.565+0000 7fecd9fa0640 1 -- 172.21.15.27:0/1497824654 wait complete. 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd9fa0640 1 Processor -- start 2024-04-25T01:25:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd9fa0640 1 -- start start 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd9fa0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd4069cc0 0x7fecd4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd9fa0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd406a660 0x7fecd4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd9fa0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fecd41077f0 0x7fecd419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd9fa0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fecd4111b50 con 0x7fecd406a660 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd9fa0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fecd41119d0 con 0x7fecd41077f0 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd9fa0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fecd4111cd0 con 0x7fecd4069cc0 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7feccbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd406a660 0x7fecd4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7feccbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd406a660 0x7fecd4198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50450/0 (socket says 172.21.15.27:50450) 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7feccbfff640 1 -- 172.21.15.27:0/3792721379 learned_addr learned my addr 172.21.15.27:0/3792721379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.566+0000 7fecd8f9e640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd4069cc0 0x7fecd4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7feccbfff640 1 -- 172.21.15.27:0/3792721379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd4069cc0 msgr2=0x7fecd4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7feccbfff640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd4069cc0 0x7fecd4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7feccbfff640 1 -- 172.21.15.27:0/3792721379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fecd41077f0 msgr2=0x7fecd419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7feccbfff640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fecd41077f0 0x7fecd419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7feccbfff640 1 -- 172.21.15.27:0/3792721379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd419cfc0 con 0x7fecd406a660 2024-04-25T01:25:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7fecd8f9e640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd4069cc0 0x7fecd4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7feccbfff640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd406a660 0x7fecd4198510 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fecc400b4a0 tx=0x7fecc400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7fecc9ffb640 1 -- 172.21.15.27:0/3792721379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4014070 con 0x7fecd406a660 2024-04-25T01:25:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7fecc9ffb640 1 -- 172.21.15.27:0/3792721379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc4010af0 con 0x7fecd406a660 2024-04-25T01:25:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd419d2b0 con 0x7fecd406a660 2024-04-25T01:25:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7fecc9ffb640 1 -- 172.21.15.27:0/3792721379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4017d10 con 0x7fecd406a660 2024-04-25T01:25:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.567+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd41a4b40 con 0x7fecd406a660 2024-04-25T01:25:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.568+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd4102d30 con 0x7fecd406a660 2024-04-25T01:25:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.569+0000 7fecc9ffb640 1 -- 172.21.15.27:0/3792721379 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7fecc4017070 con 0x7fecd406a660 2024-04-25T01:25:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.570+0000 7fecc9ffb640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fecbc0a2e20 0x7fecbc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.570+0000 7fecd8f9e640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fecbc0a2e20 0x7fecbc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.570+0000 7fecc9ffb640 1 -- 172.21.15.27:0/3792721379 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fecc40cee00 con 0x7fecd406a660 2024-04-25T01:25:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.571+0000 7fecd8f9e640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fecbc0a2e20 0x7fecbc0a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fecd0005ab0 tx=0x7fecd0005a20 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:31.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.575+0000 7fecc9ffb640 1 -- 172.21.15.27:0/3792721379 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fecc409b550 con 0x7fecd406a660 2024-04-25T01:25:31.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.759+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fecd4109c50 con 0x7fecd406a660 2024-04-25T01:25:31.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.759+0000 7fecc9ffb640 1 -- 172.21.15.27:0/3792721379 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fecc409ae50 con 0x7fecd406a660 2024-04-25T01:25:31.740 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fecbc0a2e20 msgr2=0x7fecbc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fecbc0a2e20 0x7fecbc0a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fecd0005ab0 tx=0x7fecd0005a20 comp rx=0 tx=0).stop 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd406a660 msgr2=0x7fecd4198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd406a660 0x7fecd4198510 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fecc400b4a0 tx=0x7fecc400b960 comp rx=0 tx=0).stop 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 shutdown_connections 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fecd41077f0 0x7fecd419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fecbc0a2e20 0x7fecbc0a52e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecd406a660 0x7fecd4198510 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 --2- 172.21.15.27:0/3792721379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd4069cc0 0x7fecd4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 >> 172.21.15.27:0/3792721379 conn(0x7fecd4077410 msgr2=0x7fecd40ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:31.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 shutdown_connections 2024-04-25T01:25:31.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.761+0000 7fecd9fa0640 1 -- 172.21.15.27:0/3792721379 wait complete. 2024-04-25T01:25:31.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:25:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 -- 172.21.15.27:0/3434400167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee806a780 msgr2=0x7feee810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 --2- 172.21.15.27:0/3434400167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee806a780 0x7feee810e5f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7feee0009cd0 tx=0x7feee002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 -- 172.21.15.27:0/3434400167 shutdown_connections 2024-04-25T01:25:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 --2- 172.21.15.27:0/3434400167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee806a780 0x7feee810e5f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 --2- 172.21.15.27:0/3434400167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feee8069dc0 0x7feee806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 --2- 172.21.15.27:0/3434400167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee8107730 0x7feee8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 -- 172.21.15.27:0/3434400167 >> 172.21.15.27:0/3434400167 conn(0x7feee80776a0 msgr2=0x7feee8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 -- 172.21.15.27:0/3434400167 shutdown_connections 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.877+0000 7feeee088640 1 -- 172.21.15.27:0/3434400167 wait complete. 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.878+0000 7feeee088640 1 Processor -- start 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.878+0000 7feeee088640 1 -- start start 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.878+0000 7feeee088640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feee8069dc0 0x7feee8197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.878+0000 7feeee088640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee806a780 0x7feee81984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.878+0000 7feeee088640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee8107730 0x7feee819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.878+0000 7feeee088640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feee8111b90 con 0x7feee8107730 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.878+0000 7feeee088640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feee8111a10 con 0x7feee8069dc0 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.878+0000 7feeee088640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feee8111d10 con 0x7feee806a780 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee8107730 0x7feee819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee8107730 0x7feee819c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50468/0 (socket says 172.21.15.27:50468) 2024-04-25T01:25:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 -- 172.21.15.27:0/4033714577 learned_addr learned my addr 172.21.15.27:0/4033714577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee6ffd640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee806a780 0x7feee81984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee77fe640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feee8069dc0 0x7feee8197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 -- 172.21.15.27:0/4033714577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee806a780 msgr2=0x7feee81984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee806a780 0x7feee81984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 -- 172.21.15.27:0/4033714577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feee8069dc0 msgr2=0x7feee8197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feee8069dc0 0x7feee8197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 -- 172.21.15.27:0/4033714577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee819cf60 con 0x7feee8107730 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee7fff640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee8107730 0x7feee819c830 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feee0033040 tx=0x7feee0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee4ff9640 1 -- 172.21.15.27:0/4033714577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0004040 con 0x7feee8107730 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee4ff9640 1 -- 172.21.15.27:0/4033714577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feee00046c0 con 0x7feee8107730 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee819d1f0 con 0x7feee8107730 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feee4ff9640 1 -- 172.21.15.27:0/4033714577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0046480 con 0x7feee8107730 2024-04-25T01:25:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.879+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee81a4ae0 con 0x7feee8107730 2024-04-25T01:25:31.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.880+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee80689b0 con 0x7feee8107730 2024-04-25T01:25:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.881+0000 7feee4ff9640 1 -- 172.21.15.27:0/4033714577 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7feee00305a0 con 0x7feee8107730 2024-04-25T01:25:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.882+0000 7feee4ff9640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feed00a2e20 0x7feed00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.882+0000 7feee77fe640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feed00a2e20 0x7feed00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.882+0000 7feee4ff9640 1 -- 172.21.15.27:0/4033714577 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7feee002e030 con 0x7feee8107730 2024-04-25T01:25:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.882+0000 7feee77fe640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feed00a2e20 0x7feed00a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feed8005fd0 tx=0x7feed8005850 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:31.886+0000 7feee4ff9640 1 -- 172.21.15.27:0/4033714577 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feee004b060 con 0x7feee8107730 2024-04-25T01:25:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.070+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feee802d170 con 0x7feee8107730 2024-04-25T01:25:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.070+0000 7feee4ff9640 1 -- 172.21.15.27:0/4033714577 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7feee00bd880 con 0x7feee8107730 2024-04-25T01:25:32.051 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feed00a2e20 msgr2=0x7feed00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feed00a2e20 0x7feed00a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feed8005fd0 tx=0x7feed8005850 comp rx=0 tx=0).stop 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee8107730 msgr2=0x7feee819c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee8107730 0x7feee819c830 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feee0033040 tx=0x7feee0003970 comp rx=0 tx=0).stop 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 shutdown_connections 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feed00a2e20 0x7feed00a52e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee8107730 0x7feee819c830 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee806a780 0x7feee81984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 --2- 172.21.15.27:0/4033714577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feee8069dc0 0x7feee8197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 >> 172.21.15.27:0/4033714577 conn(0x7feee80776a0 msgr2=0x7feee80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 shutdown_connections 2024-04-25T01:25:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.072+0000 7feeee088640 1 -- 172.21.15.27:0/4033714577 wait complete. 2024-04-25T01:25:32.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:32 smithi027 ceph-mon[29197]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:32.062 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:32 smithi027 ceph-mon[29197]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s rd, 205 B/s wr, 1 op/s 2024-04-25T01:25:32.063 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:32 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2430237075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:32.063 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:32 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2712155445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:32.063 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:32 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3792721379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:32.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 -- 172.21.15.27:0/3539980490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa44069dc0 msgr2=0x7ffa4406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 --2- 172.21.15.27:0/3539980490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa44069dc0 0x7ffa4406a240 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009930 tx=0x7ffa2c02a420 comp rx=0 tx=0).stop 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 -- 172.21.15.27:0/3539980490 shutdown_connections 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 --2- 172.21.15.27:0/3539980490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa4406a780 0x7ffa4410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 --2- 172.21.15.27:0/3539980490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa44069dc0 0x7ffa4406a240 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 --2- 172.21.15.27:0/3539980490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa44107730 0x7ffa44107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 -- 172.21.15.27:0/3539980490 >> 172.21.15.27:0/3539980490 conn(0x7ffa440776a0 msgr2=0x7ffa44077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 -- 172.21.15.27:0/3539980490 shutdown_connections 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.188+0000 7ffa488e1640 1 -- 172.21.15.27:0/3539980490 wait complete. 2024-04-25T01:25:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa488e1640 1 Processor -- start 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa488e1640 1 -- start start 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa488e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa44069dc0 0x7ffa44100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa488e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa4406a780 0x7ffa44100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa488e1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa44107730 0x7ffa441010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa488e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffa44111b40 con 0x7ffa4406a780 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa488e1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441119c0 con 0x7ffa44107730 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa488e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa44111cc0 con 0x7ffa44069dc0 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa4406a780 0x7ffa44100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa4406a780 0x7ffa44100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50494/0 (socket says 172.21.15.27:50494) 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.189+0000 7ffa3bfff640 1 -- 172.21.15.27:0/904830094 learned_addr learned my addr 172.21.15.27:0/904830094 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa4259c640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa44069dc0 0x7ffa44100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa42d9d640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa44107730 0x7ffa441010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3bfff640 1 -- 172.21.15.27:0/904830094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa44069dc0 msgr2=0x7ffa44100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3bfff640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa44069dc0 0x7ffa44100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3bfff640 1 -- 172.21.15.27:0/904830094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa44107730 msgr2=0x7ffa441010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3bfff640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa44107730 0x7ffa441010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3bfff640 1 -- 172.21.15.27:0/904830094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa441a6500 con 0x7ffa4406a780 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3bfff640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa4406a780 0x7ffa44100bb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009230 tx=0x7ffa2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3b7fe640 1 -- 172.21.15.27:0/904830094 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c03f030 con 0x7ffa4406a780 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa441a6790 con 0x7ffa4406a780 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3b7fe640 1 -- 172.21.15.27:0/904830094 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa2c003ad0 con 0x7ffa4406a780 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa3b7fe640 1 -- 172.21.15.27:0/904830094 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c035070 con 0x7ffa4406a780 2024-04-25T01:25:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.190+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa441a6c80 con 0x7ffa4406a780 2024-04-25T01:25:32.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.191+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa440689b0 con 0x7ffa4406a780 2024-04-25T01:25:32.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.192+0000 7ffa3b7fe640 1 -- 172.21.15.27:0/904830094 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7ffa2c042030 con 0x7ffa4406a780 2024-04-25T01:25:32.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.193+0000 7ffa3b7fe640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffa300a2e50 0x7ffa300a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.193+0000 7ffa4259c640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffa300a2e50 0x7ffa300a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.193+0000 7ffa3b7fe640 1 -- 172.21.15.27:0/904830094 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffa2c0efe00 con 0x7ffa4406a780 2024-04-25T01:25:32.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.193+0000 7ffa4259c640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffa300a2e50 0x7ffa300a5310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005d00 tx=0x7ffa3400b410 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.197+0000 7ffa3b7fe640 1 -- 172.21.15.27:0/904830094 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffa2c0bbec0 con 0x7ffa4406a780 2024-04-25T01:25:32.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:32 smithi153 ceph-mon[32621]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:32.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:32 smithi153 ceph-mon[32621]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s rd, 205 B/s wr, 1 op/s 2024-04-25T01:25:32.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:32 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2430237075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:32.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:32 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2712155445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:32.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:32 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3792721379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:32.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:32 smithi139 ceph-mon[32760]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:32.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:32 smithi139 ceph-mon[32760]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s rd, 205 B/s wr, 1 op/s 2024-04-25T01:25:32.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:32 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2430237075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:32.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:32 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2712155445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:32.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:32 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3792721379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:25:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.380+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ffa44109c90 con 0x7ffa4406a780 2024-04-25T01:25:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.381+0000 7ffa3b7fe640 1 -- 172.21.15.27:0/904830094 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7ffa2c0c01c0 con 0x7ffa4406a780 2024-04-25T01:25:32.361 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:25:32.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffa300a2e50 msgr2=0x7ffa300a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffa300a2e50 0x7ffa300a5310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005d00 tx=0x7ffa3400b410 comp rx=0 tx=0).stop 2024-04-25T01:25:32.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa4406a780 msgr2=0x7ffa44100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa4406a780 0x7ffa44100bb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009230 tx=0x7ffa2c004270 comp rx=0 tx=0).stop 2024-04-25T01:25:32.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 shutdown_connections 2024-04-25T01:25:32.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffa44107730 0x7ffa441010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffa300a2e50 0x7ffa300a5310 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa4406a780 0x7ffa44100bb0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 --2- 172.21.15.27:0/904830094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa44069dc0 0x7ffa44100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 >> 172.21.15.27:0/904830094 conn(0x7ffa440776a0 msgr2=0x7ffa440fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:32.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 shutdown_connections 2024-04-25T01:25:32.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.383+0000 7ffa488e1640 1 -- 172.21.15.27:0/904830094 wait complete. 2024-04-25T01:25:32.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:32.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 -- 172.21.15.27:0/975515033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 msgr2=0x7ffb64100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 --2- 172.21.15.27:0/975515033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 0x7ffb64100b10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009930 tx=0x7ffb5002a420 comp rx=0 tx=0).stop 2024-04-25T01:25:32.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 -- 172.21.15.27:0/975515033 shutdown_connections 2024-04-25T01:25:32.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 --2- 172.21.15.27:0/975515033 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb64101050 0x7ffb6410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 --2- 172.21.15.27:0/975515033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 0x7ffb64100b10 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 --2- 172.21.15.27:0/975515033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb64106650 0x7ffb64106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 -- 172.21.15.27:0/975515033 >> 172.21.15.27:0/975515033 conn(0x7ffb640777a0 msgr2=0x7ffb64077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 -- 172.21.15.27:0/975515033 shutdown_connections 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.495+0000 7ffb6b012640 1 -- 172.21.15.27:0/975515033 wait complete. 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb6b012640 1 Processor -- start 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb6b012640 1 -- start start 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb6b012640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 0x7ffb64198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb6b012640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb64101050 0x7ffb64199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb6b012640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb64106650 0x7ffb641a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb6b012640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64110a20 con 0x7ffb64100690 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb6b012640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffb641108a0 con 0x7ffb64106650 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb6b012640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64110ba0 con 0x7ffb64101050 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb68dae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 0x7ffb64198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb68dae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 0x7ffb64198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50504/0 (socket says 172.21.15.27:50504) 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb68dae640 1 -- 172.21.15.27:0/654666469 learned_addr learned my addr 172.21.15.27:0/654666469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:32.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb63fff640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb64101050 0x7ffb64199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb695af640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb64106650 0x7ffb641a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb68dae640 1 -- 172.21.15.27:0/654666469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb64101050 msgr2=0x7ffb64199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb68dae640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb64101050 0x7ffb64199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb68dae640 1 -- 172.21.15.27:0/654666469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb64106650 msgr2=0x7ffb641a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb68dae640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb64106650 0x7ffb641a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb68dae640 1 -- 172.21.15.27:0/654666469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb641a44b0 con 0x7ffb64100690 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.496+0000 7ffb695af640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb64106650 0x7ffb641a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.497+0000 7ffb68dae640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 0x7ffb64198ef0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800d950 tx=0x7ffb5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.497+0000 7ffb61ffb640 1 -- 172.21.15.27:0/654666469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58017030 con 0x7ffb64100690 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.497+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb641a4680 con 0x7ffb64100690 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.497+0000 7ffb61ffb640 1 -- 172.21.15.27:0/654666469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb58004020 con 0x7ffb64100690 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.497+0000 7ffb61ffb640 1 -- 172.21.15.27:0/654666469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb580047a0 con 0x7ffb64100690 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.497+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb641a4ba0 con 0x7ffb64100690 2024-04-25T01:25:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.498+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb64101d80 con 0x7ffb64100690 2024-04-25T01:25:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.499+0000 7ffb61ffb640 1 -- 172.21.15.27:0/654666469 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7ffb580041d0 con 0x7ffb64100690 2024-04-25T01:25:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.500+0000 7ffb61ffb640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffb540a2e20 0x7ffb540a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.500+0000 7ffb61ffb640 1 -- 172.21.15.27:0/654666469 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffb580cf3d0 con 0x7ffb64100690 2024-04-25T01:25:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.500+0000 7ffb63fff640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffb540a2e20 0x7ffb540a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.500+0000 7ffb63fff640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffb540a2e20 0x7ffb540a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009900 tx=0x7ffb500058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:32.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.504+0000 7ffb61ffb640 1 -- 172.21.15.27:0/654666469 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffb5809ba10 con 0x7ffb64100690 2024-04-25T01:25:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.727+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb640689b0 con 0x7ffb64100690 2024-04-25T01:25:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.728+0000 7ffb61ffb640 1 -- 172.21.15.27:0/654666469 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7ffb5809b310 con 0x7ffb64100690 2024-04-25T01:25:32.709 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:32.709 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:25:23.250708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24505,24517]},"id":1}]} 2024-04-25T01:25:32.709 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffb540a2e20 msgr2=0x7ffb540a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffb540a2e20 0x7ffb540a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009900 tx=0x7ffb500058d0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 msgr2=0x7ffb64198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 0x7ffb64198ef0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800d950 tx=0x7ffb5800de10 comp rx=0 tx=0).stop 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 shutdown_connections 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffb64106650 0x7ffb641a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb64101050 0x7ffb64199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffb540a2e20 0x7ffb540a52e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 --2- 172.21.15.27:0/654666469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb64100690 0x7ffb64198ef0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 >> 172.21.15.27:0/654666469 conn(0x7ffb640777a0 msgr2=0x7ffb64103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:32.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 shutdown_connections 2024-04-25T01:25:32.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.730+0000 7ffb6b012640 1 -- 172.21.15.27:0/654666469 wait complete. 2024-04-25T01:25:32.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:32.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 -- 172.21.15.27:0/3969789680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0106650 msgr2=0x7fd3e0106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 --2- 172.21.15.27:0/3969789680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0106650 0x7fd3e0106a30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009cd0 tx=0x7fd3d002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 -- 172.21.15.27:0/3969789680 shutdown_connections 2024-04-25T01:25:32.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 --2- 172.21.15.27:0/3969789680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3e0101050 0x7fd3e010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 --2- 172.21.15.27:0/3969789680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3e0100690 0x7fd3e0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 --2- 172.21.15.27:0/3969789680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0106650 0x7fd3e0106a30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 -- 172.21.15.27:0/3969789680 >> 172.21.15.27:0/3969789680 conn(0x7fd3e00777a0 msgr2=0x7fd3e0077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 -- 172.21.15.27:0/3969789680 shutdown_connections 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.843+0000 7fd3e579e640 1 -- 172.21.15.27:0/3969789680 wait complete. 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3e579e640 1 Processor -- start 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3e579e640 1 -- start start 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3e579e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3e0100690 0x7fd3e0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3e579e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0101050 0x7fd3e01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3e579e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3e0106650 0x7fd3e019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3e579e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e0110a60 con 0x7fd3e0101050 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3e579e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e01108e0 con 0x7fd3e0100690 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3e579e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e0110be0 con 0x7fd3e0106650 2024-04-25T01:25:32.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.844+0000 7fd3de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0101050 0x7fd3e01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0101050 0x7fd3e01985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50518/0 (socket says 172.21.15.27:50518) 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3de7fc640 1 -- 172.21.15.27:0/339464069 learned_addr learned my addr 172.21.15.27:0/339464069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3df7fe640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3e0106650 0x7fd3e019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3deffd640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3e0100690 0x7fd3e0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3de7fc640 1 -- 172.21.15.27:0/339464069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3e0106650 msgr2=0x7fd3e019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3de7fc640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3e0106650 0x7fd3e019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3de7fc640 1 -- 172.21.15.27:0/339464069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3e0100690 msgr2=0x7fd3e0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3de7fc640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3e0100690 0x7fd3e0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3de7fc640 1 -- 172.21.15.27:0/339464069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e019d000 con 0x7fd3e0101050 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3de7fc640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0101050 0x7fd3e01985a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd3d400e9f0 tx=0x7fd3d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3c7fff640 1 -- 172.21.15.27:0/339464069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d400cd80 con 0x7fd3e0101050 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3c7fff640 1 -- 172.21.15.27:0/339464069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d4010910 con 0x7fd3e0101050 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e019d2f0 con 0x7fd3e0101050 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3c7fff640 1 -- 172.21.15.27:0/339464069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4018990 con 0x7fd3e0101050 2024-04-25T01:25:32.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.845+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e01a4b80 con 0x7fd3e0101050 2024-04-25T01:25:32.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.846+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e00689b0 con 0x7fd3e0101050 2024-04-25T01:25:32.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.847+0000 7fd3c7fff640 1 -- 172.21.15.27:0/339464069 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7fd3d4010ac0 con 0x7fd3e0101050 2024-04-25T01:25:32.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.848+0000 7fd3c7fff640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3c80a2e20 0x7fd3c80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.848+0000 7fd3deffd640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3c80a2e20 0x7fd3c80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.848+0000 7fd3c7fff640 1 -- 172.21.15.27:0/339464069 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd3d40cfee0 con 0x7fd3e0101050 2024-04-25T01:25:32.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.849+0000 7fd3deffd640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3c80a2e20 0x7fd3c80a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0006010 tx=0x7fd3d00093f0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:32.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:32.853+0000 7fd3c7fff640 1 -- 172.21.15.27:0/339464069 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd3d409c520 con 0x7fd3e0101050 2024-04-25T01:25:33.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.042+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3e0109630 con 0x7fd3e0101050 2024-04-25T01:25:33.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.044+0000 7fd3c7fff640 1 -- 172.21.15.27:0/339464069 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25170 (secure 0 0 0) 0x7fd3d409be20 con 0x7fd3e0101050 2024-04-25T01:25:33.024 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:25:33.025 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:25:24.976745+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2238094164":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/3292898694":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/4288305036":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6800/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2504114047":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3c80a2e20 msgr2=0x7fd3c80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3c80a2e20 0x7fd3c80a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0006010 tx=0x7fd3d00093f0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0101050 msgr2=0x7fd3e01985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0101050 0x7fd3e01985a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd3d400e9f0 tx=0x7fd3d400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 shutdown_connections 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3e0106650 0x7fd3e019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3c80a2e20 0x7fd3c80a52e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e0101050 0x7fd3e01985a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 --2- 172.21.15.27:0/339464069 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3e0100690 0x7fd3e0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 >> 172.21.15.27:0/339464069 conn(0x7fd3e00777a0 msgr2=0x7fd3e006a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 shutdown_connections 2024-04-25T01:25:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.046+0000 7fd3e579e640 1 -- 172.21.15.27:0/339464069 wait complete. 2024-04-25T01:25:33.038 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T01:25:33.038 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:25:33.048 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:25:33.048 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:25:33.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T01:25:33.048 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T01:25:33.049 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:25:33.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:25:33.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:25:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 -- 172.21.15.27:0/1764171454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8100550 msgr2=0x7f53c8100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 --2- 172.21.15.27:0/1764171454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8100550 0x7f53c8100930 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009900 tx=0x7f53b802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 -- 172.21.15.27:0/1764171454 shutdown_connections 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 --2- 172.21.15.27:0/1764171454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c810e100 0x7f53c81104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 --2- 172.21.15.27:0/1764171454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8100e70 0x7f53c810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 --2- 172.21.15.27:0/1764171454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8100550 0x7f53c8100930 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 -- 172.21.15.27:0/1764171454 >> 172.21.15.27:0/1764171454 conn(0x7f53c80777c0 msgr2=0x7f53c8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 -- 172.21.15.27:0/1764171454 shutdown_connections 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.173+0000 7f53cea38640 1 -- 172.21.15.27:0/1764171454 wait complete. 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53cea38640 1 Processor -- start 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53cea38640 1 -- start start 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53cea38640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c8100550 0x7f53c819a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53cea38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8100e70 0x7f53c819a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53cea38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c810e100 0x7f53c819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53cea38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8113d30 con 0x7f53c810e100 2024-04-25T01:25:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53cea38640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8113bb0 con 0x7f53c8100550 2024-04-25T01:25:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53cea38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8113eb0 con 0x7f53c8100e70 2024-04-25T01:25:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 -- 172.21.15.27:0/3925269143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8670101790 msgr2=0x7f8670101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 --2- 172.21.15.27:0/3925269143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8670101790 0x7f8670101c10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f865c009930 tx=0x7f865c02a420 comp rx=0 tx=0).stop 2024-04-25T01:25:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 -- 172.21.15.27:0/3925269143 shutdown_connections 2024-04-25T01:25:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 --2- 172.21.15.27:0/3925269143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8670102150 0x7f867010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 --2- 172.21.15.27:0/3925269143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8670101790 0x7f8670101c10 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 --2- 172.21.15.27:0/3925269143 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86701077a0 0x7f8670107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 -- 172.21.15.27:0/3925269143 >> 172.21.15.27:0/3925269143 conn(0x7f86700fd2a0 msgr2=0x7f86700ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53ccfd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c810e100 0x7f53c819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 -- 172.21.15.27:0/3925269143 shutdown_connections 2024-04-25T01:25:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53ccfd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c810e100 0x7f53c819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50546/0 (socket says 172.21.15.27:50546) 2024-04-25T01:25:33.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53ccfd5640 1 -- 172.21.15.27:0/3872890767 learned_addr learned my addr 172.21.15.27:0/3872890767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f8677031640 1 -- 172.21.15.27:0/3925269143 wait complete. 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53c7fff640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c8100550 0x7f53c819a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.174+0000 7f53c77fe640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8100e70 0x7f53c819a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53ccfd5640 1 -- 172.21.15.27:0/3872890767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8100e70 msgr2=0x7f53c819a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53ccfd5640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8100e70 0x7f53c819a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53ccfd5640 1 -- 172.21.15.27:0/3872890767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c8100550 msgr2=0x7f53c819a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53ccfd5640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c8100550 0x7f53c819a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53ccfd5640 1 -- 172.21.15.27:0/3872890767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c819f2d0 con 0x7f53c810e100 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53ccfd5640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c810e100 0x7f53c819eba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f53c000eb40 tx=0x7f53c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53c57fa640 1 -- 172.21.15.27:0/3872890767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0004b20 con 0x7f53c810e100 2024-04-25T01:25:33.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f8677031640 1 Processor -- start 2024-04-25T01:25:33.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53c57fa640 1 -- 172.21.15.27:0/3872890767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c0004cc0 con 0x7f53c810e100 2024-04-25T01:25:33.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c819f5c0 con 0x7f53c810e100 2024-04-25T01:25:33.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53c57fa640 1 -- 172.21.15.27:0/3872890767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0010850 con 0x7f53c810e100 2024-04-25T01:25:33.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f8677031640 1 -- start start 2024-04-25T01:25:33.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c81a6e50 con 0x7f53c810e100 2024-04-25T01:25:33.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f8677031640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8670101790 0x7f8670198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f8677031640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8670102150 0x7f8670198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f8677031640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86701077a0 0x7f867019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f8677031640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8670111bb0 con 0x7f86701077a0 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f8677031640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8670111a30 con 0x7f8670101790 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f8677031640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8670111d30 con 0x7f8670102150 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f866ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8670102150 0x7f8670198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f866ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8670102150 0x7f8670198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:48802/0 (socket says 172.21.15.27:48802) 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.175+0000 7f86755ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86701077a0 0x7f867019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866ffff640 1 -- 172.21.15.27:0/3170363766 learned_addr learned my addr 172.21.15.27:0/3170363766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f8674dcd640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8670101790 0x7f8670198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866ffff640 1 -- 172.21.15.27:0/3170363766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8670101790 msgr2=0x7f8670198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866ffff640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8670101790 0x7f8670198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866ffff640 1 -- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86701077a0 msgr2=0x7f867019c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866ffff640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86701077a0 0x7f867019c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866ffff640 1 -- 172.21.15.27:0/3170363766 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867019d100 con 0x7f8670102150 2024-04-25T01:25:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f86755ce640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86701077a0 0x7f867019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866ffff640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8670102150 0x7f8670198650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f865c009230 tx=0x7f865c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866dffb640 1 -- 172.21.15.27:0/3170363766 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c03f030 con 0x7f8670102150 2024-04-25T01:25:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866dffb640 1 -- 172.21.15.27:0/3170363766 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865c004440 con 0x7f8670102150 2024-04-25T01:25:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f8677031640 1 -- 172.21.15.27:0/3170363766 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f867019d390 con 0x7f8670102150 2024-04-25T01:25:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c804f500 con 0x7f53c810e100 2024-04-25T01:25:33.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.176+0000 7f866dffb640 1 -- 172.21.15.27:0/3170363766 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c035070 con 0x7f8670102150 2024-04-25T01:25:33.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.177+0000 7f8677031640 1 -- 172.21.15.27:0/3170363766 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86701a4c80 con 0x7f8670102150 2024-04-25T01:25:33.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.177+0000 7f53c57fa640 1 -- 172.21.15.27:0/3872890767 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f53c0005020 con 0x7f53c810e100 2024-04-25T01:25:33.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.178+0000 7f53c57fa640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2f40 0x7f53b00a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.178+0000 7f53c7fff640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2f40 0x7f53b00a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f53c57fa640 1 -- 172.21.15.27:0/3872890767 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f53c00d01a0 con 0x7f53c810e100 2024-04-25T01:25:33.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f53c7fff640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2f40 0x7f53b00a5400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009230 tx=0x7f53b80058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f8679e1a740 1 Processor -- start 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f8679e1a740 1 -- start start 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f866dffb640 1 -- 172.21.15.27:0/3170363766 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f865c042030 con 0x7f8670102150 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e240cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 0x562d6e241710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e252810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f8679e1a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562d6e1a52c0 con 0x562d6e242980 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f8679e1a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562d6e14d910 con 0x562d6e241290 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.179+0000 7f8679e1a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562d6e1a1280 con 0x562d6e250420 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e240cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e240cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50574/0 (socket says 172.21.15.27:50574) 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 -- 172.21.15.27:0/1949404562 learned_addr learned my addr 172.21.15.27:0/1949404562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f8674dcd640 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 0x562d6e241710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f866ffff640 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e252810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 -- 172.21.15.27:0/1949404562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 msgr2=0x562d6e252810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e252810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 -- 172.21.15.27:0/1949404562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 msgr2=0x562d6e241710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 0x562d6e241710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f866dffb640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2ea0 0x7f86600a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 -- 172.21.15.27:0/1949404562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d6e15efb0 con 0x562d6e242980 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f8674dcd640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2ea0 0x7f86600a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f866dffb640 1 -- 172.21.15.27:0/3170363766 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f865c0efbf0 con 0x7f8670102150 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f86755ce640 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e240cc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f86680117e0 tx=0x7f8668011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b53dcff106aa96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.180+0000 7f8674dcd640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2ea0 0x7f86600a5360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x562d6e243ed0 tx=0x7f8664005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.183+0000 7f865affd640 1 -- 172.21.15.27:0/1949404562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8668011310 con 0x562d6e242980 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.183+0000 7f865affd640 1 -- 172.21.15.27:0/1949404562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8668003e50 con 0x562d6e242980 2024-04-25T01:25:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.183+0000 7f865affd640 1 -- 172.21.15.27:0/1949404562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8668004620 con 0x562d6e242980 2024-04-25T01:25:33.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.183+0000 7f53c57fa640 1 -- 172.21.15.27:0/3872890767 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f53c009c7e0 con 0x7f53c810e100 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 -- 172.21.15.27:0/1949404562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 msgr2=0x562d6e240cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e240cc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f86680117e0 tx=0x7f8668011ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 -- 172.21.15.27:0/1949404562 shutdown_connections 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e252810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 0x562d6e241710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1949404562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e240cc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 -- 172.21.15.27:0/1949404562 >> 172.21.15.27:0/1949404562 conn(0x562d6e209950 msgr2=0x562d6e23d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 -- 172.21.15.27:0/1949404562 shutdown_connections 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 -- 172.21.15.27:0/1949404562 wait complete. 2024-04-25T01:25:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.184+0000 7f8679e1a740 1 -- start start 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8679e1a740 1 Processor -- start 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e241290 0x562d6e24d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e242980 0x562d6e24d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e24ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8679e1a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562d6e2532b0 con 0x562d6e241290 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8679e1a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x562d6e160e90 con 0x562d6e242980 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8679e1a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562d6e253430 con 0x562d6e250420 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8679e1a740 10 client.0 _ll_register_callbacks cb 0x562d6e19c970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8674dcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e241290 0x562d6e24d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8674dcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e241290 0x562d6e24d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50584/0 (socket says 172.21.15.27:50584) 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8674dcd640 1 -- 172.21.15.27:0/1245547222 learned_addr learned my addr 172.21.15.27:0/1245547222 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f866ffff640 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e242980 0x562d6e24d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f86755ce640 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e24ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8674dcd640 1 -- 172.21.15.27:0/1245547222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 msgr2=0x562d6e24ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8674dcd640 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e24ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8674dcd640 1 -- 172.21.15.27:0/1245547222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e242980 msgr2=0x562d6e24d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8674dcd640 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e242980 0x562d6e24d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f8674dcd640 1 -- 172.21.15.27:0/1245547222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d6df817e0 con 0x562d6e241290 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.185+0000 7f86755ce640 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e24ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8674dcd640 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e241290 0x562d6e24d340 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8664004620 tx=0x7f8664004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae918de0f1b276d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f864ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f864ffff640 1 -- 172.21.15.27:0/1245547222 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866401f640 con 0x562d6e241290 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f864ffff640 1 -- 172.21.15.27:0/1245547222 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f866401f7e0 con 0x562d6e241290 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 1 -- 172.21.15.27:0/1245547222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8668003e50 con 0x562d6e241290 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f864ffff640 1 -- 172.21.15.27:0/1245547222 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8664032050 con 0x562d6e241290 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 10 client.15702 fetch_fsmap learned FSMap version 25 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 1 -- 172.21.15.27:0/1245547222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562d6dfe1f70 con 0x562d6e241290 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f864ffff640 1 -- 172.21.15.27:0/1245547222 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8668003e50 con 0x562d6e241290 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f864ffff640 1 -- 172.21.15.27:0/1245547222 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8664027020 con 0x562d6e241290 2024-04-25T01:25:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 10 client.15702 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 20 client.15702 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x562d6e260570 0x562d6e2629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 4 client.15702 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.186+0000 7f8679e1a740 1 -- 172.21.15.27:0/1245547222 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d6e1795a0 con 0x562d6e260570 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.187+0000 7f866ffff640 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x562d6e260570 0x562d6e2629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.187+0000 7f864ffff640 10 client.15702 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.187+0000 7f866ffff640 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x562d6e260570 0x562d6e2629e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3958e69111af9709 server_cookie=bfaf9239c096c494 in_seq=0 out_seq=0 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f864ffff640 1 -- 172.21.15.27:0/1245547222 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d6e1795a0 con 0x562d6e260570 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f864ffff640 10 client.15702 handle_command_reply: tid=0 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 client.15702 shutdown 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 -- 172.21.15.27:0/1245547222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e241290 msgr2=0x562d6e24d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e241290 0x562d6e24d340 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8664004620 tx=0x7f8664004730 comp rx=0 tx=0).stop 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 -- 172.21.15.27:0/1245547222 shutdown_connections 2024-04-25T01:25:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x562d6e260570 0x562d6e2629e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f864ffff640 0 client.15702 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e250420 0x562d6e24ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e242980 0x562d6e24d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1245547222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e241290 0x562d6e24d340 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 -- 172.21.15.27:0/1245547222 >> 172.21.15.27:0/1245547222 conn(0x562d6e209950 msgr2=0x562d6e2435f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 -- 172.21.15.27:0/1245547222 shutdown_connections 2024-04-25T01:25:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.188+0000 7f8679e1a740 1 -- 172.21.15.27:0/1245547222 wait complete. 2024-04-25T01:25:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.189+0000 7f8679e1a740 20 client.15702 trim_cache size 0 max 16384 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.206+0000 7f8679e1a740 1 Processor -- start 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.206+0000 7f8679e1a740 1 -- start start 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e241290 0x562d6e25c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e24c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e250420 0x562d6e23c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f8679e1a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f86701af330 con 0x562d6e242980 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f8679e1a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f86701acc00 con 0x562d6e250420 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f8679e1a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86701aaa50 con 0x562d6e241290 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e24c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e24c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50596/0 (socket says 172.21.15.27:50596) 2024-04-25T01:25:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 -- 172.21.15.27:0/1405224708 learned_addr learned my addr 172.21.15.27:0/1405224708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f8674dcd640 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e241290 0x562d6e25c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f86755ce640 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e250420 0x562d6e23c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 -- 172.21.15.27:0/1405224708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e241290 msgr2=0x562d6e25c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e241290 0x562d6e25c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 -- 172.21.15.27:0/1405224708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e250420 msgr2=0x562d6e23c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e250420 0x562d6e23c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 -- 172.21.15.27:0/1405224708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d6e23da00 con 0x562d6e242980 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f866ffff640 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e24c4a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f865c0ecf40 tx=0x7f865c0f2100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d83f4760e6acfd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f865a7fc640 1 -- 172.21.15.27:0/1405224708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c049040 con 0x562d6e242980 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f865a7fc640 1 -- 172.21.15.27:0/1405224708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865c0bffb0 con 0x562d6e242980 2024-04-25T01:25:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.207+0000 7f865a7fc640 1 -- 172.21.15.27:0/1405224708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c0c43e0 con 0x562d6e242980 2024-04-25T01:25:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 -- 172.21.15.27:0/1405224708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 msgr2=0x562d6e24c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e24c4a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f865c0ecf40 tx=0x7f865c0f2100 comp rx=0 tx=0).stop 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 -- 172.21.15.27:0/1405224708 shutdown_connections 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e250420 0x562d6e23c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e242980 0x562d6e24c4a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 --2- 172.21.15.27:0/1405224708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e241290 0x562d6e25c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 -- 172.21.15.27:0/1405224708 >> 172.21.15.27:0/1405224708 conn(0x562d6e209950 msgr2=0x562d6e23c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 -- 172.21.15.27:0/1405224708 shutdown_connections 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 -- 172.21.15.27:0/1405224708 wait complete. 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.208+0000 7f8679e1a740 1 -- start start 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8679e1a740 1 Processor -- start 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 0x562d6e2586c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e242980 0x562d6e258c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8679e1a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e250420 0x562d6e255690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8679e1a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f86701ab0c0 con 0x562d6e250420 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8679e1a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f86701ac530 con 0x562d6e241290 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8679e1a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86701ab790 con 0x562d6e242980 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8679e1a740 10 client.0 _ll_register_callbacks cb 0x7f86640044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f86755ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e250420 0x562d6e255690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f86755ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e250420 0x562d6e255690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50602/0 (socket says 172.21.15.27:50602) 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f86755ce640 1 -- 172.21.15.27:0/4247605661 learned_addr learned my addr 172.21.15.27:0/4247605661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f8674dcd640 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 0x562d6e2586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f866ffff640 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e242980 0x562d6e258c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f86755ce640 1 -- 172.21.15.27:0/4247605661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e242980 msgr2=0x562d6e258c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f86755ce640 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e242980 0x562d6e258c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f86755ce640 1 -- 172.21.15.27:0/4247605661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 msgr2=0x562d6e2586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f86755ce640 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 0x562d6e2586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.209+0000 7f86755ce640 1 -- 172.21.15.27:0/4247605661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d6de79170 con 0x562d6e250420 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f86755ce640 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e250420 0x562d6e255690 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8668014040 tx=0x7f8668003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ead46ab4941fa42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f864ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f864ffff640 1 -- 172.21.15.27:0/4247605661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866800fba0 con 0x562d6e250420 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f864ffff640 1 -- 172.21.15.27:0/4247605661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f866800fd40 con 0x562d6e250420 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 1 -- 172.21.15.27:0/4247605661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f865c0bffb0 con 0x562d6e250420 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f864ffff640 1 -- 172.21.15.27:0/4247605661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8668030050 con 0x562d6e250420 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 10 client.15708 fetch_fsmap learned FSMap version 25 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 1 -- 172.21.15.27:0/4247605661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562d6e2c7210 con 0x562d6e250420 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f864ffff640 1 -- 172.21.15.27:0/4247605661 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865c0bffb0 con 0x562d6e250420 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f864ffff640 1 -- 172.21.15.27:0/4247605661 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8668018020 con 0x562d6e250420 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 10 client.15708 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 20 client.15708 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x562d6e260570 0x562d6e2c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 4 client.15708 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:25:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.210+0000 7f8679e1a740 1 -- 172.21.15.27:0/4247605661 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562d6e1b2800 con 0x562d6e260570 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.211+0000 7f8674dcd640 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x562d6e260570 0x562d6e2c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.211+0000 7f8674dcd640 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x562d6e260570 0x562d6e2c97a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b6a5887d80b8d16 server_cookie=b77c55eb29eb21fa in_seq=0 out_seq=0 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.211+0000 7f864ffff640 10 client.15708 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.212+0000 7f864ffff640 1 -- 172.21.15.27:0/4247605661 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562d6e1b2800 con 0x562d6e260570 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.212+0000 7f864ffff640 10 client.15708 handle_command_reply: tid=0 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.212+0000 7f8679e1a740 1 client.15708 shutdown 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 -- 172.21.15.27:0/4247605661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e250420 msgr2=0x562d6e255690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e250420 0x562d6e255690 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8668014040 tx=0x7f8668003ba0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 -- 172.21.15.27:0/4247605661 shutdown_connections 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x562d6e260570 0x562d6e2c97a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562d6e250420 0x562d6e255690 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f864ffff640 0 client.15708 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562d6e242980 0x562d6e258c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 --2- 172.21.15.27:0/4247605661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x562d6e241290 0x562d6e2586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 -- 172.21.15.27:0/4247605661 >> 172.21.15.27:0/4247605661 conn(0x562d6e209950 msgr2=0x562d6e24eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 -- 172.21.15.27:0/4247605661 shutdown_connections 2024-04-25T01:25:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 1 -- 172.21.15.27:0/4247605661 wait complete. 2024-04-25T01:25:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8679e1a740 20 client.15708 trim_cache size 0 max 16384 2024-04-25T01:25:33.198 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:25:33.198 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:25:33.198 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "26f9a104-d95f-4eb2-9faf-2e2e693ce76d", 2024-04-25T01:25:33.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:25:33.198 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:25:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8677031640 1 -- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2ea0 msgr2=0x7f86600a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8677031640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2ea0 0x7f86600a5360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x562d6e243ed0 tx=0x7f8664005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8677031640 1 -- 172.21.15.27:0/3170363766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8670102150 msgr2=0x7f8670198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.213+0000 7f8677031640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8670102150 0x7f8670198650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f865c009230 tx=0x7f865c004040 comp rx=0 tx=0).stop 2024-04-25T01:25:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.214+0000 7f8677031640 1 -- 172.21.15.27:0/3170363766 shutdown_connections 2024-04-25T01:25:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.214+0000 7f8677031640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2ea0 0x7f86600a5360 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.214+0000 7f8677031640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86701077a0 0x7f867019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.214+0000 7f8677031640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8670102150 0x7f8670198650 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.214+0000 7f8677031640 1 --2- 172.21.15.27:0/3170363766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8670101790 0x7f8670198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.214+0000 7f8677031640 1 -- 172.21.15.27:0/3170363766 >> 172.21.15.27:0/3170363766 conn(0x7f86700fd2a0 msgr2=0x7f86700fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.214+0000 7f8677031640 1 -- 172.21.15.27:0/3170363766 shutdown_connections 2024-04-25T01:25:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.214+0000 7f8677031640 1 -- 172.21.15.27:0/3170363766 wait complete. 2024-04-25T01:25:33.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:25:33.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:25:33.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:33 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4033714577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:33.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:33 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/904830094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:33.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:33 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/654666469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:33.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:33 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4033714577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:33.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:33 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/904830094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:33.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:33 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/654666469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 -- 172.21.15.27:0/1446706041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c100620 msgr2=0x7f2f2c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 --2- 172.21.15.27:0/1446706041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c100620 0x7f2f2c100aa0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009930 tx=0x7f2f2002a420 comp rx=0 tx=0).stop 2024-04-25T01:25:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 -- 172.21.15.27:0/1446706041 shutdown_connections 2024-04-25T01:25:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 --2- 172.21.15.27:0/1446706041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c100fe0 0x7f2f2c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 --2- 172.21.15.27:0/1446706041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c100620 0x7f2f2c100aa0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 --2- 172.21.15.27:0/1446706041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f2c106630 0x7f2f2c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 -- 172.21.15.27:0/1446706041 >> 172.21.15.27:0/1446706041 conn(0x7f2f2c0776a0 msgr2=0x7f2f2c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 -- 172.21.15.27:0/1446706041 shutdown_connections 2024-04-25T01:25:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 -- 172.21.15.27:0/1446706041 wait complete. 2024-04-25T01:25:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 Processor -- start 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.329+0000 7f2f31709640 1 -- start start 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f31709640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f2c100620 0x7f2f2c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f31709640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c100fe0 0x7f2f2c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f31709640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c106630 0x7f2f2c19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f31709640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c110a90 con 0x7f2f2c106630 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f31709640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c110910 con 0x7f2f2c100620 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f31709640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c110c10 con 0x7f2f2c100fe0 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c106630 0x7f2f2c19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c106630 0x7f2f2c19c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50620/0 (socket says 172.21.15.27:50620) 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 -- 172.21.15.27:0/3120893617 learned_addr learned my addr 172.21.15.27:0/3120893617 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2affd640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f2c100620 0x7f2f2c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c100fe0 0x7f2f2c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 -- 172.21.15.27:0/3120893617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c100fe0 msgr2=0x7f2f2c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c100fe0 0x7f2f2c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 -- 172.21.15.27:0/3120893617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f2c100620 msgr2=0x7f2f2c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f2c100620 0x7f2f2c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 -- 172.21.15.27:0/3120893617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c19d0a0 con 0x7f2f2c106630 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c100fe0 0x7f2f2c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2affd640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f2c100620 0x7f2f2c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f2b7fe640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c106630 0x7f2f2c19c970 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009fd0 tx=0x7f2f2400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f13fff640 1 -- 172.21.15.27:0/3120893617 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24017070 con 0x7f2f2c106630 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f31709640 1 -- 172.21.15.27:0/3120893617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2c19d390 con 0x7f2f2c106630 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f13fff640 1 -- 172.21.15.27:0/3120893617 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f24003e50 con 0x7f2f2c106630 2024-04-25T01:25:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.330+0000 7f2f13fff640 1 -- 172.21.15.27:0/3120893617 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f240106c0 con 0x7f2f2c106630 2024-04-25T01:25:33.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.331+0000 7f2f31709640 1 -- 172.21.15.27:0/3120893617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2c1a4c20 con 0x7f2f2c106630 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f13fff640 1 -- 172.21.15.27:0/3120893617 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f2f24003970 con 0x7f2f2c106630 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f3440d740 1 Processor -- start 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f3440d740 1 -- start start 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b310a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f3440d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56051b2756b0 con 0x56051b310f50 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f3440d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56051b21d880 con 0x56051b31b620 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f3440d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56051b271670 con 0x56051b310630 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f2affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f2affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50628/0 (socket says 172.21.15.27:50628) 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f2affd640 1 -- 172.21.15.27:0/2820083598 learned_addr learned my addr 172.21.15.27:0/2820083598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b310a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.333+0000 7f2f13fff640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2f140a2e20 0x7f2f140a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2affd640 1 -- 172.21.15.27:0/2820083598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 msgr2=0x56051b310a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2affd640 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b310a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2affd640 1 -- 172.21.15.27:0/2820083598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 msgr2=0x56051b31da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2affd640 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2affd640 1 -- 172.21.15.27:0/2820083598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56051b0537d0 con 0x56051b310f50 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2b7fe640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2f140a2e20 0x7f2f140a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f13fff640 1 -- 172.21.15.27:0/3120893617 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2f240cf940 con 0x7f2f2c106630 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f2affd640 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31b0e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00dc40 tx=0x7f2f1c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c857a068c07fee1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f117fa640 1 -- 172.21.15.27:0/2820083598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c005020 con 0x56051b310f50 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f117fa640 1 -- 172.21.15.27:0/2820083598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f1c0047a0 con 0x56051b310f50 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.334+0000 7f2f117fa640 1 -- 172.21.15.27:0/2820083598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c0116a0 con 0x56051b310f50 2024-04-25T01:25:33.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 -- 172.21.15.27:0/2820083598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 msgr2=0x56051b31b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31b0e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00dc40 tx=0x7f2f1c00f730 comp rx=0 tx=0).stop 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f2b7fe640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2f140a2e20 0x7f2f140a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f240ccb80 tx=0x7f2f2406a750 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 -- 172.21.15.27:0/2820083598 shutdown_connections 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31b0e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 --2- 172.21.15.27:0/2820083598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b310a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 -- 172.21.15.27:0/2820083598 >> 172.21.15.27:0/2820083598 conn(0x56051b30c950 msgr2=0x56051b30cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 -- 172.21.15.27:0/2820083598 shutdown_connections 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 -- 172.21.15.27:0/2820083598 wait complete. 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 -- start start 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.335+0000 7f2f3440d740 1 Processor -- start 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310630 0x56051b3185c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 0x56051b318b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b31b620 0x56051b319040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f3440d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56051b31e410 con 0x56051b310630 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f3440d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56051b230f50 con 0x56051b310f50 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f3440d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56051b31e590 con 0x56051b31b620 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f3440d740 10 client.0 _ll_register_callbacks cb 0x56051b26c000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310630 0x56051b3185c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310630 0x56051b3185c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50640/0 (socket says 172.21.15.27:50640) 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 -- 172.21.15.27:0/750843461 learned_addr learned my addr 172.21.15.27:0/750843461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b31b620 0x56051b319040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 -- 172.21.15.27:0/750843461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b31b620 msgr2=0x56051b319040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b31b620 0x56051b319040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2affd640 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 0x56051b318b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 -- 172.21.15.27:0/750843461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 msgr2=0x56051b318b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 0x56051b318b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 -- 172.21.15.27:0/750843461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56051af893f0 con 0x56051b310630 2024-04-25T01:25:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310630 0x56051b3185c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2f200057c0 tx=0x7f2f20009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229ee6ef521bb4e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f0eff5640 1 -- 172.21.15.27:0/750843461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20045030 con 0x56051b310630 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f3440d740 1 -- 172.21.15.27:0/750843461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f1c0047a0 con 0x56051b310630 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.336+0000 7f2f0eff5640 1 -- 172.21.15.27:0/750843461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2f20003bb0 con 0x56051b310630 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f0eff5640 1 -- 172.21.15.27:0/750843461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2f2004f050 con 0x56051b310630 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f3440d740 10 client.15720 fetch_fsmap learned FSMap version 25 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f3440d740 1 -- 172.21.15.27:0/750843461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56051af490f0 con 0x56051b310630 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f0eff5640 1 -- 172.21.15.27:0/750843461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f1c0047a0 con 0x56051b310630 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f0eff5640 1 -- 172.21.15.27:0/750843461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2f2003a050 con 0x56051b310630 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f3440d740 10 client.15720 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f3440d740 10 client.15720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f3440d740 20 client.15720 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f3440d740 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56051b32b840 0x56051b32dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f3440d740 4 client.15720 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:25:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f3440d740 1 -- 172.21.15.27:0/750843461 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56051b282bf0 con 0x56051b32b840 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.337+0000 7f2f2affd640 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56051b32b840 0x56051b32dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.338+0000 7f2f2affd640 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56051b32b840 0x56051b32dce0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd1e867969a66e83 server_cookie=5afacdfe23ea2afd in_seq=0 out_seq=0 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.338+0000 7f2f0eff5640 10 client.15720 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.338+0000 7f2f0eff5640 1 -- 172.21.15.27:0/750843461 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56051b282bf0 con 0x56051b32b840 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.338+0000 7f2f0eff5640 10 client.15720 handle_command_reply: tid=0 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.338+0000 7f2f3440d740 1 client.15720 shutdown 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.338+0000 7f2f3440d740 1 -- 172.21.15.27:0/750843461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310630 msgr2=0x56051b3185c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.338+0000 7f2f3440d740 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310630 0x56051b3185c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2f200057c0 tx=0x7f2f20009260 comp rx=0 tx=0).stop 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 1 -- 172.21.15.27:0/750843461 shutdown_connections 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56051b32b840 0x56051b32dce0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b31b620 0x56051b319040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 0x56051b318b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 1 --2- 172.21.15.27:0/750843461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310630 0x56051b3185c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f0eff5640 0 client.15720 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 1 -- 172.21.15.27:0/750843461 >> 172.21.15.27:0/750843461 conn(0x56051b30c950 msgr2=0x56051b312650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 1 -- 172.21.15.27:0/750843461 shutdown_connections 2024-04-25T01:25:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 1 -- 172.21.15.27:0/750843461 wait complete. 2024-04-25T01:25:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.339+0000 7f2f3440d740 20 client.15720 trim_cache size 0 max 16384 2024-04-25T01:25:33.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f3440d740 1 Processor -- start 2024-04-25T01:25:33.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f3440d740 1 -- start start 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b30c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 0x56051b327300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b31b620 0x56051b31a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f3440d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1b1520 con 0x56051b31b620 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f3440d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1aedf0 con 0x56051b310f50 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f3440d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1acc40 con 0x56051b310630 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f2a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b31b620 0x56051b31a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f2a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b31b620 0x56051b31a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50654/0 (socket says 172.21.15.27:50654) 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.356+0000 7f2f2a7fc640 1 -- 172.21.15.27:0/1526615297 learned_addr learned my addr 172.21.15.27:0/1526615297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b30c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f2affd640 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 0x56051b327300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f2a7fc640 1 -- 172.21.15.27:0/1526615297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 msgr2=0x56051b30c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b30c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f2a7fc640 1 -- 172.21.15.27:0/1526615297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 msgr2=0x56051b327300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 0x56051b327300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f2a7fc640 1 -- 172.21.15.27:0/1526615297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56051b2fe090 con 0x56051b31b620 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b31b620 0x56051b31a3f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f200098d0 tx=0x7f2f20005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce01689dc4f439d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f10ff9640 1 -- 172.21.15.27:0/1526615297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20045070 con 0x56051b31b620 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f10ff9640 1 -- 172.21.15.27:0/1526615297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f20046040 con 0x56051b31b620 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.357+0000 7f2f10ff9640 1 -- 172.21.15.27:0/1526615297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20035040 con 0x56051b31b620 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- 172.21.15.27:0/1526615297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b31b620 msgr2=0x56051b31a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b31b620 0x56051b31a3f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f200098d0 tx=0x7f2f20005fd0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- 172.21.15.27:0/1526615297 shutdown_connections 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b31b620 0x56051b31a3f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b310f50 0x56051b327300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 --2- 172.21.15.27:0/1526615297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b30c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- 172.21.15.27:0/1526615297 >> 172.21.15.27:0/1526615297 conn(0x56051b30c950 msgr2=0x56051b319ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- 172.21.15.27:0/1526615297 shutdown_connections 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- 172.21.15.27:0/1526615297 wait complete. 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- start start 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 Processor -- start 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b31f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1ad2b0 con 0x56051b310f50 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1ae720 con 0x56051b31b620 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1ad980 con 0x56051b310630 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f3440d740 10 client.0 _ll_register_callbacks cb 0x7f2f20009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f2affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.358+0000 7f2f2affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50658/0 (socket says 172.21.15.27:50658) 2024-04-25T01:25:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2affd640 1 -- 172.21.15.27:0/194852736 learned_addr learned my addr 172.21.15.27:0/194852736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b31f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2affd640 1 -- 172.21.15.27:0/194852736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 msgr2=0x56051b31f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2affd640 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b31f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2affd640 1 -- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 msgr2=0x56051b31fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2affd640 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2affd640 1 -- 172.21.15.27:0/194852736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56051b395330 con 0x56051b310f50 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f2affd640 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31f870 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c000c00 tx=0x7f2f1c00ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcfd06d95be0e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f0eff5640 1 -- 172.21.15.27:0/194852736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c005520 con 0x56051b310f50 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f0eff5640 1 -- 172.21.15.27:0/194852736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2f1c0035f0 con 0x56051b310f50 2024-04-25T01:25:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f3440d740 1 -- 172.21.15.27:0/194852736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f20046040 con 0x56051b310f50 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f0eff5640 1 -- 172.21.15.27:0/194852736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2f1c031050 con 0x56051b310f50 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f3440d740 10 client.15726 fetch_fsmap learned FSMap version 25 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f3440d740 1 -- 172.21.15.27:0/194852736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56051b3966a0 con 0x56051b310f50 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.359+0000 7f2f0eff5640 1 -- 172.21.15.27:0/194852736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f20046040 con 0x56051b310f50 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f0eff5640 1 -- 172.21.15.27:0/194852736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2f1c0283a0 con 0x56051b310f50 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f3440d740 10 client.15726 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f3440d740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f3440d740 20 client.15726 populate_metadata read hostname 'smithi027' 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f3440d740 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56051b396b90 0x56051b398f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f3440d740 4 client.15726 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f3440d740 1 -- 172.21.15.27:0/194852736 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56051b27cf40 con 0x56051b396b90 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56051b396b90 0x56051b398f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f0eff5640 10 client.15726 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.360+0000 7f2f2a7fc640 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56051b396b90 0x56051b398f80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d5e8f7ba0c2b018 server_cookie=181b1c5a5758dc0a in_seq=0 out_seq=0 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f0eff5640 1 -- 172.21.15.27:0/194852736 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56051b27cf40 con 0x56051b396b90 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f0eff5640 10 client.15726 handle_command_reply: tid=0 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 client.15726 shutdown 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 -- 172.21.15.27:0/194852736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 msgr2=0x56051b31f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31f870 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c000c00 tx=0x7f2f1c00ded0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 -- 172.21.15.27:0/194852736 shutdown_connections 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56051b396b90 0x56051b398f80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56051b31b620 0x56051b31fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051b310f50 0x56051b31f870 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f0eff5640 0 client.15726 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 --2- 172.21.15.27:0/194852736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56051b310630 0x56051b31f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 -- 172.21.15.27:0/194852736 >> 172.21.15.27:0/194852736 conn(0x56051b30c950 msgr2=0x56051b318e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 -- 172.21.15.27:0/194852736 shutdown_connections 2024-04-25T01:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 1 -- 172.21.15.27:0/194852736 wait complete. 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f3440d740 20 client.15726 trim_cache size 0 max 16384 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout: "26f9a104-d95f-4eb2-9faf-2e2e693ce76d": { 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "26f9a104-d95f-4eb2-9faf-2e2e693ce76d", 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:25:33.348 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:25:33.349 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:25:33.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f31709640 1 -- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2f140a2e20 msgr2=0x7f2f140a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f31709640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2f140a2e20 0x7f2f140a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f240ccb80 tx=0x7f2f2406a750 comp rx=0 tx=0).stop 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f31709640 1 -- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c106630 msgr2=0x7f2f2c19c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f31709640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c106630 0x7f2f2c19c970 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009fd0 tx=0x7f2f2400b260 comp rx=0 tx=0).stop 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f31709640 1 -- 172.21.15.27:0/3120893617 shutdown_connections 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f31709640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2f140a2e20 0x7f2f140a52e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.361+0000 7f2f31709640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f2c106630 0x7f2f2c19c970 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.362+0000 7f2f31709640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c100fe0 0x7f2f2c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.362+0000 7f2f31709640 1 --2- 172.21.15.27:0/3120893617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f2c100620 0x7f2f2c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.362+0000 7f2f31709640 1 -- 172.21.15.27:0/3120893617 >> 172.21.15.27:0/3120893617 conn(0x7f2f2c0776a0 msgr2=0x7f2f2c06a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.362+0000 7f2f31709640 1 -- 172.21.15.27:0/3120893617 shutdown_connections 2024-04-25T01:25:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.362+0000 7f2f31709640 1 -- 172.21.15.27:0/3120893617 wait complete. 2024-04-25T01:25:33.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:33 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4033714577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:25:33.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:33 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/904830094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:25:33.352 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:33 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/654666469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:33.352 INFO:tasks.cephfs.filesystem:scrub status for tag:26f9a104-d95f-4eb2-9faf-2e2e693ce76d - {'path': '/', 'tag': '26f9a104-d95f-4eb2-9faf-2e2e693ce76d', 'options': 'recursive,force'} 2024-04-25T01:25:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.397+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f53c81092f0 con 0x7f53b00a2f40 2024-04-25T01:25:33.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.461+0000 7f53c57fa640 1 -- 172.21.15.27:0/3872890767 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f53c81092f0 con 0x7f53b00a2f40 2024-04-25T01:25:33.442 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path / 2024-04-25T01:25:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2f40 msgr2=0x7f53b00a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2f40 0x7f53b00a5400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009230 tx=0x7f53b80058d0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c810e100 msgr2=0x7f53c819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c810e100 0x7f53c819eba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f53c000eb40 tx=0x7f53c000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 shutdown_connections 2024-04-25T01:25:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2f40 0x7f53b00a5400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c810e100 0x7f53c819eba0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8100e70 0x7f53c819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 --2- 172.21.15.27:0/3872890767 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c8100550 0x7f53c819a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 >> 172.21.15.27:0/3872890767 conn(0x7f53c80777c0 msgr2=0x7f53c810f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 shutdown_connections 2024-04-25T01:25:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.464+0000 7f53cea38640 1 -- 172.21.15.27:0/3872890767 wait complete. 2024-04-25T01:25:33.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T01:25:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 -- 172.21.15.27:0/2348368117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 msgr2=0x7f7fc010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/2348368117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 0x7f7fc010d4f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8009cd0 tx=0x7f7fb802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 -- 172.21.15.27:0/2348368117 shutdown_connections 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/2348368117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 0x7f7fc010d4f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/2348368117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fc0100670 0x7f7fc0100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/2348368117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc0106630 0x7f7fc0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 -- 172.21.15.27:0/2348368117 >> 172.21.15.27:0/2348368117 conn(0x7f7fc00777c0 msgr2=0x7f7fc0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 -- 172.21.15.27:0/2348368117 shutdown_connections 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 -- 172.21.15.27:0/2348368117 wait complete. 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 Processor -- start 2024-04-25T01:25:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.620+0000 7f7fc65f9640 1 -- start start 2024-04-25T01:25:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc65f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc0100670 0x7f7fc0198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc65f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 0x7f7fc01992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc65f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fc0106630 0x7f7fc01a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc65f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0110a90 con 0x7f7fc0101030 2024-04-25T01:25:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc65f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0110910 con 0x7f7fc0106630 2024-04-25T01:25:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc65f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0110c10 con 0x7f7fc0100670 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 0x7f7fc01992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 0x7f7fc01992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50672/0 (socket says 172.21.15.27:50672) 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 -- 172.21.15.27:0/835567057 learned_addr learned my addr 172.21.15.27:0/835567057 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbffff640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc0100670 0x7f7fc0198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc4b96640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fc0106630 0x7f7fc01a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 -- 172.21.15.27:0/835567057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc0100670 msgr2=0x7f7fc0198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc0100670 0x7f7fc0198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 -- 172.21.15.27:0/835567057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fc0106630 msgr2=0x7f7fc01a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fc0106630 0x7f7fc01a3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 -- 172.21.15.27:0/835567057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc01a4330 con 0x7f7fc0101030 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbffff640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc0100670 0x7f7fc0198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc4b96640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fc0106630 0x7f7fc01a3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbf7fe640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 0x7f7fc01992b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400d950 tx=0x7f7fb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/835567057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4017070 con 0x7f7fc0101030 2024-04-25T01:25:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/835567057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb4009c30 con 0x7f7fc0101030 2024-04-25T01:25:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.621+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc01a4560 con 0x7f7fc0101030 2024-04-25T01:25:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.622+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/835567057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400f6b0 con 0x7f7fc0101030 2024-04-25T01:25:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.622+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc01a4ad0 con 0x7f7fc0101030 2024-04-25T01:25:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.623+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc004f500 con 0x7f7fc0101030 2024-04-25T01:25:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.624+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/835567057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f7fb40109d0 con 0x7f7fc0101030 2024-04-25T01:25:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.624+0000 7f7fbd7fa640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fa80a2f40 0x7f7fa80a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.625+0000 7f7fbffff640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fa80a2f40 0x7f7fa80a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.625+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/835567057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7fb40cf6d0 con 0x7f7fc0101030 2024-04-25T01:25:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.625+0000 7f7fbffff640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fa80a2f40 0x7f7fa80a5400 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0005d70 tx=0x7f7fb0005d00 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.629+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/835567057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7fb4009de0 con 0x7f7fc0101030 2024-04-25T01:25:33.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.835+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f7fc0061b80 con 0x7f7fa80a2f40 2024-04-25T01:25:33.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.896+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/835567057 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f7fc0061b80 con 0x7f7fa80a2f40 2024-04-25T01:25:33.876 INFO:teuthology.orchestra.run.smithi027.stdout:Retention added to path / 2024-04-25T01:25:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fa80a2f40 msgr2=0x7f7fa80a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fa80a2f40 0x7f7fa80a5400 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0005d70 tx=0x7f7fb0005d00 comp rx=0 tx=0).stop 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 msgr2=0x7f7fc01992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 0x7f7fc01992b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400d950 tx=0x7f7fb400de10 comp rx=0 tx=0).stop 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 shutdown_connections 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fc0106630 0x7f7fc01a3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fa80a2f40 0x7f7fa80a5400 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc0101030 0x7f7fc01992b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 --2- 172.21.15.27:0/835567057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc0100670 0x7f7fc0198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 >> 172.21.15.27:0/835567057 conn(0x7f7fc00777c0 msgr2=0x7f7fc0102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 shutdown_connections 2024-04-25T01:25:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:33.898+0000 7f7fc65f9640 1 -- 172.21.15.27:0/835567057 wait complete. 2024-04-25T01:25:33.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.011+0000 7f91bb332640 1 -- 172.21.15.27:0/3085507524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4101030 msgr2=0x7f91b410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.011+0000 7f91bb332640 1 --2- 172.21.15.27:0/3085507524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4101030 0x7f91b410d4f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009cd0 tx=0x7f91ac02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.011+0000 7f91bb332640 1 -- 172.21.15.27:0/3085507524 shutdown_connections 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.011+0000 7f91bb332640 1 --2- 172.21.15.27:0/3085507524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4101030 0x7f91b410d4f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.011+0000 7f91bb332640 1 --2- 172.21.15.27:0/3085507524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91b4100670 0x7f91b4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.011+0000 7f91bb332640 1 --2- 172.21.15.27:0/3085507524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f91b4106630 0x7f91b4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.011+0000 7f91bb332640 1 -- 172.21.15.27:0/3085507524 >> 172.21.15.27:0/3085507524 conn(0x7f91b40777c0 msgr2=0x7f91b4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 -- 172.21.15.27:0/3085507524 shutdown_connections 2024-04-25T01:25:33.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 -- 172.21.15.27:0/3085507524 wait complete. 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 Processor -- start 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 -- start start 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4100670 0x7f91b4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91b4101030 0x7f91b4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f91b4106630 0x7f91b419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f91b4110a40 con 0x7f91b4100670 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f91b41108c0 con 0x7f91b4106630 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91bb332640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f91b4110bc0 con 0x7f91b4101030 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91b90ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4100670 0x7f91b4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.012+0000 7f91b90ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4100670 0x7f91b4197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50688/0 (socket says 172.21.15.27:50688) 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b90ce640 1 -- 172.21.15.27:0/1160507155 learned_addr learned my addr 172.21.15.27:0/1160507155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b98cf640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f91b4106630 0x7f91b419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b88cd640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91b4101030 0x7f91b4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b90ce640 1 -- 172.21.15.27:0/1160507155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91b4101030 msgr2=0x7f91b4198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b90ce640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91b4101030 0x7f91b4198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b90ce640 1 -- 172.21.15.27:0/1160507155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f91b4106630 msgr2=0x7f91b419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b90ce640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f91b4106630 0x7f91b419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b90ce640 1 -- 172.21.15.27:0/1160507155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b419cfb0 con 0x7f91b4100670 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91b90ce640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4100670 0x7f91b4197fc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f91b000d950 tx=0x7f91b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91aa7fc640 1 -- 172.21.15.27:0/1160507155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0017030 con 0x7f91b4100670 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b419d2a0 con 0x7f91b4100670 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91aa7fc640 1 -- 172.21.15.27:0/1160507155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91b0004020 con 0x7f91b4100670 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91aa7fc640 1 -- 172.21.15.27:0/1160507155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b00047a0 con 0x7f91b4100670 2024-04-25T01:25:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.013+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b41a4b30 con 0x7f91b4100670 2024-04-25T01:25:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.014+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b404f500 con 0x7f91b4100670 2024-04-25T01:25:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.016+0000 7f91aa7fc640 1 -- 172.21.15.27:0/1160507155 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7f91b00041d0 con 0x7f91b4100670 2024-04-25T01:25:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.016+0000 7f91aa7fc640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f91a00a2ec0 0x7f91a00a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.017+0000 7f91b88cd640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f91a00a2ec0 0x7f91a00a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.017+0000 7f91aa7fc640 1 -- 172.21.15.27:0/1160507155 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f91b00cf9e0 con 0x7f91b4100670 2024-04-25T01:25:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.017+0000 7f91b88cd640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f91a00a2ec0 0x7f91a00a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f91b4077ee0 tx=0x7f91a400b390 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:34.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.025+0000 7f91aa7fc640 1 -- 172.21.15.27:0/1160507155 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f91b009c020 con 0x7f91b4100670 2024-04-25T01:25:34.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:34 smithi027 ceph-mon[29197]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 732 B/s wr, 22 op/s 2024-04-25T01:25:34.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:34 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/339464069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:34.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:34 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:25:34.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:34 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:25:34.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:34 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:25:34.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:34 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:25:34.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:34 smithi027 ceph-mon[29197]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.229+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f91b4109c30 con 0x7f91a00a2ec0 2024-04-25T01:25:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.237+0000 7f91aa7fc640 1 -- 172.21.15.27:0/1160507155 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f91a4003890 con 0x7f91a00a2ec0 2024-04-25T01:25:34.218 INFO:teuthology.orchestra.run.smithi027.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T01:25:33", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f91a00a2ec0 msgr2=0x7f91a00a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f91a00a2ec0 0x7f91a00a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f91b4077ee0 tx=0x7f91a400b390 comp rx=0 tx=0).stop 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4100670 msgr2=0x7f91b4197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4100670 0x7f91b4197fc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f91b000d950 tx=0x7f91b000de10 comp rx=0 tx=0).stop 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 shutdown_connections 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f91b4106630 0x7f91b419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91b4101030 0x7f91b4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f91a00a2ec0 0x7f91a00a5380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 --2- 172.21.15.27:0/1160507155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b4100670 0x7f91b4197fc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 >> 172.21.15.27:0/1160507155 conn(0x7f91b40777c0 msgr2=0x7f91b4069130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 shutdown_connections 2024-04-25T01:25:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.239+0000 7f91bb332640 1 -- 172.21.15.27:0/1160507155 wait complete. 2024-04-25T01:25:34.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T01:25:34.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:34 smithi153 ceph-mon[32621]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 732 B/s wr, 22 op/s 2024-04-25T01:25:34.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:34 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/339464069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:34.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:34 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:25:34.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:34 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:25:34.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:34 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:25:34.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:34 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:25:34.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:34 smithi153 ceph-mon[32621]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:34.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:34 smithi139 ceph-mon[32760]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 732 B/s wr, 22 op/s 2024-04-25T01:25:34.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:34 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/339464069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:34.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:34 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:25:34.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:34 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:25:34.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:34 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:25:34.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:34 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:25:34.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:34 smithi139 ceph-mon[32760]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:34.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.353+0000 7ff1e3215640 1 -- 172.21.15.27:0/1087641686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 msgr2=0x7ff1dc10e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:34.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.353+0000 7ff1e3215640 1 --2- 172.21.15.27:0/1087641686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 0x7ff1dc10e1c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009900 tx=0x7ff1cc02a480 comp rx=0 tx=0).stop 2024-04-25T01:25:34.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.353+0000 7ff1e3215640 1 -- 172.21.15.27:0/1087641686 shutdown_connections 2024-04-25T01:25:34.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.353+0000 7ff1e3215640 1 --2- 172.21.15.27:0/1087641686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 0x7ff1dc10e1c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.353+0000 7ff1e3215640 1 --2- 172.21.15.27:0/1087641686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1dc103170 0x7ff1dc10b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.353+0000 7ff1e3215640 1 --2- 172.21.15.27:0/1087641686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1dc102850 0x7ff1dc102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.353+0000 7ff1e3215640 1 -- 172.21.15.27:0/1087641686 >> 172.21.15.27:0/1087641686 conn(0x7ff1dc0777c0 msgr2=0x7ff1dc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:34.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 -- 172.21.15.27:0/1087641686 shutdown_connections 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 -- 172.21.15.27:0/1087641686 wait complete. 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 Processor -- start 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 -- start start 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1dc102850 0x7ff1dc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1dc103170 0x7ff1dc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 0x7ff1dc19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc111b10 con 0x7ff1dc10bd70 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc111990 con 0x7ff1dc103170 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.354+0000 7ff1e3215640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc111c90 con 0x7ff1dc102850 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 0x7ff1dc19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 0x7ff1dc19c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50704/0 (socket says 172.21.15.27:50704) 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 -- 172.21.15.27:0/678525796 learned_addr learned my addr 172.21.15.27:0/678525796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1dbfff640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1dc103170 0x7ff1dc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e0fb1640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1dc102850 0x7ff1dc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 -- 172.21.15.27:0/678525796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1dc102850 msgr2=0x7ff1dc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1dc102850 0x7ff1dc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 -- 172.21.15.27:0/678525796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1dc103170 msgr2=0x7ff1dc198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1dc103170 0x7ff1dc198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 -- 172.21.15.27:0/678525796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc19d000 con 0x7ff1dc10bd70 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e17b2640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 0x7ff1dc19c8d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc005ab0 tx=0x7ff1cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1d9ffb640 1 -- 172.21.15.27:0/678525796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc03f030 con 0x7ff1dc10bd70 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1d9ffb640 1 -- 172.21.15.27:0/678525796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1cc003ce0 con 0x7ff1dc10bd70 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1dc19d290 con 0x7ff1dc10bd70 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1d9ffb640 1 -- 172.21.15.27:0/678525796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc035070 con 0x7ff1dc10bd70 2024-04-25T01:25:34.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.355+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1dc1a4b80 con 0x7ff1dc10bd70 2024-04-25T01:25:34.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.356+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1dc04f500 con 0x7ff1dc10bd70 2024-04-25T01:25:34.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.358+0000 7ff1d9ffb640 1 -- 172.21.15.27:0/678525796 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) ==== 138060+0+0 (secure 0 0 0) 0x7ff1cc042030 con 0x7ff1dc10bd70 2024-04-25T01:25:34.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.358+0000 7ff1d9ffb640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1c40a2f40 0x7ff1c40a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:34.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.359+0000 7ff1e0fb1640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1c40a2f40 0x7ff1c40a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:34.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.359+0000 7ff1d9ffb640 1 -- 172.21.15.27:0/678525796 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff1cc0f00a0 con 0x7ff1dc10bd70 2024-04-25T01:25:34.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.359+0000 7ff1e0fb1640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1c40a2f40 0x7ff1c40a5400 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4005c40 tx=0x7ff1d400a590 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:34.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.363+0000 7ff1d9ffb640 1 -- 172.21.15.27:0/678525796 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff1cc0bc160 con 0x7ff1dc10bd70 2024-04-25T01:25:34.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.569+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ff1dc106f50 con 0x7ff1c40a2f40 2024-04-25T01:25:34.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.575+0000 7ff1d9ffb640 1 -- 172.21.15.27:0/678525796 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ff1dc106f50 con 0x7ff1c40a2f40 2024-04-25T01:25:34.556 INFO:teuthology.orchestra.run.smithi027.stdout:/ 1m 6m3h 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1c40a2f40 msgr2=0x7ff1c40a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1c40a2f40 0x7ff1c40a5400 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4005c40 tx=0x7ff1d400a590 comp rx=0 tx=0).stop 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 msgr2=0x7ff1dc19c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 0x7ff1dc19c8d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc005ab0 tx=0x7ff1cc0042f0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 shutdown_connections 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1c40a2f40 0x7ff1c40a5400 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1dc10bd70 0x7ff1dc19c8d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1dc103170 0x7ff1dc198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 --2- 172.21.15.27:0/678525796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1dc102850 0x7ff1dc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 >> 172.21.15.27:0/678525796 conn(0x7ff1dc0777c0 msgr2=0x7ff1dc0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:34.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 shutdown_connections 2024-04-25T01:25:34.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:25:34.577+0000 7ff1e3215640 1 -- 172.21.15.27:0/678525796 wait complete. 2024-04-25T01:25:34.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T01:25:34.593 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T01:25:34.604 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T01:25:34.618 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:25:34.618 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T01:25:34.618 INFO:tasks.workunit:timeout=3h 2024-04-25T01:25:34.618 INFO:tasks.workunit:cleanup=True 2024-04-25T01:25:34.619 DEBUG:teuthology.orchestra.run.smithi027:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T01:25:34.649 INFO:teuthology.orchestra.run.smithi027.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T01:25:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-25T01:25:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-25T01:25:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:25:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-25T01:25:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 01:23:25.381738576 +0000 2024-04-25T01:25:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 01:23:25.381738576 +0000 2024-04-25T01:25:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 01:25:22.580850838 +0000 2024-04-25T01:25:34.650 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 01:23:25.381738576 +0000 2024-04-25T01:25:34.650 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T01:25:34.650 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T01:25:34.724 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:25:34.780 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T01:25:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:35 smithi153 ceph-mon[32621]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:35 smithi153 ceph-mon[32621]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:35.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:35 smithi153 ceph-mon[32621]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:35.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:35 smithi139 ceph-mon[32760]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:35.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:35 smithi139 ceph-mon[32760]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:35.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:35 smithi139 ceph-mon[32760]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:35.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:35 smithi027 ceph-mon[29197]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:35.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:35 smithi027 ceph-mon[29197]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:35.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:35 smithi027 ceph-mon[29197]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:36.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:36 smithi153 ceph-mon[32621]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 569 B/s wr, 17 op/s 2024-04-25T01:25:36.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:36 smithi139 ceph-mon[32760]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 569 B/s wr, 17 op/s 2024-04-25T01:25:36.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:36 smithi027 ceph-mon[29197]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 569 B/s wr, 17 op/s 2024-04-25T01:25:38.262 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:38 smithi027 ceph-mon[29197]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-25T01:25:38.276 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:38 smithi153 ceph-mon[32621]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-25T01:25:38.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:38 smithi139 ceph-mon[32760]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-25T01:25:40.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:40 smithi153 ceph-mon[32621]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-25T01:25:40.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:40 smithi139 ceph-mon[32760]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-25T01:25:40.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:40 smithi027 ceph-mon[29197]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-25T01:25:42.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:42 smithi153 ceph-mon[32621]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-25T01:25:42.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:42 smithi139 ceph-mon[32760]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-25T01:25:42.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:42 smithi027 ceph-mon[29197]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-25T01:25:44.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:44 smithi153 ceph-mon[32621]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-25T01:25:44.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:44 smithi139 ceph-mon[32760]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-25T01:25:44.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:44 smithi027 ceph-mon[29197]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-25T01:25:46.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:46 smithi153 ceph-mon[32621]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-25T01:25:46.291 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:46 smithi139 ceph-mon[32760]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-25T01:25:46.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:46 smithi027 ceph-mon[29197]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-25T01:25:48.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:48 smithi153 ceph-mon[32621]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-25T01:25:48.277 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:48 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:25:48.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:48 smithi139 ceph-mon[32760]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-25T01:25:48.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:48 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:25:48.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:48 smithi027 ceph-mon[29197]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-25T01:25:48.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:48 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:25:50.275 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:50 smithi153 ceph-mon[32621]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:50.291 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:50 smithi139 ceph-mon[32760]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:50.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:50 smithi027 ceph-mon[29197]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:52.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:52 smithi139 ceph-mon[32760]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:25:52.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:52 smithi027 ceph-mon[29197]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:25:52.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:52 smithi153 ceph-mon[32621]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:25:54.291 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:54 smithi139 ceph-mon[32760]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:25:54.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:54 smithi027 ceph-mon[29197]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:25:54.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:54 smithi153 ceph-mon[32621]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:25:56.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:56 smithi139 ceph-mon[32760]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:56.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:56 smithi027 ceph-mon[29197]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:56.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:56 smithi153 ceph-mon[32621]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:25:58.291 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:25:58 smithi139 ceph-mon[32760]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:25:58.327 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:25:58 smithi153 ceph-mon[32621]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:25:58.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:25:58 smithi027 ceph-mon[29197]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:00.292 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:00 smithi139 ceph-mon[32760]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:00.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:00 smithi027 ceph-mon[29197]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:00.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:00 smithi153 ceph-mon[32621]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:01.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:26:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:26:00.907+0000 7f37e36c6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:26:01.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:26:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:26:00.907+0000 7f37e36c6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:26:01.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:26:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:26:00.907+0000 7f37e36c6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:26:01.195 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:26:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:26:00.907+0000 7f37e36c6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:26:01.196 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:26:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x[29427]: 2024-04-25T01:26:00.907+0000 7f37e36c6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:26:02.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:02 smithi027 ceph-mon[29197]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-25T01:26:02.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:02 smithi027 ceph-mon[29197]: mgrmap e24: x(active, since 36s), standbys: y, z 2024-04-25T01:26:02.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:02 smithi153 ceph-mon[32621]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-25T01:26:02.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:02 smithi153 ceph-mon[32621]: mgrmap e24: x(active, since 36s), standbys: y, z 2024-04-25T01:26:02.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:02 smithi139 ceph-mon[32760]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-25T01:26:02.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:02 smithi139 ceph-mon[32760]: mgrmap e24: x(active, since 36s), standbys: y, z 2024-04-25T01:26:03.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:26:03.353 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 -- 172.21.15.27:0/1934697636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 msgr2=0x7f8108100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 --2- 172.21.15.27:0/1934697636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 0x7f8108100aa0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009930 tx=0x7f80fc02a420 comp rx=0 tx=0).stop 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 -- 172.21.15.27:0/1934697636 shutdown_connections 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 --2- 172.21.15.27:0/1934697636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8108100fe0 0x7f810810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 --2- 172.21.15.27:0/1934697636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 0x7f8108100aa0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 --2- 172.21.15.27:0/1934697636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8108106630 0x7f8108106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 -- 172.21.15.27:0/1934697636 >> 172.21.15.27:0/1934697636 conn(0x7f81080776a0 msgr2=0x7f8108077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 -- 172.21.15.27:0/1934697636 shutdown_connections 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 -- 172.21.15.27:0/1934697636 wait complete. 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.481+0000 7f810effb640 1 Processor -- start 2024-04-25T01:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810effb640 1 -- start start 2024-04-25T01:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810effb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 0x7f81080fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810effb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8108100fe0 0x7f81080fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810effb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8108106630 0x7f81080fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810effb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8108110a40 con 0x7f8108100620 2024-04-25T01:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810effb640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f81081108c0 con 0x7f8108106630 2024-04-25T01:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810effb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8108110bc0 con 0x7f8108100fe0 2024-04-25T01:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 0x7f81080fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 0x7f81080fd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55580/0 (socket says 172.21.15.27:55580) 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 -- 172.21.15.27:0/45686875 learned_addr learned my addr 172.21.15.27:0/45686875 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810d598640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8108106630 0x7f81080fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f8107fff640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8108100fe0 0x7f81080fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 -- 172.21.15.27:0/45686875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8108100fe0 msgr2=0x7f81080fdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8108100fe0 0x7f81080fdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 -- 172.21.15.27:0/45686875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8108106630 msgr2=0x7f81080fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8108106630 0x7f81080fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 -- 172.21.15.27:0/45686875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81081a6450 con 0x7f8108100620 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f8107fff640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8108100fe0 0x7f81080fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.482+0000 7f810cd97640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 0x7f81080fd650 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f80f800ea80 tx=0x7f80f800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.483+0000 7f8105ffb640 1 -- 172.21.15.27:0/45686875 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f800ce40 con 0x7f8108100620 2024-04-25T01:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.483+0000 7f8105ffb640 1 -- 172.21.15.27:0/45686875 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80f8005020 con 0x7f8108100620 2024-04-25T01:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.483+0000 7f810effb640 1 -- 172.21.15.27:0/45686875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81081a6740 con 0x7f8108100620 2024-04-25T01:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.483+0000 7f8105ffb640 1 -- 172.21.15.27:0/45686875 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8011760 con 0x7f8108100620 2024-04-25T01:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.483+0000 7f810effb640 1 -- 172.21.15.27:0/45686875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81081a6c90 con 0x7f8108100620 2024-04-25T01:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.485+0000 7f8105ffb640 1 -- 172.21.15.27:0/45686875 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7f80f80051d0 con 0x7f8108100620 2024-04-25T01:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.485+0000 7f8111de0740 1 Processor -- start 2024-04-25T01:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8111de0740 1 -- start start 2024-04-25T01:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8105ffb640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80f00a2f30 0x7f80f00a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8111de0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8111de0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61e20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8111de0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8111de0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dcb613c6b0 con 0x55dcb61d7f50 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8111de0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dcb60e4880 con 0x55dcb61e2620 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8111de0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dcb6138670 con 0x55dcb61d7630 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8105ffb640 1 -- 172.21.15.27:0/45686875 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f80f80d0130 con 0x7f8108100620 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f810d598640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80f00a2f30 0x7f80f00a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8107fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61e20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8107fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61e20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55588/0 (socket says 172.21.15.27:55588) 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8107fff640 1 -- 172.21.15.27:0/1437568720 learned_addr learned my addr 172.21.15.27:0/1437568720 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f810d598640 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f810cd97640 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8107fff640 1 -- 172.21.15.27:0/1437568720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 msgr2=0x55dcb61d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8107fff640 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8107fff640 1 -- 172.21.15.27:0/1437568720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 msgr2=0x55dcb61e4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8107fff640 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f8107fff640 1 -- 172.21.15.27:0/1437568720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcb5f1a7d0 con 0x55dcb61d7f50 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.486+0000 7f810cd97640 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.487+0000 7f8107fff640 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61e20e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f80fc006010 tx=0x7f80fc009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cf7a1397740ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.487+0000 7f810d598640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80f00a2f30 0x7f80f00a53f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8100013800 tx=0x7f810000aa40 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.487+0000 7f80eeffd640 1 -- 172.21.15.27:0/1437568720 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc03f070 con 0x55dcb61d7f50 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.487+0000 7f80eeffd640 1 -- 172.21.15.27:0/1437568720 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80fc0317b0 con 0x55dcb61d7f50 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.487+0000 7f80eeffd640 1 -- 172.21.15.27:0/1437568720 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc035070 con 0x55dcb61d7f50 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.487+0000 7f8111de0740 1 -- 172.21.15.27:0/1437568720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 msgr2=0x55dcb61e20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.487+0000 7f8111de0740 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61e20e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f80fc006010 tx=0x7f80fc009680 comp rx=0 tx=0).stop 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 -- 172.21.15.27:0/1437568720 shutdown_connections 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61e20e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 --2- 172.21.15.27:0/1437568720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 -- 172.21.15.27:0/1437568720 >> 172.21.15.27:0/1437568720 conn(0x55dcb61d3950 msgr2=0x55dcb61d3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 -- 172.21.15.27:0/1437568720 shutdown_connections 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 -- 172.21.15.27:0/1437568720 wait complete. 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 -- start start 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 Processor -- start 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.488+0000 7f8111de0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7630 0x55dcb61df580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7f50 0x55dcb61dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dcb61e53b0 con 0x55dcb61d7630 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55dcb60f7f50 con 0x55dcb61e2620 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dcb61e5530 con 0x55dcb61d7f50 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 10 client.0 _ll_register_callbacks cb 0x55dcb6133000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7630 0x55dcb61df580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7630 0x55dcb61df580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55596/0 (socket says 172.21.15.27:55596) 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 -- 172.21.15.27:0/3358372918 learned_addr learned my addr 172.21.15.27:0/3358372918 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8107fff640 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7f50 0x55dcb61dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810d598640 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 -- 172.21.15.27:0/3358372918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7f50 msgr2=0x55dcb61dfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7f50 0x55dcb61dfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 -- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 msgr2=0x55dcb61e0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 -- 172.21.15.27:0/3358372918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcb5e503f0 con 0x55dcb61d7630 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810d598640 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f810cd97640 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7630 0x55dcb61df580 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f80f80984e0 tx=0x7f80f809c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fe198f4e8a3554 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f80e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f80e7fff640 1 -- 172.21.15.27:0/3358372918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f801f070 con 0x55dcb61d7630 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f80e7fff640 1 -- 172.21.15.27:0/3358372918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f80f8005540 con 0x55dcb61d7630 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 1 -- 172.21.15.27:0/3358372918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80fc0317b0 con 0x55dcb61d7630 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f80e7fff640 1 -- 172.21.15.27:0/3358372918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f80f809c2e0 con 0x55dcb61d7630 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 10 client.15762 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.489+0000 7f8111de0740 1 -- 172.21.15.27:0/3358372918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dcb5e100f0 con 0x55dcb61d7630 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f80e7fff640 1 -- 172.21.15.27:0/3358372918 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80fc0317b0 con 0x55dcb61d7630 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f80e7fff640 1 -- 172.21.15.27:0/3358372918 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f80f800f040 con 0x55dcb61d7630 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f8111de0740 10 client.15762 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f8111de0740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f8111de0740 20 client.15762 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f8111de0740 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55dcb61f27b0 0x55dcb61f4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f8111de0740 4 client.15762 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f8111de0740 1 -- 172.21.15.27:0/3358372918 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcb6149bf0 con 0x55dcb61f27b0 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.490+0000 7f8107fff640 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55dcb61f27b0 0x55dcb61f4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.491+0000 7f8107fff640 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55dcb61f27b0 0x55dcb61f4be0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e713ada65d8fe325 server_cookie=aff3bea917085986 in_seq=0 out_seq=0 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.491+0000 7f80e7fff640 10 client.15762 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.491+0000 7f80e7fff640 1 -- 172.21.15.27:0/3358372918 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dcb6149bf0 con 0x55dcb61f27b0 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.491+0000 7f80e7fff640 10 client.15762 handle_command_reply: tid=0 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.491+0000 7f8111de0740 1 client.15762 shutdown 2024-04-25T01:26:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 -- 172.21.15.27:0/3358372918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7630 msgr2=0x55dcb61df580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7630 0x55dcb61df580 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f80f80984e0 tx=0x7f80f809c140 comp rx=0 tx=0).stop 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 -- 172.21.15.27:0/3358372918 shutdown_connections 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55dcb61f27b0 0x55dcb61f4be0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61e0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7f50 0x55dcb61dfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f80e7fff640 0 client.15762 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 --2- 172.21.15.27:0/3358372918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7630 0x55dcb61df580 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 -- 172.21.15.27:0/3358372918 >> 172.21.15.27:0/3358372918 conn(0x55dcb61d3950 msgr2=0x55dcb61d9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 -- 172.21.15.27:0/3358372918 shutdown_connections 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 1 -- 172.21.15.27:0/3358372918 wait complete. 2024-04-25T01:26:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.492+0000 7f8111de0740 20 client.15762 trim_cache size 0 max 16384 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.509+0000 7f8111de0740 1 Processor -- start 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f8111de0740 1 -- start start 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f8111de0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f8111de0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61d7f50 0x55dcb61ee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f8111de0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61e2620 0x55dcb61d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f8111de0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f81081b35b0 con 0x55dcb61e2620 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f8111de0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f81081b0e80 con 0x55dcb61d7f50 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f8111de0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81081aecd0 con 0x55dcb61d7630 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61e2620 0x55dcb61d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61e2620 0x55dcb61d3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55598/0 (socket says 172.21.15.27:55598) 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 -- 172.21.15.27:0/1480045817 learned_addr learned my addr 172.21.15.27:0/1480045817 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810cd97640 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f8107fff640 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61d7f50 0x55dcb61ee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 -- 172.21.15.27:0/1480045817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 msgr2=0x55dcb61d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 -- 172.21.15.27:0/1480045817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61d7f50 msgr2=0x55dcb61ee2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61d7f50 0x55dcb61ee2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 -- 172.21.15.27:0/1480045817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcb61d88c0 con 0x55dcb61e2620 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.510+0000 7f810d598640 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61e2620 0x55dcb61d3d30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8100005880 tx=0x7f810001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c779f6951275f70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f80ee7fc640 1 -- 172.21.15.27:0/1480045817 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100021b80 con 0x55dcb61e2620 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f80ee7fc640 1 -- 172.21.15.27:0/1480045817 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8100021d20 con 0x55dcb61e2620 2024-04-25T01:26:03.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f80ee7fc640 1 -- 172.21.15.27:0/1480045817 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100011640 con 0x55dcb61e2620 2024-04-25T01:26:03.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 -- 172.21.15.27:0/1480045817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61e2620 msgr2=0x55dcb61d3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61e2620 0x55dcb61d3d30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8100005880 tx=0x7f810001d000 comp rx=0 tx=0).stop 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 -- 172.21.15.27:0/1480045817 shutdown_connections 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61e2620 0x55dcb61d3d30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61d7f50 0x55dcb61ee2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 --2- 172.21.15.27:0/1480045817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 -- 172.21.15.27:0/1480045817 >> 172.21.15.27:0/1480045817 conn(0x55dcb61d3950 msgr2=0x55dcb61df540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 -- 172.21.15.27:0/1480045817 shutdown_connections 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.511+0000 7f8111de0740 1 -- 172.21.15.27:0/1480045817 wait complete. 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 1 -- start start 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 1 Processor -- start 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61dbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f81081af340 con 0x55dcb61d7f50 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f81081b07b0 con 0x55dcb61e2620 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81081afa10 con 0x55dcb61d7630 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8111de0740 10 client.0 _ll_register_callbacks cb 0x7f80f809be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8107fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8107fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61eacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55610/0 (socket says 172.21.15.27:55610) 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f8107fff640 1 -- 172.21.15.27:0/4045997176 learned_addr learned my addr 172.21.15.27:0/4045997176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f810d598640 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61dbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.512+0000 7f810cd97640 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8107fff640 1 -- 172.21.15.27:0/4045997176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 msgr2=0x55dcb61ea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8107fff640 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61ea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8107fff640 1 -- 172.21.15.27:0/4045997176 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 msgr2=0x55dcb61dbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8107fff640 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61dbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8107fff640 1 -- 172.21.15.27:0/4045997176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcb61dc600 con 0x55dcb61d7f50 2024-04-25T01:26:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8107fff640 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61eacf0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f80fc02a940 tx=0x7f80fc005360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2acbadbf98c9e70d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f80e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f80e7fff640 1 -- 172.21.15.27:0/4045997176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc035040 con 0x55dcb61d7f50 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f80e7fff640 1 -- 172.21.15.27:0/4045997176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f80fc04d7a0 con 0x55dcb61d7f50 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8111de0740 1 -- 172.21.15.27:0/4045997176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8100021d20 con 0x55dcb61d7f50 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f80e7fff640 1 -- 172.21.15.27:0/4045997176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f80fc05e050 con 0x55dcb61d7f50 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8111de0740 10 client.15768 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f8111de0740 1 -- 172.21.15.27:0/4045997176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dcb62589f0 con 0x55dcb61d7f50 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.513+0000 7f80e7fff640 1 -- 172.21.15.27:0/4045997176 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8100021d20 con 0x55dcb61d7f50 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f80e7fff640 1 -- 172.21.15.27:0/4045997176 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f80fc04d070 con 0x55dcb61d7f50 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f8111de0740 10 client.15768 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f8111de0740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f8111de0740 20 client.15768 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f8111de0740 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55dcb6258d30 0x55dcb625b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f8111de0740 4 client.15768 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f8111de0740 1 -- 172.21.15.27:0/4045997176 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dcb6143f40 con 0x55dcb6258d30 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f810cd97640 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55dcb6258d30 0x55dcb625b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f810cd97640 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55dcb6258d30 0x55dcb625b120 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef4392aabd996c58 server_cookie=dc5727b54a86cd78 in_seq=0 out_seq=0 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.514+0000 7f80e7fff640 10 client.15768 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f80e7fff640 1 -- 172.21.15.27:0/4045997176 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dcb6143f40 con 0x55dcb6258d30 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f80e7fff640 10 client.15768 handle_command_reply: tid=0 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 client.15768 shutdown 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 -- 172.21.15.27:0/4045997176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 msgr2=0x55dcb61eacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61eacf0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f80fc02a940 tx=0x7f80fc005360 comp rx=0 tx=0).stop 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 -- 172.21.15.27:0/4045997176 shutdown_connections 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55dcb6258d30 0x55dcb625b120 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55dcb61e2620 0x55dcb61dbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dcb61d7f50 0x55dcb61eacf0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f80e7fff640 0 client.15768 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 --2- 172.21.15.27:0/4045997176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dcb61d7630 0x55dcb61ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 -- 172.21.15.27:0/4045997176 >> 172.21.15.27:0/4045997176 conn(0x55dcb61d3950 msgr2=0x55dcb61d9610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 -- 172.21.15.27:0/4045997176 shutdown_connections 2024-04-25T01:26:03.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 1 -- 172.21.15.27:0/4045997176 wait complete. 2024-04-25T01:26:03.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.515+0000 7f8111de0740 20 client.15768 trim_cache size 0 max 16384 2024-04-25T01:26:03.501 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:26:03.501 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:26:03.501 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:26:03.501 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:26:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 -- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80f00a2f30 msgr2=0x7f80f00a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80f00a2f30 0x7f80f00a53f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8100013800 tx=0x7f810000aa40 comp rx=0 tx=0).stop 2024-04-25T01:26:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 -- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 msgr2=0x7f81080fd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 0x7f81080fd650 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f80f800ea80 tx=0x7f80f800ef40 comp rx=0 tx=0).stop 2024-04-25T01:26:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 -- 172.21.15.27:0/45686875 shutdown_connections 2024-04-25T01:26:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8108106630 0x7f81080fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8108100fe0 0x7f81080fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80f00a2f30 0x7f80f00a53f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 --2- 172.21.15.27:0/45686875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8108100620 0x7f81080fd650 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 -- 172.21.15.27:0/45686875 >> 172.21.15.27:0/45686875 conn(0x7f81080776a0 msgr2=0x7f81080ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 -- 172.21.15.27:0/45686875 shutdown_connections 2024-04-25T01:26:03.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.516+0000 7f810effb640 1 -- 172.21.15.27:0/45686875 wait complete. 2024-04-25T01:26:03.507 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:26:03.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:26:03.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 -- 172.21.15.27:0/3189212958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01029d0 msgr2=0x7f0ed0102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/3189212958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01029d0 0x7f0ed0102db0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009900 tx=0x7f0ec002a510 comp rx=0 tx=0).stop 2024-04-25T01:26:03.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 -- 172.21.15.27:0/3189212958 shutdown_connections 2024-04-25T01:26:03.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/3189212958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed010bd10 0x7f0ed010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/3189212958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed01032f0 0x7f0ed010b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/3189212958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01029d0 0x7f0ed0102db0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 -- 172.21.15.27:0/3189212958 >> 172.21.15.27:0/3189212958 conn(0x7f0ed0077740 msgr2=0x7f0ed0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 -- 172.21.15.27:0/3189212958 shutdown_connections 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.630+0000 7f0ed4d42640 1 -- 172.21.15.27:0/3189212958 wait complete. 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ed4d42640 1 Processor -- start 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ed4d42640 1 -- start start 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ed4d42640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed01029d0 0x7f0ed019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ed4d42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01032f0 0x7f0ed019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ed4d42640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed010bd10 0x7f0ed019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ed4d42640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed0111b10 con 0x7f0ed01032f0 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ed4d42640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed0111990 con 0x7f0ed010bd10 2024-04-25T01:26:03.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ed4d42640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed0111c90 con 0x7f0ed01029d0 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01032f0 0x7f0ed019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01032f0 0x7f0ed019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55630/0 (socket says 172.21.15.27:55630) 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 -- 172.21.15.27:0/2886400858 learned_addr learned my addr 172.21.15.27:0/2886400858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0eced9d640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed010bd10 0x7f0ed019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ece59c640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed01029d0 0x7f0ed019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 -- 172.21.15.27:0/2886400858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed01029d0 msgr2=0x7f0ed019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed01029d0 0x7f0ed019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 -- 172.21.15.27:0/2886400858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed010bd10 msgr2=0x7f0ed019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed010bd10 0x7f0ed019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 -- 172.21.15.27:0/2886400858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed019f220 con 0x7f0ed01032f0 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ece59c640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed01029d0 0x7f0ed019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.631+0000 7f0ecdd9b640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01032f0 0x7f0ed019a770 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400d770 tx=0x7f0ec400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.632+0000 7f0ebf7fe640 1 -- 172.21.15.27:0/2886400858 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4017070 con 0x7f0ed01032f0 2024-04-25T01:26:03.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.632+0000 7f0ebf7fe640 1 -- 172.21.15.27:0/2886400858 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec4005500 con 0x7f0ed01032f0 2024-04-25T01:26:03.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.632+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed019f510 con 0x7f0ed01032f0 2024-04-25T01:26:03.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.632+0000 7f0ebf7fe640 1 -- 172.21.15.27:0/2886400858 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4004e00 con 0x7f0ed01032f0 2024-04-25T01:26:03.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.632+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed01a6da0 con 0x7f0ed01032f0 2024-04-25T01:26:03.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.633+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed0068560 con 0x7f0ed01032f0 2024-04-25T01:26:03.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.634+0000 7f0ebf7fe640 1 -- 172.21.15.27:0/2886400858 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7f0ec4004fa0 con 0x7f0ed01032f0 2024-04-25T01:26:03.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.635+0000 7f0ebf7fe640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0eb80a2eb0 0x7f0eb80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.635+0000 7f0ebf7fe640 1 -- 172.21.15.27:0/2886400858 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0ec40cf8f0 con 0x7f0ed01032f0 2024-04-25T01:26:03.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.635+0000 7f0ece59c640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0eb80a2eb0 0x7f0eb80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.635+0000 7f0ece59c640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0eb80a2eb0 0x7f0eb80a5370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0000c00 tx=0x7f0ec0032c70 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.643+0000 7f0ebf7fe640 1 -- 172.21.15.27:0/2886400858 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ec409bf40 con 0x7f0ed01032f0 2024-04-25T01:26:03.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.894+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ed01a7000 con 0x7f0ed01032f0 2024-04-25T01:26:04.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.894+0000 7f0ebf7fe640 1 -- 172.21.15.27:0/2886400858 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f0ec409b840 con 0x7f0ed01032f0 2024-04-25T01:26:04.092 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:26:04.093 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:25:23.250708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24505,24517]},"id":1}]} 2024-04-25T01:26:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-04-25T01:26:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0eb80a2eb0 msgr2=0x7f0eb80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0eb80a2eb0 0x7f0eb80a5370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0000c00 tx=0x7f0ec0032c70 comp rx=0 tx=0).stop 2024-04-25T01:26:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01032f0 msgr2=0x7f0ed019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01032f0 0x7f0ed019a770 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400d770 tx=0x7f0ec400dc30 comp rx=0 tx=0).stop 2024-04-25T01:26:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 shutdown_connections 2024-04-25T01:26:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ed010bd10 0x7f0ed019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0eb80a2eb0 0x7f0eb80a5370 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed01032f0 0x7f0ed019a770 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 --2- 172.21.15.27:0/2886400858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ed01029d0 0x7f0ed019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.896+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 >> 172.21.15.27:0/2886400858 conn(0x7f0ed0077740 msgr2=0x7f0ed0107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.897+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 shutdown_connections 2024-04-25T01:26:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:03.897+0000 7f0ed4d42640 1 -- 172.21.15.27:0/2886400858 wait complete. 2024-04-25T01:26:04.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:26:04.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:04 smithi027 ceph-mon[29197]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-25T01:26:04.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2886400858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.230+0000 7fef038cd640 1 -- 172.21.15.27:0/3656771364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc10bd00 msgr2=0x7feefc10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.230+0000 7fef038cd640 1 --2- 172.21.15.27:0/3656771364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc10bd00 0x7feefc10e1a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feef4009ca0 tx=0x7feef402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.230+0000 7fef038cd640 1 -- 172.21.15.27:0/3656771364 shutdown_connections 2024-04-25T01:26:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.230+0000 7fef038cd640 1 --2- 172.21.15.27:0/3656771364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc10bd00 0x7feefc10e1a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.230+0000 7fef038cd640 1 --2- 172.21.15.27:0/3656771364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feefc1032e0 0x7feefc10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.230+0000 7fef038cd640 1 --2- 172.21.15.27:0/3656771364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feefc1029c0 0x7feefc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.230+0000 7fef038cd640 1 -- 172.21.15.27:0/3656771364 >> 172.21.15.27:0/3656771364 conn(0x7feefc077740 msgr2=0x7feefc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 -- 172.21.15.27:0/3656771364 shutdown_connections 2024-04-25T01:26:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 -- 172.21.15.27:0/3656771364 wait complete. 2024-04-25T01:26:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 Processor -- start 2024-04-25T01:26:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 -- start start 2024-04-25T01:26:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc1029c0 0x7feefc100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feefc1032e0 0x7feefc100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feefc10bd00 0x7feefc101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feefc111b50 con 0x7feefc1029c0 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feefc1119d0 con 0x7feefc1032e0 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef038cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feefc111cd0 con 0x7feefc10bd00 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.231+0000 7fef01669640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc1029c0 0x7feefc100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01669640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc1029c0 0x7feefc100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55642/0 (socket says 172.21.15.27:55642) 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01669640 1 -- 172.21.15.27:0/3005519062 learned_addr learned my addr 172.21.15.27:0/3005519062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01e6a640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feefc10bd00 0x7feefc101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef00e68640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feefc1032e0 0x7feefc100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01669640 1 -- 172.21.15.27:0/3005519062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feefc10bd00 msgr2=0x7feefc101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01669640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feefc10bd00 0x7feefc101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01669640 1 -- 172.21.15.27:0/3005519062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feefc1032e0 msgr2=0x7feefc100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01669640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feefc1032e0 0x7feefc100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01669640 1 -- 172.21.15.27:0/3005519062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feefc1a63f0 con 0x7feefc1029c0 2024-04-25T01:26:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef00e68640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feefc1032e0 0x7feefc100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef01669640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc1029c0 0x7feefc100610 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feef800e9f0 tx=0x7feef800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7feef27fc640 1 -- 172.21.15.27:0/3005519062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef800cd80 con 0x7feefc1029c0 2024-04-25T01:26:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7feef27fc640 1 -- 172.21.15.27:0/3005519062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef8010910 con 0x7feefc1029c0 2024-04-25T01:26:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7feef27fc640 1 -- 172.21.15.27:0/3005519062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8018990 con 0x7feefc1029c0 2024-04-25T01:26:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef038cd640 1 -- 172.21.15.27:0/3005519062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feefc1a66e0 con 0x7feefc1029c0 2024-04-25T01:26:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.232+0000 7fef038cd640 1 -- 172.21.15.27:0/3005519062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feefc1a6c30 con 0x7feefc1029c0 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.234+0000 7feef27fc640 1 -- 172.21.15.27:0/3005519062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7feef8010ac0 con 0x7feefc1029c0 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.235+0000 7feef27fc640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feee80a2ee0 0x7feee80a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.235+0000 7feef27fc640 1 -- 172.21.15.27:0/3005519062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7feef80cfe20 con 0x7feefc1029c0 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.235+0000 7fef00e68640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feee80a2ee0 0x7feee80a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef00e68640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feee80a2ee0 0x7feee80a53a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feefc102050 tx=0x7feeec005d20 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef065dc740 1 Processor -- start 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef065dc740 1 -- start start 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef065dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 0x55eddc7c8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef065dc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef065dc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef065dc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eddc72d670 con 0x55eddc7d3570 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef065dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eddc6d5970 con 0x55eddc7c8580 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef065dc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eddc729630 con 0x55eddc7c8ea0 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.236+0000 7fef01e6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01e6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55646/0 (socket says 172.21.15.27:55646) 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01e6a640 1 -- 172.21.15.27:0/797857840 learned_addr learned my addr 172.21.15.27:0/797857840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01669640 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 0x55eddc7c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef00e68640 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01e6a640 1 -- 172.21.15.27:0/797857840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 msgr2=0x55eddc7d3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01e6a640 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01e6a640 1 -- 172.21.15.27:0/797857840 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 msgr2=0x55eddc7c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01e6a640 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 0x55eddc7c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01e6a640 1 -- 172.21.15.27:0/797857840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eddc6f9400 con 0x55eddc7d3570 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7fef01e6a640 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d5960 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feef4009ca0 tx=0x7feef4005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0b5777070b399b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7feee77fe640 1 -- 172.21.15.27:0/797857840 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4033050 con 0x55eddc7d3570 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7feee77fe640 1 -- 172.21.15.27:0/797857840 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef4047070 con 0x55eddc7d3570 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.237+0000 7feee77fe640 1 -- 172.21.15.27:0/797857840 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef402f840 con 0x55eddc7d3570 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 -- 172.21.15.27:0/797857840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 msgr2=0x55eddc7d5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d5960 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feef4009ca0 tx=0x7feef4005d80 comp rx=0 tx=0).stop 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 -- 172.21.15.27:0/797857840 shutdown_connections 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d5960 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 --2- 172.21.15.27:0/797857840 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 0x55eddc7c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 -- 172.21.15.27:0/797857840 >> 172.21.15.27:0/797857840 conn(0x55eddc7c48a0 msgr2=0x55eddc7c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 -- 172.21.15.27:0/797857840 shutdown_connections 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 -- 172.21.15.27:0/797857840 wait complete. 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.238+0000 7fef065dc740 1 -- start start 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef065dc740 1 Processor -- start 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef065dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 0x55eddc7d2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef065dc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef065dc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef065dc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eddc7d6300 con 0x55eddc7d3570 2024-04-25T01:26:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef065dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55eddc6e9050 con 0x55eddc7c8580 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef065dc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eddc7d6480 con 0x55eddc7c8ea0 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef065dc740 10 client.0 _ll_register_callbacks cb 0x55eddc440be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef01e6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef01e6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55650/0 (socket says 172.21.15.27:55650) 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef01e6a640 1 -- 172.21.15.27:0/936655242 learned_addr learned my addr 172.21.15.27:0/936655242 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef00e68640 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef01669640 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 0x55eddc7d2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef01e6a640 1 -- 172.21.15.27:0/936655242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 msgr2=0x55eddc7d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef01e6a640 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.239+0000 7fef01e6a640 1 -- 172.21.15.27:0/936655242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 msgr2=0x55eddc7d2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7fef01e6a640 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 0x55eddc7d2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7fef01e6a640 1 -- 172.21.15.27:0/936655242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eddc6badc0 con 0x55eddc7d3570 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7fef01e6a640 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d0690 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feef4009bf0 tx=0x7feef402a9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b50805c9707030 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7feee4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7feee4ff9640 1 -- 172.21.15.27:0/936655242 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef404f070 con 0x55eddc7d3570 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7feee4ff9640 1 -- 172.21.15.27:0/936655242 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7feef402f840 con 0x55eddc7d3570 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7fef065dc740 1 -- 172.21.15.27:0/936655242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef4047070 con 0x55eddc7d3570 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7feee4ff9640 1 -- 172.21.15.27:0/936655242 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7feef40098e0 con 0x55eddc7d3570 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7fef065dc740 10 client.15786 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7fef065dc740 1 -- 172.21.15.27:0/936655242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eddc4413f0 con 0x55eddc7d3570 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7feee4ff9640 1 -- 172.21.15.27:0/936655242 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef4047070 con 0x55eddc7d3570 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.240+0000 7feee4ff9640 1 -- 172.21.15.27:0/936655242 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7feef4004680 con 0x55eddc7d3570 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7fef065dc740 10 client.15786 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7fef065dc740 10 client.15786 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7fef065dc740 20 client.15786 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7fef065dc740 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55eddc7e33c0 0x55eddc7e57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7fef065dc740 4 client.15786 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7fef065dc740 1 -- 172.21.15.27:0/936655242 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eddc73abb0 con 0x55eddc7e33c0 2024-04-25T01:26:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7fef01669640 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55eddc7e33c0 0x55eddc7e57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7fef01669640 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55eddc7e33c0 0x55eddc7e57b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9730984291f87159 server_cookie=c791aa9c3a6c824a in_seq=0 out_seq=0 2024-04-25T01:26:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.241+0000 7feee4ff9640 10 client.15786 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:26:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7feee4ff9640 1 -- 172.21.15.27:0/936655242 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eddc73abb0 con 0x55eddc7e33c0 2024-04-25T01:26:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7feee4ff9640 10 client.15786 handle_command_reply: tid=0 2024-04-25T01:26:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 client.15786 shutdown 2024-04-25T01:26:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 -- 172.21.15.27:0/936655242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 msgr2=0x55eddc7d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d0690 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feef4009bf0 tx=0x7feef402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 -- 172.21.15.27:0/936655242 shutdown_connections 2024-04-25T01:26:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55eddc7e33c0 0x55eddc7e57b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7feee4ff9640 0 client.15786 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:26:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7d3570 0x55eddc7d0690 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 --2- 172.21.15.27:0/936655242 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7c8580 0x55eddc7d2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.242+0000 7fef065dc740 1 -- 172.21.15.27:0/936655242 >> 172.21.15.27:0/936655242 conn(0x55eddc7c48a0 msgr2=0x55eddc7ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.243+0000 7fef065dc740 1 -- 172.21.15.27:0/936655242 shutdown_connections 2024-04-25T01:26:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.243+0000 7fef065dc740 1 -- 172.21.15.27:0/936655242 wait complete. 2024-04-25T01:26:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.243+0000 7fef065dc740 20 client.15786 trim_cache size 0 max 16384 2024-04-25T01:26:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.256+0000 7fef065dc740 1 Processor -- start 2024-04-25T01:26:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.256+0000 7fef065dc740 1 -- start start 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef065dc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7df1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef065dc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef065dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7eae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef065dc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feefc1b3530 con 0x55eddc7c8580 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef065dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feefc1b0e00 con 0x55eddc7d3570 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef065dc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feefc1aec50 con 0x55eddc7c8ea0 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7df1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7df1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55666/0 (socket says 172.21.15.27:55666) 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 -- 172.21.15.27:0/1300125846 learned_addr learned my addr 172.21.15.27:0/1300125846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef00e68640 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01e6a640 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7eae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 -- 172.21.15.27:0/1300125846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 msgr2=0x55eddc7cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 -- 172.21.15.27:0/1300125846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 msgr2=0x55eddc7eae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7eae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 -- 172.21.15.27:0/1300125846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eddc50b7d0 con 0x55eddc7c8580 2024-04-25T01:26:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7fef01669640 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7df1f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feef809ba70 tx=0x7feef8018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a614caeea5f1ae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7feee6ffd640 1 -- 172.21.15.27:0/1300125846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef809a070 con 0x55eddc7c8580 2024-04-25T01:26:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7feee6ffd640 1 -- 172.21.15.27:0/1300125846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef80983e0 con 0x55eddc7c8580 2024-04-25T01:26:04.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.257+0000 7feee6ffd640 1 -- 172.21.15.27:0/1300125846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef809b1c0 con 0x55eddc7c8580 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 -- 172.21.15.27:0/1300125846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 msgr2=0x55eddc7df1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7df1f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feef809ba70 tx=0x7feef8018420 comp rx=0 tx=0).stop 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 -- 172.21.15.27:0/1300125846 shutdown_connections 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7eae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 --2- 172.21.15.27:0/1300125846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7df1f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 -- 172.21.15.27:0/1300125846 >> 172.21.15.27:0/1300125846 conn(0x55eddc7c48a0 msgr2=0x55eddc7c32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 -- 172.21.15.27:0/1300125846 shutdown_connections 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.258+0000 7fef065dc740 1 -- 172.21.15.27:0/1300125846 wait complete. 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 1 -- start start 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 1 Processor -- start 2024-04-25T01:26:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7db7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7d9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feefc1af2c0 con 0x55eddc7c8580 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feefc1b0730 con 0x55eddc7d3570 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feefc1af990 con 0x55eddc7c8ea0 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef065dc740 10 client.0 _ll_register_callbacks cb 0x7feef402eb30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef01669640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7db7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef01669640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7db7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55672/0 (socket says 172.21.15.27:55672) 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.259+0000 7fef01669640 1 -- 172.21.15.27:0/2683730732 learned_addr learned my addr 172.21.15.27:0/2683730732 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef01e6a640 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7d9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef00e68640 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef01669640 1 -- 172.21.15.27:0/2683730732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 msgr2=0x55eddc7d8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef01669640 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef01669640 1 -- 172.21.15.27:0/2683730732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 msgr2=0x55eddc7d9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef01669640 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7d9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef01669640 1 -- 172.21.15.27:0/2683730732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eddc7d9750 con 0x55eddc7c8580 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef01e6a640 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7d9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef01669640 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7db7c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7feef80cd090 tx=0x7feef80ccf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7342c415fba5aed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7feee4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7feee4ff9640 1 -- 172.21.15.27:0/2683730732 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef80a4750 con 0x55eddc7c8580 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7feee4ff9640 1 -- 172.21.15.27:0/2683730732 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7feef8018b30 con 0x55eddc7c8580 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef065dc740 1 -- 172.21.15.27:0/2683730732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef80983e0 con 0x55eddc7c8580 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7feee4ff9640 1 -- 172.21.15.27:0/2683730732 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7feef8018d70 con 0x55eddc7c8580 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef065dc740 10 client.15792 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7fef065dc740 1 -- 172.21.15.27:0/2683730732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eddc8493d0 con 0x55eddc7c8580 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.260+0000 7feee4ff9640 1 -- 172.21.15.27:0/2683730732 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef80983e0 con 0x55eddc7c8580 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7feee4ff9640 1 -- 172.21.15.27:0/2683730732 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7feef80cd470 con 0x55eddc7c8580 2024-04-25T01:26:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7fef065dc740 10 client.15792 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7fef065dc740 10 client.15792 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7fef065dc740 20 client.15792 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7fef065dc740 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55eddc7e33c0 0x55eddc84bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7fef065dc740 4 client.15792 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7fef065dc740 1 -- 172.21.15.27:0/2683730732 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eddc734f00 con 0x55eddc7e33c0 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7fef00e68640 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55eddc7e33c0 0x55eddc84bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7feee4ff9640 10 client.15792 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.261+0000 7fef00e68640 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55eddc7e33c0 0x55eddc84bb90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86d259001c5b0fd6 server_cookie=f9006e2b3283eca0 in_seq=0 out_seq=0 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7feee4ff9640 1 -- 172.21.15.27:0/2683730732 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eddc734f00 con 0x55eddc7e33c0 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7feee4ff9640 10 client.15792 handle_command_reply: tid=0 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 client.15792 shutdown 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 -- 172.21.15.27:0/2683730732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 msgr2=0x55eddc7db7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7db7c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7feef80cd090 tx=0x7feef80ccf10 comp rx=0 tx=0).stop 2024-04-25T01:26:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 -- 172.21.15.27:0/2683730732 shutdown_connections 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55eddc7e33c0 0x55eddc84bb90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55eddc7d3570 0x55eddc7d9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7feee4ff9640 0 client.15792 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eddc7c8ea0 0x55eddc7d8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 --2- 172.21.15.27:0/2683730732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eddc7c8580 0x55eddc7db7c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 -- 172.21.15.27:0/2683730732 >> 172.21.15.27:0/2683730732 conn(0x55eddc7c48a0 msgr2=0x55eddc7cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 -- 172.21.15.27:0/2683730732 shutdown_connections 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 1 -- 172.21.15.27:0/2683730732 wait complete. 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.262+0000 7fef065dc740 20 client.15792 trim_cache size 0 max 16384 2024-04-25T01:26:04.248 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:26:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 -- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feee80a2ee0 msgr2=0x7feee80a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feee80a2ee0 0x7feee80a53a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feefc102050 tx=0x7feeec005d20 comp rx=0 tx=0).stop 2024-04-25T01:26:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 -- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc1029c0 msgr2=0x7feefc100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc1029c0 0x7feefc100610 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feef800e9f0 tx=0x7feef800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 -- 172.21.15.27:0/3005519062 shutdown_connections 2024-04-25T01:26:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feefc10bd00 0x7feefc101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feefc1032e0 0x7feefc100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feee80a2ee0 0x7feee80a53a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 --2- 172.21.15.27:0/3005519062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feefc1029c0 0x7feefc100610 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 -- 172.21.15.27:0/3005519062 >> 172.21.15.27:0/3005519062 conn(0x7feefc077740 msgr2=0x7feefc0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 -- 172.21.15.27:0/3005519062 shutdown_connections 2024-04-25T01:26:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.263+0000 7fef038cd640 1 -- 172.21.15.27:0/3005519062 wait complete. 2024-04-25T01:26:04.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:26:04.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 -- 172.21.15.27:0/371995630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 msgr2=0x7fa67c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 --2- 172.21.15.27:0/371995630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 0x7fa67c102da0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa66c009900 tx=0x7fa66c02a510 comp rx=0 tx=0).stop 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 -- 172.21.15.27:0/371995630 shutdown_connections 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 --2- 172.21.15.27:0/371995630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa67c10bd00 0x7fa67c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 --2- 172.21.15.27:0/371995630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c1032e0 0x7fa67c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 --2- 172.21.15.27:0/371995630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 0x7fa67c102da0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 -- 172.21.15.27:0/371995630 >> 172.21.15.27:0/371995630 conn(0x7fa67c077740 msgr2=0x7fa67c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 -- 172.21.15.27:0/371995630 shutdown_connections 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.383+0000 7fa682514640 1 -- 172.21.15.27:0/371995630 wait complete. 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa682514640 1 Processor -- start 2024-04-25T01:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa682514640 1 -- start start 2024-04-25T01:26:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa682514640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 0x7fa67c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa682514640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa67c1032e0 0x7fa67c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa682514640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c10bd00 0x7fa67c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa682514640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c111b00 con 0x7fa67c1029c0 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa682514640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c111980 con 0x7fa67c10bd00 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa682514640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c111c80 con 0x7fa67c1032e0 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa67bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 0x7fa67c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa67bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 0x7fa67c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55684/0 (socket says 172.21.15.27:55684) 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa67bfff640 1 -- 172.21.15.27:0/2278150445 learned_addr learned my addr 172.21.15.27:0/2278150445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa680ab1640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c10bd00 0x7fa67c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa67c1032e0 0x7fa67c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa67bfff640 1 -- 172.21.15.27:0/2278150445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa67c1032e0 msgr2=0x7fa67c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.384+0000 7fa67bfff640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa67c1032e0 0x7fa67c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa67bfff640 1 -- 172.21.15.27:0/2278150445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c10bd00 msgr2=0x7fa67c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa67bfff640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c10bd00 0x7fa67c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa67bfff640 1 -- 172.21.15.27:0/2278150445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c19cff0 con 0x7fa67c1029c0 2024-04-25T01:26:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa67bfff640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 0x7fa67c198000 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa66c006010 tx=0x7fa66c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa6797fa640 1 -- 172.21.15.27:0/2278150445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c03f030 con 0x7fa67c1029c0 2024-04-25T01:26:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa682514640 1 -- 172.21.15.27:0/2278150445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa67c19d280 con 0x7fa67c1029c0 2024-04-25T01:26:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa682514640 1 -- 172.21.15.27:0/2278150445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa67c1a4b70 con 0x7fa67c1029c0 2024-04-25T01:26:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa6797fa640 1 -- 172.21.15.27:0/2278150445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa66c003ce0 con 0x7fa67c1029c0 2024-04-25T01:26:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.385+0000 7fa6797fa640 1 -- 172.21.15.27:0/2278150445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c049060 con 0x7fa67c1029c0 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.387+0000 7fa6797fa640 1 -- 172.21.15.27:0/2278150445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7fa66c042030 con 0x7fa67c1029c0 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa685227740 1 Processor -- start 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa685227740 1 -- start start 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa685227740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1480 0x55a53b7c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa685227740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 0x55a53b7d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa685227740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7d2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa685227740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a53b726670 con 0x55a53b7d0980 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa6797fa640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6640a2f30 0x7fa6640a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa685227740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a53b6ce970 con 0x55a53b7c1480 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa685227740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a53b722630 con 0x55a53b7c1da0 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7d2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7d2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55696/0 (socket says 172.21.15.27:55696) 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67b7fe640 1 -- 172.21.15.27:0/848484240 learned_addr learned my addr 172.21.15.27:0/848484240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa680ab1640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6640a2f30 0x7fa6640a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67bfff640 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 0x55a53b7d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa680ab1640 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1480 0x55a53b7c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa6797fa640 1 -- 172.21.15.27:0/2278150445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa66c0effb0 con 0x7fa67c1029c0 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67b7fe640 1 -- 172.21.15.27:0/848484240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 msgr2=0x55a53b7d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 0x55a53b7d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67b7fe640 1 -- 172.21.15.27:0/848484240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1480 msgr2=0x55a53b7c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1480 0x55a53b7c1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.388+0000 7fa67b7fe640 1 -- 172.21.15.27:0/848484240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53b6f2400 con 0x55a53b7d0980 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.389+0000 7fa680ab1640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6640a2f30 0x7fa6640a53f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6740098b0 tx=0x7fa674009d90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.389+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7d2d70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa670009c00 tx=0x7fa670010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59599d0faf47f8e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.389+0000 7fa6627fc640 1 -- 172.21.15.27:0/848484240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670004020 con 0x55a53b7d0980 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.389+0000 7fa6627fc640 1 -- 172.21.15.27:0/848484240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6700041c0 con 0x55a53b7d0980 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.389+0000 7fa6627fc640 1 -- 172.21.15.27:0/848484240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700126b0 con 0x55a53b7d0980 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- 172.21.15.27:0/848484240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 msgr2=0x55a53b7d2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7d2d70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa670009c00 tx=0x7fa670010740 comp rx=0 tx=0).stop 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- 172.21.15.27:0/848484240 shutdown_connections 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7d2d70 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 0x55a53b7d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 --2- 172.21.15.27:0/848484240 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1480 0x55a53b7c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- 172.21.15.27:0/848484240 >> 172.21.15.27:0/848484240 conn(0x55a53b7bd8a0 msgr2=0x55a53b7bdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- 172.21.15.27:0/848484240 shutdown_connections 2024-04-25T01:26:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- 172.21.15.27:0/848484240 wait complete. 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- start start 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 Processor -- start 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1480 0x55a53b7d0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 0x55a53b7cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a53b7d3740 con 0x55a53b7d0980 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a53b6e2050 con 0x55a53b7c1da0 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.390+0000 7fa685227740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a53b7d38c0 con 0x55a53b7c1480 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa685227740 10 client.0 _ll_register_callbacks cb 0x55a53b439be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55702/0 (socket says 172.21.15.27:55702) 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 -- 172.21.15.27:0/3731593935 learned_addr learned my addr 172.21.15.27:0/3731593935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 0x55a53b7cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa67bfff640 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1480 0x55a53b7d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 -- 172.21.15.27:0/3731593935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1480 msgr2=0x55a53b7d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1480 0x55a53b7d0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 -- 172.21.15.27:0/3731593935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 msgr2=0x55a53b7cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 0x55a53b7cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 -- 172.21.15.27:0/3731593935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53b6b3dc0 con 0x55a53b7d0980 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa680ab1640 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7cda90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa6740092f0 tx=0x7fa674009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871b6e1097429866 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa65b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa65b7fe640 1 -- 172.21.15.27:0/3731593935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa674004b20 con 0x55a53b7d0980 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa65b7fe640 1 -- 172.21.15.27:0/3731593935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa674004cc0 con 0x55a53b7d0980 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa685227740 1 -- 172.21.15.27:0/3731593935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6700041c0 con 0x55a53b7d0980 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa65b7fe640 1 -- 172.21.15.27:0/3731593935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa674030050 con 0x55a53b7d0980 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa685227740 10 client.15804 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.391+0000 7fa685227740 1 -- 172.21.15.27:0/3731593935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a53b43a3f0 con 0x55a53b7d0980 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa65b7fe640 1 -- 172.21.15.27:0/3731593935 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6700041c0 con 0x55a53b7d0980 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa65b7fe640 1 -- 172.21.15.27:0/3731593935 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa67401f070 con 0x55a53b7d0980 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa685227740 10 client.15804 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa685227740 10 client.15804 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa685227740 20 client.15804 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa685227740 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55a53b7e0910 0x55a53b7e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa685227740 4 client.15804 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa685227740 1 -- 172.21.15.27:0/3731593935 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a53b733bb0 con 0x55a53b7e0910 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.392+0000 7fa67bfff640 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55a53b7e0910 0x55a53b7e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.393+0000 7fa67bfff640 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55a53b7e0910 0x55a53b7e2d40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1355e25009a2b09c server_cookie=f8e32964afdc77a1 in_seq=0 out_seq=0 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.393+0000 7fa65b7fe640 10 client.15804 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.393+0000 7fa65b7fe640 1 -- 172.21.15.27:0/3731593935 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a53b733bb0 con 0x55a53b7e0910 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.393+0000 7fa65b7fe640 10 client.15804 handle_command_reply: tid=0 2024-04-25T01:26:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.393+0000 7fa685227740 1 client.15804 shutdown 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.393+0000 7fa685227740 1 -- 172.21.15.27:0/3731593935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 msgr2=0x55a53b7cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.393+0000 7fa685227740 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7cda90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa6740092f0 tx=0x7fa674009600 comp rx=0 tx=0).stop 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 1 -- 172.21.15.27:0/3731593935 shutdown_connections 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55a53b7e0910 0x55a53b7e2d40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7d0980 0x55a53b7cda90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 0x55a53b7cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa65b7fe640 0 client.15804 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 1 --2- 172.21.15.27:0/3731593935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1480 0x55a53b7d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 1 -- 172.21.15.27:0/3731593935 >> 172.21.15.27:0/3731593935 conn(0x55a53b7bd8a0 msgr2=0x55a53b7c36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 1 -- 172.21.15.27:0/3731593935 shutdown_connections 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 1 -- 172.21.15.27:0/3731593935 wait complete. 2024-04-25T01:26:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.394+0000 7fa685227740 20 client.15804 trim_cache size 0 max 16384 2024-04-25T01:26:04.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.407+0000 7fa685227740 1 Processor -- start 2024-04-25T01:26:04.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.407+0000 7fa685227740 1 -- start start 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.407+0000 7fa685227740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.407+0000 7fa685227740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 0x55a53b7cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.407+0000 7fa685227740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7d0980 0x55a53b7c3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.407+0000 7fa685227740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c1af220 con 0x55a53b7c1480 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.407+0000 7fa685227740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c1acaf0 con 0x55a53b7c1da0 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.407+0000 7fa685227740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c1aa940 con 0x55a53b7d0980 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55714/0 (socket says 172.21.15.27:55714) 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 -- 172.21.15.27:0/526237300 learned_addr learned my addr 172.21.15.27:0/526237300 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa680ab1640 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7d0980 0x55a53b7c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 0x55a53b7cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 -- 172.21.15.27:0/526237300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7d0980 msgr2=0x55a53b7c3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7d0980 0x55a53b7c3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 -- 172.21.15.27:0/526237300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 msgr2=0x55a53b7cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 0x55a53b7cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 -- 172.21.15.27:0/526237300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53b5047d0 con 0x55a53b7c1480 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa680ab1640 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7d0980 0x55a53b7c3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa67bfff640 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7c8c70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0b7be0 tx=0x7fa66c0dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4e2d9503ed2f47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa661ffb640 1 -- 172.21.15.27:0/526237300 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c035040 con 0x55a53b7c1480 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa661ffb640 1 -- 172.21.15.27:0/526237300 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa66c0bc600 con 0x55a53b7c1480 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.408+0000 7fa661ffb640 1 -- 172.21.15.27:0/526237300 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0ba290 con 0x55a53b7c1480 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 -- 172.21.15.27:0/526237300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 msgr2=0x55a53b7c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7c8c70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0b7be0 tx=0x7fa66c0dc040 comp rx=0 tx=0).stop 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 -- 172.21.15.27:0/526237300 shutdown_connections 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7d0980 0x55a53b7c3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7c1da0 0x55a53b7cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 --2- 172.21.15.27:0/526237300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7c8c70 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 -- 172.21.15.27:0/526237300 >> 172.21.15.27:0/526237300 conn(0x55a53b7bd8a0 msgr2=0x55a53b7c2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 -- 172.21.15.27:0/526237300 shutdown_connections 2024-04-25T01:26:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.409+0000 7fa685227740 1 -- 172.21.15.27:0/526237300 wait complete. 2024-04-25T01:26:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 1 -- start start 2024-04-25T01:26:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 1 Processor -- start 2024-04-25T01:26:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7d8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 0x55a53b7c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7d0980 0x55a53b7ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c1aafb0 con 0x55a53b7c1480 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c1ac420 con 0x55a53b7d0980 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c1ab680 con 0x55a53b7c1da0 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa685227740 10 client.0 _ll_register_callbacks cb 0x7fa6740125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa67bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7d8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa67bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7d8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55724/0 (socket says 172.21.15.27:55724) 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa67bfff640 1 -- 172.21.15.27:0/74952700 learned_addr learned my addr 172.21.15.27:0/74952700 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa680ab1640 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7d0980 0x55a53b7ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.410+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 0x55a53b7c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa67bfff640 1 -- 172.21.15.27:0/74952700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 msgr2=0x55a53b7c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa67bfff640 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 0x55a53b7c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa67bfff640 1 -- 172.21.15.27:0/74952700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7d0980 msgr2=0x55a53b7ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa67bfff640 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7d0980 0x55a53b7ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa67bfff640 1 -- 172.21.15.27:0/74952700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53b7caa80 con 0x55a53b7c1480 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa67bfff640 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7d8e20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0c4640 tx=0x7fa66c0bbf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a5842ade7cabe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa65b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa65b7fe640 1 -- 172.21.15.27:0/74952700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c035040 con 0x55a53b7c1480 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa65b7fe640 1 -- 172.21.15.27:0/74952700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa66c0b8570 con 0x55a53b7c1480 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa685227740 1 -- 172.21.15.27:0/74952700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa66c0bc600 con 0x55a53b7c1480 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa65b7fe640 1 -- 172.21.15.27:0/74952700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa66c0c02b0 con 0x55a53b7c1480 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa685227740 10 client.15810 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa685227740 1 -- 172.21.15.27:0/74952700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a53b846dd0 con 0x55a53b7c1480 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa65b7fe640 1 -- 172.21.15.27:0/74952700 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa66c0bc600 con 0x55a53b7c1480 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa65b7fe640 1 -- 172.21.15.27:0/74952700 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa66c0ba4c0 con 0x55a53b7c1480 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa685227740 10 client.15810 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa685227740 10 client.15810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.411+0000 7fa685227740 20 client.15810 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa685227740 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55a53b8472b0 0x55a53b8496a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa685227740 4 client.15810 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa685227740 1 -- 172.21.15.27:0/74952700 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a53b72df00 con 0x55a53b8472b0 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55a53b8472b0 0x55a53b8496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa67b7fe640 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55a53b8472b0 0x55a53b8496a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=973190ac915e7dfe server_cookie=df92ad2d2f07cd79 in_seq=0 out_seq=0 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa65b7fe640 10 client.15810 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa65b7fe640 1 -- 172.21.15.27:0/74952700 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a53b72df00 con 0x55a53b8472b0 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa65b7fe640 10 client.15810 handle_command_reply: tid=0 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.412+0000 7fa685227740 1 client.15810 shutdown 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 -- 172.21.15.27:0/74952700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 msgr2=0x55a53b7d8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7d8e20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0c4640 tx=0x7fa66c0bbf00 comp rx=0 tx=0).stop 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 -- 172.21.15.27:0/74952700 shutdown_connections 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55a53b8472b0 0x55a53b8496a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a53b7d0980 0x55a53b7ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a53b7c1da0 0x55a53b7c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 --2- 172.21.15.27:0/74952700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a53b7c1480 0x55a53b7d8e20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa65b7fe640 0 client.15810 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:26:04.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 -- 172.21.15.27:0/74952700 >> 172.21.15.27:0/74952700 conn(0x55a53b7bd8a0 msgr2=0x55a53b7c3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 -- 172.21.15.27:0/74952700 shutdown_connections 2024-04-25T01:26:04.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 1 -- 172.21.15.27:0/74952700 wait complete. 2024-04-25T01:26:04.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa685227740 20 client.15810 trim_cache size 0 max 16384 2024-04-25T01:26:04.399 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:26:04.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa682514640 1 -- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6640a2f30 msgr2=0x7fa6640a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa682514640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6640a2f30 0x7fa6640a53f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6740098b0 tx=0x7fa674009d90 comp rx=0 tx=0).stop 2024-04-25T01:26:04.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa682514640 1 -- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 msgr2=0x7fa67c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.413+0000 7fa682514640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 0x7fa67c198000 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa66c006010 tx=0x7fa66c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.414+0000 7fa682514640 1 -- 172.21.15.27:0/2278150445 shutdown_connections 2024-04-25T01:26:04.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.414+0000 7fa682514640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa67c10bd00 0x7fa67c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.414+0000 7fa682514640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa67c1032e0 0x7fa67c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.414+0000 7fa682514640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6640a2f30 0x7fa6640a53f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.414+0000 7fa682514640 1 --2- 172.21.15.27:0/2278150445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67c1029c0 0x7fa67c198000 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.414+0000 7fa682514640 1 -- 172.21.15.27:0/2278150445 >> 172.21.15.27:0/2278150445 conn(0x7fa67c077740 msgr2=0x7fa67c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.414+0000 7fa682514640 1 -- 172.21.15.27:0/2278150445 shutdown_connections 2024-04-25T01:26:04.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.414+0000 7fa682514640 1 -- 172.21.15.27:0/2278150445 wait complete. 2024-04-25T01:26:04.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:26:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.530+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3276306429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 msgr2=0x7fe6c010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.530+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3276306429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 0x7fe6c010d4e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009cd0 tx=0x7fe6b802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3276306429 shutdown_connections 2024-04-25T01:26:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3276306429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 0x7fe6c010d4e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3276306429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6c0100610 0x7fe6c0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3276306429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6c01066b0 0x7fe6c0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3276306429 >> 172.21.15.27:0/3276306429 conn(0x7fe6c0077740 msgr2=0x7fe6c0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3276306429 shutdown_connections 2024-04-25T01:26:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3276306429 wait complete. 2024-04-25T01:26:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 Processor -- start 2024-04-25T01:26:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.531+0000 7fe6c4e31640 1 -- start start 2024-04-25T01:26:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6c4e31640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6c0100610 0x7fe6c0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6c4e31640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 0x7fe6c01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6c4e31640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6c01066b0 0x7fe6c019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6c4e31640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c0110a30 con 0x7fe6c0100fd0 2024-04-25T01:26:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6c4e31640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c01108b0 con 0x7fe6c01066b0 2024-04-25T01:26:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6c4e31640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c0110bb0 con 0x7fe6c0100610 2024-04-25T01:26:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 0x7fe6c01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 0x7fe6c01984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55742/0 (socket says 172.21.15.27:55742) 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/3875174197 learned_addr learned my addr 172.21.15.27:0/3875174197 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6be59c640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6c0100610 0x7fe6c0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6c01066b0 0x7fe6c019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/3875174197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6c0100610 msgr2=0x7fe6c0197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6c0100610 0x7fe6c0197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/3875174197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6c01066b0 msgr2=0x7fe6c019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6c01066b0 0x7fe6c019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/3875174197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c019cf60 con 0x7fe6c0100fd0 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6c01066b0 0x7fe6c019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 0x7fe6c01984b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400e9f0 tx=0x7fe6b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6af7fe640 1 -- 172.21.15.27:0/3875174197 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b400cd80 con 0x7fe6c0100fd0 2024-04-25T01:26:04.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.532+0000 7fe6af7fe640 1 -- 172.21.15.27:0/3875174197 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b4005500 con 0x7fe6c0100fd0 2024-04-25T01:26:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.533+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3875174197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c019d250 con 0x7fe6c0100fd0 2024-04-25T01:26:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.533+0000 7fe6af7fe640 1 -- 172.21.15.27:0/3875174197 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4010690 con 0x7fe6c0100fd0 2024-04-25T01:26:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.533+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3875174197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6c01a4ae0 con 0x7fe6c0100fd0 2024-04-25T01:26:04.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6af7fe640 1 -- 172.21.15.27:0/3875174197 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7fe6b40046c0 con 0x7fe6c0100fd0 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6c7c1a740 1 Processor -- start 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6c7c1a740 1 -- start start 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889da960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d388932670 con 0x55d3889d8570 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d3888da970 con 0x55d3889cd580 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d38892e630 con 0x55d3889cdea0 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.536+0000 7fe6af7fe640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6a80a2eb0 0x7fe6a80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6a80a2eb0 0x7fe6a80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6af7fe640 1 -- 172.21.15.27:0/3875174197 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe6b40cff90 con 0x7fe6c0100fd0 2024-04-25T01:26:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889da960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:35022/0 (socket says 172.21.15.27:35022) 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889da960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55758/0 (socket says 172.21.15.27:55758) 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 -- 172.21.15.27:0/1965961692 learned_addr learned my addr 172.21.15.27:0/1965961692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 -- 172.21.15.27:0/1965961692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 msgr2=0x55d3889cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 -- 172.21.15.27:0/1965961692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 msgr2=0x55d3889da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889da960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 -- 172.21.15.27:0/1965961692 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3888fe400 con 0x55d3889cdea0 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889da960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6a80a2eb0 0x7fe6a80a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8031860 tx=0x7fe6b8009910 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6be59c640 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d8030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000bcb0 tx=0x7fe6b00107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2821b0b2d729178f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.537+0000 7fe6acff9640 1 -- 172.21.15.27:0/1965961692 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0004020 con 0x55d3889cdea0 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6acff9640 1 -- 172.21.15.27:0/1965961692 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b0005500 con 0x55d3889cdea0 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6acff9640 1 -- 172.21.15.27:0/1965961692 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0004ce0 con 0x55d3889cdea0 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/1965961692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 msgr2=0x55d3889d8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d8030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000bcb0 tx=0x7fe6b00107a0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/1965961692 shutdown_connections 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889da960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d8030 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/1965961692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/1965961692 >> 172.21.15.27:0/1965961692 conn(0x55d3889c98a0 msgr2=0x55d3889c9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/1965961692 shutdown_connections 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.538+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/1965961692 wait complete. 2024-04-25T01:26:04.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 1 -- start start 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 1 Processor -- start 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889d7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889d56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d3889db360 con 0x55d3889d8570 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d3888ee050 con 0x55d3889cd580 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d3889db4e0 con 0x55d3889cdea0 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6c7c1a740 10 client.0 _ll_register_callbacks cb 0x55d388645be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6bed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889d56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6bed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889d56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55770/0 (socket says 172.21.15.27:55770) 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.539+0000 7fe6bed9d640 1 -- 172.21.15.27:0/800308849 learned_addr learned my addr 172.21.15.27:0/800308849 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6be59c640 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889d7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6bed9d640 1 -- 172.21.15.27:0/800308849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 msgr2=0x55d3889d5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6bed9d640 1 -- 172.21.15.27:0/800308849 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 msgr2=0x55d3889d7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889d7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6bed9d640 1 -- 172.21.15.27:0/800308849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3888bfdc0 con 0x55d3889d8570 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889d56a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8035000 tx=0x7fe6b8046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfff962bc9bcbeda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/800308849 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8050030 con 0x55d3889d8570 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/800308849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6b0005500 con 0x55d3889d8570 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/800308849 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe6b8004250 con 0x55d3889d8570 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/800308849 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe6b8031d10 con 0x55d3889d8570 2024-04-25T01:26:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6c7c1a740 10 client.15822 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.540+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/800308849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d3886463f0 con 0x55d3889d8570 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/800308849 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b0005500 con 0x55d3889d8570 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/800308849 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe6b8030070 con 0x55d3889d8570 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6c7c1a740 10 client.15822 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6c7c1a740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6c7c1a740 20 client.15822 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d3889e8550 0x55d3889ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6c7c1a740 4 client.15822 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/800308849 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d38893fbb0 con 0x55d3889e8550 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6be59c640 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d3889e8550 0x55d3889ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6be59c640 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d3889e8550 0x55d3889ea9a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19c96c0f06854570 server_cookie=d1610702aca5b57b in_seq=0 out_seq=0 2024-04-25T01:26:04.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.541+0000 7fe6a5ffb640 10 client.15822 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:04.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:04 smithi153 ceph-mon[32621]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-25T01:26:04.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:04 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2886400858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/800308849 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d38893fbb0 con 0x55d3889e8550 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6a5ffb640 10 client.15822 handle_command_reply: tid=0 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 client.15822 shutdown 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/800308849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 msgr2=0x55d3889d56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889d56a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8035000 tx=0x7fe6b8046000 comp rx=0 tx=0).stop 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/800308849 shutdown_connections 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d3889e8550 0x55d3889ea9a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889d8570 0x55d3889d56a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cdea0 0x55d3889d5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6a5ffb640 0 client.15822 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/800308849 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889d7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.542+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/800308849 >> 172.21.15.27:0/800308849 conn(0x55d3889c98a0 msgr2=0x55d3889c8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.543+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/800308849 shutdown_connections 2024-04-25T01:26:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.543+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/800308849 wait complete. 2024-04-25T01:26:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.543+0000 7fe6c7c1a740 20 client.15822 trim_cache size 0 max 16384 2024-04-25T01:26:04.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:04 smithi139 ceph-mon[32760]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-25T01:26:04.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:04 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2886400858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6c7c1a740 1 Processor -- start 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6c7c1a740 1 -- start start 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cd580 0x55d3889e4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889f0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889d8570 0x55d3889f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c01b1450 con 0x55d3889cdea0 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c01aed20 con 0x55d3889d8570 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c01acb70 con 0x55d3889cd580 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889f0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55778/0 (socket says 172.21.15.27:55778) 2024-04-25T01:26:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/318102415 learned_addr learned my addr 172.21.15.27:0/318102415 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889d8570 0x55d3889f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6be59c640 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cd580 0x55d3889e4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/318102415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cd580 msgr2=0x55d3889e4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cd580 0x55d3889e4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/318102415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889d8570 msgr2=0x55d3889f07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889d8570 0x55d3889f07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/318102415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3887107d0 con 0x55d3889cdea0 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.557+0000 7fe6be59c640 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cd580 0x55d3889e4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.558+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889f0270 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40cd200 tx=0x7fe6b40056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5082ec8944e4156e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.558+0000 7fe6a7fff640 1 -- 172.21.15.27:0/318102415 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b409bee0 con 0x55d3889cdea0 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.558+0000 7fe6a7fff640 1 -- 172.21.15.27:0/318102415 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b40b7020 con 0x55d3889cdea0 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.558+0000 7fe6a7fff640 1 -- 172.21.15.27:0/318102415 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40a4920 con 0x55d3889cdea0 2024-04-25T01:26:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/318102415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 msgr2=0x55d3889f0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889f0270 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40cd200 tx=0x7fe6b40056b0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/318102415 shutdown_connections 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889d8570 0x55d3889f07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889f0270 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/318102415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889cd580 0x55d3889e4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/318102415 >> 172.21.15.27:0/318102415 conn(0x55d3889c98a0 msgr2=0x55d3889c82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/318102415 shutdown_connections 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/318102415 wait complete. 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- start start 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 Processor -- start 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889e03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889e0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889d8570 0x55d3889dd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c01ad1e0 con 0x55d3889cdea0 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c01ae650 con 0x55d3889cd580 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c01ad8b0 con 0x55d3889d8570 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6c7c1a740 10 client.0 _ll_register_callbacks cb 0x7fe6b8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889e0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6bdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889e0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55794/0 (socket says 172.21.15.27:55794) 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.559+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/2574013981 learned_addr learned my addr 172.21.15.27:0/2574013981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6be59c640 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889e03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889d8570 0x55d3889dd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/2574013981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889d8570 msgr2=0x55d3889dd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889d8570 0x55d3889dd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/2574013981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 msgr2=0x55d3889e03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889e03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6bdd9b640 1 -- 172.21.15.27:0/2574013981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3889ddf70 con 0x55d3889cdea0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6bed9d640 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889d8570 0x55d3889dd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6bdd9b640 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889e0910 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40988b0 tx=0x7fe6b40b92d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4ce30ae46bb7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/2574013981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40b4070 con 0x55d3889cdea0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/2574013981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe6b40a48d0 con 0x55d3889cdea0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/2574013981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6b40b7020 con 0x55d3889cdea0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/2574013981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe6b40a4b50 con 0x55d3889cdea0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6c7c1a740 10 client.15828 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/2574013981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d388a4e4d0 con 0x55d3889cdea0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.560+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/2574013981 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b40b7020 con 0x55d3889cdea0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.561+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/2574013981 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe6b4097c30 con 0x55d3889cdea0 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.561+0000 7fe6c7c1a740 10 client.15828 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.561+0000 7fe6c7c1a740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.561+0000 7fe6c7c1a740 20 client.15828 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.561+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d388a4eb60 0x55d388a50f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.561+0000 7fe6c7c1a740 4 client.15828 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.561+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/2574013981 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3889d5a30 con 0x55d388a4eb60 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.561+0000 7fe6be59c640 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d388a4eb60 0x55d388a50f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6be59c640 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d388a4eb60 0x55d388a50f80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dce0c0d2dbbdba2 server_cookie=2089b816e1035ead in_seq=0 out_seq=0 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6a5ffb640 10 client.15828 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6a5ffb640 1 -- 172.21.15.27:0/2574013981 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3889d5a30 con 0x55d388a4eb60 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6a5ffb640 10 client.15828 handle_command_reply: tid=0 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 client.15828 shutdown 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/2574013981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 msgr2=0x55d3889e0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889e0910 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40988b0 tx=0x7fe6b40b92d0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/2574013981 shutdown_connections 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d388a4eb60 0x55d388a50f80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d3889d8570 0x55d3889dd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d3889cdea0 0x55d3889e0910 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6a5ffb640 0 client.15828 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 --2- 172.21.15.27:0/2574013981 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d3889cd580 0x55d3889e03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/2574013981 >> 172.21.15.27:0/2574013981 conn(0x55d3889c98a0 msgr2=0x55d3889c82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/2574013981 shutdown_connections 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 1 -- 172.21.15.27:0/2574013981 wait complete. 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.562+0000 7fe6c7c1a740 20 client.15828 trim_cache size 0 max 16384 2024-04-25T01:26:04.549 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:26:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6a80a2eb0 msgr2=0x7fe6a80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6a80a2eb0 0x7fe6a80a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8031860 tx=0x7fe6b8009910 comp rx=0 tx=0).stop 2024-04-25T01:26:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 msgr2=0x7fe6c01984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 0x7fe6c01984b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400e9f0 tx=0x7fe6b400edf0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3875174197 shutdown_connections 2024-04-25T01:26:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6c01066b0 0x7fe6c019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6a80a2eb0 0x7fe6a80a5370 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c0100fd0 0x7fe6c01984b0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 --2- 172.21.15.27:0/3875174197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6c0100610 0x7fe6c0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3875174197 >> 172.21.15.27:0/3875174197 conn(0x7fe6c0077740 msgr2=0x7fe6c00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3875174197 shutdown_connections 2024-04-25T01:26:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:04.563+0000 7fe6c4e31640 1 -- 172.21.15.27:0/3875174197 wait complete. 2024-04-25T01:26:05.554 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:26:05.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:26:05.555 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 -- 172.21.15.27:0/3607136475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c81020e0 msgr2=0x7f53c810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 --2- 172.21.15.27:0/3607136475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c81020e0 0x7f53c810e5f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009cd0 tx=0x7f53c002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 -- 172.21.15.27:0/3607136475 shutdown_connections 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 --2- 172.21.15.27:0/3607136475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c81020e0 0x7f53c810e5f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 --2- 172.21.15.27:0/3607136475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8101720 0x7f53c8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 --2- 172.21.15.27:0/3607136475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c8107730 0x7f53c8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 -- 172.21.15.27:0/3607136475 >> 172.21.15.27:0/3607136475 conn(0x7f53c8078b40 msgr2=0x7f53c8078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 -- 172.21.15.27:0/3607136475 shutdown_connections 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.684+0000 7f53ce361640 1 -- 172.21.15.27:0/3607136475 wait complete. 2024-04-25T01:26:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53ce361640 1 Processor -- start 2024-04-25T01:26:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53ce361640 1 -- start start 2024-04-25T01:26:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53ce361640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8101720 0x7f53c8073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53ce361640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c81020e0 0x7f53c8073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53ce361640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8107730 0x7f53c80744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53ce361640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8111b40 con 0x7f53c8101720 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53ce361640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81119c0 con 0x7f53c81020e0 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53ce361640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8111cc0 con 0x7f53c8107730 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53c7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8101720 0x7f53c8073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53c7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8101720 0x7f53c8073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55820/0 (socket says 172.21.15.27:55820) 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.685+0000 7f53c7fff640 1 -- 172.21.15.27:0/749447834 learned_addr learned my addr 172.21.15.27:0/749447834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c77fe640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c81020e0 0x7f53c8073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8107730 0x7f53c80744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c7fff640 1 -- 172.21.15.27:0/749447834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8107730 msgr2=0x7f53c80744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c7fff640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8107730 0x7f53c80744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c7fff640 1 -- 172.21.15.27:0/749447834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c81020e0 msgr2=0x7f53c8073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c7fff640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c81020e0 0x7f53c8073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c7fff640 1 -- 172.21.15.27:0/749447834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c81a6420 con 0x7f53c8101720 2024-04-25T01:26:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c77fe640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c81020e0 0x7f53c8073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c7fff640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8101720 0x7f53c8073a40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f53b800e9f0 tx=0x7f53b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c57fa640 1 -- 172.21.15.27:0/749447834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b800cd80 con 0x7f53c8101720 2024-04-25T01:26:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53ce361640 1 -- 172.21.15.27:0/749447834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c81a6710 con 0x7f53c8101720 2024-04-25T01:26:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c57fa640 1 -- 172.21.15.27:0/749447834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b8010910 con 0x7f53c8101720 2024-04-25T01:26:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53c57fa640 1 -- 172.21.15.27:0/749447834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b80188f0 con 0x7f53c8101720 2024-04-25T01:26:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.686+0000 7f53ce361640 1 -- 172.21.15.27:0/749447834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c81a6cb0 con 0x7f53c8101720 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.688+0000 7f53c57fa640 1 -- 172.21.15.27:0/749447834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7f53b8010ac0 con 0x7f53c8101720 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.689+0000 7f53c57fa640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2eb0 0x7f53b00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.689+0000 7f53c57fa640 1 -- 172.21.15.27:0/749447834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f53b80cff30 con 0x7f53c8101720 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.689+0000 7f53c77fe640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2eb0 0x7f53b00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.689+0000 7f53c77fe640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2eb0 0x7f53b00a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f53c80754d0 tx=0x7f53bc005d90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.689+0000 7f53d106f740 1 Processor -- start 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53d106f740 1 -- start start 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53d106f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e83980 0x564d75e81cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53d106f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e82290 0x564d75e82710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53d106f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e8d010 0x564d75e8f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53d106f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564d75de62c0 con 0x564d75e82290 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53d106f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564d75d8e910 con 0x564d75e83980 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53d106f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564d75de2280 con 0x564d75e8d010 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e82290 0x564d75e82710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e82290 0x564d75e82710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55836/0 (socket says 172.21.15.27:55836) 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c77fe640 1 -- 172.21.15.27:0/3616417780 learned_addr learned my addr 172.21.15.27:0/3616417780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c7fff640 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e83980 0x564d75e81cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e8d010 0x564d75e8f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c77fe640 1 -- 172.21.15.27:0/3616417780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e8d010 msgr2=0x564d75e8f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c77fe640 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e8d010 0x564d75e8f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c77fe640 1 -- 172.21.15.27:0/3616417780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e83980 msgr2=0x564d75e81cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c77fe640 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e83980 0x564d75e81cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c77fe640 1 -- 172.21.15.27:0/3616417780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d75d9ffb0 con 0x564d75e82290 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e8d010 0x564d75e8f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.690+0000 7f53c7fff640 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e83980 0x564d75e81cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.691+0000 7f53c77fe640 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e82290 0x564d75e82710 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00de10 tx=0x7f53bc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84191b418836b87a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.691+0000 7f53ae7fc640 1 -- 172.21.15.27:0/3616417780 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc020070 con 0x564d75e82290 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.691+0000 7f53ae7fc640 1 -- 172.21.15.27:0/3616417780 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53bc00fe30 con 0x564d75e82290 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.691+0000 7f53ae7fc640 1 -- 172.21.15.27:0/3616417780 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc01bcb0 con 0x564d75e82290 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.691+0000 7f53d106f740 1 -- 172.21.15.27:0/3616417780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e82290 msgr2=0x564d75e82710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.691+0000 7f53d106f740 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e82290 0x564d75e82710 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00de10 tx=0x7f53bc010000 comp rx=0 tx=0).stop 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 -- 172.21.15.27:0/3616417780 shutdown_connections 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e8d010 0x564d75e8f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e82290 0x564d75e82710 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 --2- 172.21.15.27:0/3616417780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e83980 0x564d75e81cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 -- 172.21.15.27:0/3616417780 >> 172.21.15.27:0/3616417780 conn(0x564d75e4a950 msgr2=0x564d75e7e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 -- 172.21.15.27:0/3616417780 shutdown_connections 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 -- 172.21.15.27:0/3616417780 wait complete. 2024-04-25T01:26:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 -- start start 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 Processor -- start 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e89f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e8a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564d75e8fea0 con 0x564d75e83980 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564d75da1e90 con 0x564d75e8d010 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564d75e90020 con 0x564d75e82290 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.692+0000 7f53d106f740 10 client.0 _ll_register_callbacks cb 0x564d75ddd970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55838/0 (socket says 172.21.15.27:55838) 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 -- 172.21.15.27:0/1249686930 learned_addr learned my addr 172.21.15.27:0/1249686930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e8a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c7fff640 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e89f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 -- 172.21.15.27:0/1249686930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 msgr2=0x564d75e89f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e89f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 -- 172.21.15.27:0/1249686930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 msgr2=0x564d75e8a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e8a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 -- 172.21.15.27:0/1249686930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d75bc27e0 con 0x564d75e83980 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53c77fe640 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00dde0 tx=0x7f53bc01bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a6e3b412c49f92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53a77fe640 1 -- 172.21.15.27:0/1249686930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc034060 con 0x564d75e83980 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53a77fe640 1 -- 172.21.15.27:0/1249686930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f53bc01c870 con 0x564d75e83980 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.693+0000 7f53d106f740 1 -- 172.21.15.27:0/1249686930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53bc00fe30 con 0x564d75e83980 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53a77fe640 1 -- 172.21.15.27:0/1249686930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f53bc03b050 con 0x564d75e83980 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53d106f740 10 client.15840 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53d106f740 1 -- 172.21.15.27:0/1249686930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d75c22f70 con 0x564d75e83980 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53a77fe640 1 -- 172.21.15.27:0/1249686930 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53bc00fe30 con 0x564d75e83980 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53a77fe640 1 -- 172.21.15.27:0/1249686930 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f53bc025ac0 con 0x564d75e83980 2024-04-25T01:26:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53d106f740 10 client.15840 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53d106f740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53d106f740 20 client.15840 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53d106f740 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564d75e9cff0 0x564d75e9f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53d106f740 4 client.15840 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53d106f740 1 -- 172.21.15.27:0/1249686930 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d75dba5a0 con 0x564d75e9cff0 2024-04-25T01:26:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.694+0000 7f53c7fff640 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564d75e9cff0 0x564d75e9f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.695+0000 7f53c7fff640 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564d75e9cff0 0x564d75e9f3e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae77a73ca92d0d23 server_cookie=445d9597a69b1104 in_seq=0 out_seq=0 2024-04-25T01:26:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.695+0000 7f53a77fe640 10 client.15840 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:05.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53a77fe640 1 -- 172.21.15.27:0/1249686930 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d75dba5a0 con 0x564d75e9cff0 2024-04-25T01:26:05.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53a77fe640 10 client.15840 handle_command_reply: tid=0 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 client.15840 shutdown 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 -- 172.21.15.27:0/1249686930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 msgr2=0x564d75e8a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00dde0 tx=0x7f53bc01bff0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 -- 172.21.15.27:0/1249686930 shutdown_connections 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564d75e9cff0 0x564d75e9f3e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e8a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a480 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53a77fe640 0 client.15840 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 --2- 172.21.15.27:0/1249686930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e89f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 -- 172.21.15.27:0/1249686930 >> 172.21.15.27:0/1249686930 conn(0x564d75e4a950 msgr2=0x564d75e845f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 -- 172.21.15.27:0/1249686930 shutdown_connections 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 1 -- 172.21.15.27:0/1249686930 wait complete. 2024-04-25T01:26:05.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.696+0000 7f53d106f740 20 client.15840 trim_cache size 0 max 16384 2024-04-25T01:26:05.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53d106f740 1 Processor -- start 2024-04-25T01:26:05.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53d106f740 1 -- start start 2024-04-25T01:26:05.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53d106f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e7d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53d106f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53d106f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e84470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53d106f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81b3700 con 0x564d75e83980 2024-04-25T01:26:05.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53d106f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81b0fd0 con 0x564d75e8d010 2024-04-25T01:26:05.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53d106f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81aee20 con 0x564d75e82290 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55842/0 (socket says 172.21.15.27:55842) 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c77fe640 1 -- 172.21.15.27:0/4025160228 learned_addr learned my addr 172.21.15.27:0/4025160228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e84470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c7fff640 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e7d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c77fe640 1 -- 172.21.15.27:0/4025160228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 msgr2=0x564d75e7d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c77fe640 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e7d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c77fe640 1 -- 172.21.15.27:0/4025160228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 msgr2=0x564d75e84470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c77fe640 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e84470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c77fe640 1 -- 172.21.15.27:0/4025160228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d75e7ea00 con 0x564d75e83980 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e84470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.714+0000 7f53c7fff640 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e7d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53c77fe640 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a7e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00ddb0 tx=0x7f53bc01b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe87a9217ec6df67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53adffb640 1 -- 172.21.15.27:0/4025160228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc013070 con 0x564d75e83980 2024-04-25T01:26:05.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53adffb640 1 -- 172.21.15.27:0/4025160228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53bc02d540 con 0x564d75e83980 2024-04-25T01:26:05.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53adffb640 1 -- 172.21.15.27:0/4025160228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc034040 con 0x564d75e83980 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53d106f740 1 -- 172.21.15.27:0/4025160228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 msgr2=0x564d75e8a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53d106f740 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a7e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00ddb0 tx=0x7f53bc01b6d0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53d106f740 1 -- 172.21.15.27:0/4025160228 shutdown_connections 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53d106f740 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e8d010 0x564d75e84470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53d106f740 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e83980 0x564d75e8a7e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53d106f740 1 --2- 172.21.15.27:0/4025160228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e82290 0x564d75e7d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.715+0000 7f53d106f740 1 -- 172.21.15.27:0/4025160228 >> 172.21.15.27:0/4025160228 conn(0x564d75e4a950 msgr2=0x564d75e83d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 -- 172.21.15.27:0/4025160228 shutdown_connections 2024-04-25T01:26:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 -- 172.21.15.27:0/4025160228 wait complete. 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 -- start start 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 Processor -- start 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e82290 0x564d75e95540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e83980 0x564d75e922b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e8d010 0x564d75e927f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81af490 con 0x564d75e8d010 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81b0900 con 0x564d75e82290 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81afb60 con 0x564d75e83980 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.716+0000 7f53d106f740 10 client.0 _ll_register_callbacks cb 0x7f53bc00dce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e8d010 0x564d75e927f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e8d010 0x564d75e927f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55856/0 (socket says 172.21.15.27:55856) 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 -- 172.21.15.27:0/4122590570 learned_addr learned my addr 172.21.15.27:0/4122590570 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53c77fe640 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e83980 0x564d75e922b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53c7fff640 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e82290 0x564d75e95540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 -- 172.21.15.27:0/4122590570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e83980 msgr2=0x564d75e922b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e83980 0x564d75e922b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 -- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e82290 msgr2=0x564d75e95540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e82290 0x564d75e95540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 -- 172.21.15.27:0/4122590570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d75aba170 con 0x564d75e8d010 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53c7fff640 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e82290 0x564d75e95540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53cc8fe640 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e8d010 0x564d75e927f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009ca0 tx=0x7f53c0031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddc5b06844a6980 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53a77fe640 1 -- 172.21.15.27:0/4122590570 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0003b70 con 0x564d75e8d010 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53d106f740 1 -- 172.21.15.27:0/4122590570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53bc02d540 con 0x564d75e8d010 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53a77fe640 1 -- 172.21.15.27:0/4122590570 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f53c0003d10 con 0x564d75e8d010 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53a77fe640 1 -- 172.21.15.27:0/4122590570 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f53c0054050 con 0x564d75e8d010 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.717+0000 7f53d106f740 10 client.15846 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53d106f740 1 -- 172.21.15.27:0/4122590570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d75f03f50 con 0x564d75e8d010 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53a77fe640 1 -- 172.21.15.27:0/4122590570 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53bc02d540 con 0x564d75e8d010 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53a77fe640 1 -- 172.21.15.27:0/4122590570 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f53c0030070 con 0x564d75e8d010 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53d106f740 10 client.15846 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53d106f740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53d106f740 20 client.15846 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53d106f740 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564d75f04720 0x564d75f06b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53d106f740 4 client.15846 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53d106f740 1 -- 172.21.15.27:0/4122590570 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564d75df3800 con 0x564d75f04720 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.718+0000 7f53c7fff640 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564d75f04720 0x564d75f06b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.719+0000 7f53c7fff640 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564d75f04720 0x564d75f06b40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abbb3c72741192f server_cookie=2a4477c2411392bb in_seq=0 out_seq=0 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.719+0000 7f53a77fe640 10 client.15846 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53a77fe640 1 -- 172.21.15.27:0/4122590570 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564d75df3800 con 0x564d75f04720 2024-04-25T01:26:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53a77fe640 10 client.15846 handle_command_reply: tid=0 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 client.15846 shutdown 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 -- 172.21.15.27:0/4122590570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e8d010 msgr2=0x564d75e927f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e8d010 0x564d75e927f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009ca0 tx=0x7f53c0031af0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 -- 172.21.15.27:0/4122590570 shutdown_connections 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564d75f04720 0x564d75f06b40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d75e8d010 0x564d75e927f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564d75e83980 0x564d75e922b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 --2- 172.21.15.27:0/4122590570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564d75e82290 0x564d75e95540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 -- 172.21.15.27:0/4122590570 >> 172.21.15.27:0/4122590570 conn(0x564d75e4a950 msgr2=0x564d75e845f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53a77fe640 0 client.15846 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 -- 172.21.15.27:0/4122590570 shutdown_connections 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 1 -- 172.21.15.27:0/4122590570 wait complete. 2024-04-25T01:26:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.720+0000 7f53d106f740 20 client.15846 trim_cache size 0 max 16384 2024-04-25T01:26:05.706 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:26:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:26:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "c1da276a-3301-44b9-814e-bff3319fa7ce", 2024-04-25T01:26:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:26:05.706 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 -- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2eb0 msgr2=0x7f53b00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2eb0 0x7f53b00a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f53c80754d0 tx=0x7f53bc005d90 comp rx=0 tx=0).stop 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 -- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8101720 msgr2=0x7f53c8073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8101720 0x7f53c8073a40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f53b800e9f0 tx=0x7f53b800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 -- 172.21.15.27:0/749447834 shutdown_connections 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c8107730 0x7f53c80744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c81020e0 0x7f53c8073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2eb0 0x7f53b00a5370 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 --2- 172.21.15.27:0/749447834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8101720 0x7f53c8073a40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 -- 172.21.15.27:0/749447834 >> 172.21.15.27:0/749447834 conn(0x7f53c8078b40 msgr2=0x7f53c8104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 -- 172.21.15.27:0/749447834 shutdown_connections 2024-04-25T01:26:05.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.721+0000 7f53ce361640 1 -- 172.21.15.27:0/749447834 wait complete. 2024-04-25T01:26:05.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:26:05.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:26:05.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/785034884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 msgr2=0x7f7f94102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/785034884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 0x7f7f94102da0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7f90009900 tx=0x7f7f9002a510 comp rx=0 tx=0).stop 2024-04-25T01:26:05.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/785034884 shutdown_connections 2024-04-25T01:26:05.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/785034884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f9410bd00 0x7f7f9410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/785034884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f941032e0 0x7f7f9410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/785034884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 0x7f7f94102da0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/785034884 >> 172.21.15.27:0/785034884 conn(0x7f7f94077740 msgr2=0x7f7f94077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/785034884 shutdown_connections 2024-04-25T01:26:05.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.840+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/785034884 wait complete. 2024-04-25T01:26:05.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.841+0000 7f7f9b1ce640 1 Processor -- start 2024-04-25T01:26:05.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.841+0000 7f7f9b1ce640 1 -- start start 2024-04-25T01:26:05.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.841+0000 7f7f9b1ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 0x7f7f94197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.841+0000 7f7f9b1ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f941032e0 0x7f7f94198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.841+0000 7f7f9b1ce640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f9410bd00 0x7f7f9419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.841+0000 7f7f9b1ce640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f94111b00 con 0x7f7f941029c0 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.841+0000 7f7f9b1ce640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f94111980 con 0x7f7f9410bd00 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.841+0000 7f7f9b1ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f94111c80 con 0x7f7f941032e0 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 0x7f7f94197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 0x7f7f94197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55870/0 (socket says 172.21.15.27:55870) 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 -- 172.21.15.27:0/1510873321 learned_addr learned my addr 172.21.15.27:0/1510873321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f941032e0 0x7f7f94198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f9976b640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f9410bd00 0x7f7f9419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 -- 172.21.15.27:0/1510873321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f941032e0 msgr2=0x7f7f94198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f941032e0 0x7f7f94198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 -- 172.21.15.27:0/1510873321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f9410bd00 msgr2=0x7f7f9419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f9410bd00 0x7f7f9419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 -- 172.21.15.27:0/1510873321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9419cfd0 con 0x7f7f941029c0 2024-04-25T01:26:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 0x7f7f94197fe0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7f90006010 tx=0x7f7f900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f89ffb640 1 -- 172.21.15.27:0/1510873321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9003f030 con 0x7f7f941029c0 2024-04-25T01:26:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f89ffb640 1 -- 172.21.15.27:0/1510873321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f90003ce0 con 0x7f7f941029c0 2024-04-25T01:26:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/1510873321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9419d260 con 0x7f7f941029c0 2024-04-25T01:26:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.842+0000 7f7f89ffb640 1 -- 172.21.15.27:0/1510873321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90035070 con 0x7f7f941029c0 2024-04-25T01:26:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.843+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/1510873321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f941a4b50 con 0x7f7f941029c0 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.845+0000 7f7f89ffb640 1 -- 172.21.15.27:0/1510873321 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7f7f90042030 con 0x7f7f941029c0 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.845+0000 7f7f9dfba740 1 Processor -- start 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.845+0000 7f7f9dfba740 1 -- start start 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9aea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9bba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fd1f9136b0 con 0x55fd1f9aef50 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fd1f8bb880 con 0x55fd1f9b9620 2024-04-25T01:26:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd1f90f670 con 0x55fd1f9ae630 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55872/0 (socket says 172.21.15.27:55872) 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 -- 172.21.15.27:0/1059178809 learned_addr learned my addr 172.21.15.27:0/1059178809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9976b640 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f89ffb640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f7c0a2f30 0x7f7f7c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9bba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9976b640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f7c0a2f30 0x7f7f7c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 -- 172.21.15.27:0/1059178809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 msgr2=0x55fd1f9aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 -- 172.21.15.27:0/1059178809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 msgr2=0x55fd1f9bba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9bba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f89ffb640 1 -- 172.21.15.27:0/1510873321 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7f900effa0 con 0x7f7f941029c0 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 -- 172.21.15.27:0/1059178809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd1f6f17d0 con 0x55fd1f9aef50 2024-04-25T01:26:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f9976b640 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.846+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b90e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7f900ed260 tx=0x7f7f9002a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa21cf251ac9e40d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f9976b640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f7c0a2f30 0x7f7f7c0a53f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c005c40 tx=0x7f7f8c004ab0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f7affd640 1 -- 172.21.15.27:0/1059178809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90049070 con 0x55fd1f9aef50 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f7affd640 1 -- 172.21.15.27:0/1059178809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f900bc6b0 con 0x55fd1f9aef50 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f7affd640 1 -- 172.21.15.27:0/1059178809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900c4c20 con 0x55fd1f9aef50 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f9dfba740 1 -- 172.21.15.27:0/1059178809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 msgr2=0x55fd1f9b90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b90e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7f900ed260 tx=0x7f7f9002a510 comp rx=0 tx=0).stop 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f9dfba740 1 -- 172.21.15.27:0/1059178809 shutdown_connections 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9bba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b90e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.847+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/1059178809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 -- 172.21.15.27:0/1059178809 >> 172.21.15.27:0/1059178809 conn(0x55fd1f9aa950 msgr2=0x55fd1f9aad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 -- 172.21.15.27:0/1059178809 shutdown_connections 2024-04-25T01:26:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 -- 172.21.15.27:0/1059178809 wait complete. 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 -- start start 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 Processor -- start 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fd1f9136b0 con 0x55fd1f9aef50 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fd1f8cef50 con 0x55fd1f9b9620 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd1f9bc1d0 con 0x55fd1f9ae630 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.848+0000 7f7f9dfba740 10 client.0 _ll_register_callbacks cb 0x55fd1f90a000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55884/0 (socket says 172.21.15.27:55884) 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 -- 172.21.15.27:0/2580279994 learned_addr learned my addr 172.21.15.27:0/2580279994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f9976b640 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 -- 172.21.15.27:0/2580279994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 msgr2=0x55fd1f9b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 -- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 msgr2=0x55fd1f9b6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 -- 172.21.15.27:0/2580279994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd1f6273f0 con 0x55fd1f9aef50 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f9976b640 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b6a40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400bf80 tx=0x7f7f840118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4356e97cf0c33eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f73fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f73fff640 1 -- 172.21.15.27:0/2580279994 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f84004020 con 0x55fd1f9aef50 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f73fff640 1 -- 172.21.15.27:0/2580279994 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7f840041c0 con 0x55fd1f9aef50 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.849+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2580279994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f900bc6b0 con 0x55fd1f9aef50 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f73fff640 1 -- 172.21.15.27:0/2580279994 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7f84024050 con 0x55fd1f9aef50 2024-04-25T01:26:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f9dfba740 10 client.15858 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2580279994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fd1f5e70f0 con 0x55fd1f9aef50 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f73fff640 1 -- 172.21.15.27:0/2580279994 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f900bc6b0 con 0x55fd1f9aef50 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f73fff640 1 -- 172.21.15.27:0/2580279994 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7f84004be0 con 0x55fd1f9aef50 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f9dfba740 10 client.15858 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f9dfba740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f9dfba740 20 client.15858 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fd1f9c95e0 0x55fd1f9cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f9dfba740 4 client.15858 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2580279994 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd1f920bf0 con 0x55fd1f9c95e0 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.850+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fd1f9c95e0 0x55fd1f9cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.851+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fd1f9c95e0 0x55fd1f9cba10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a9b420663ab28cb server_cookie=b0d5890ee2df0dea in_seq=0 out_seq=0 2024-04-25T01:26:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.851+0000 7f7f73fff640 10 client.15858 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.851+0000 7f7f73fff640 1 -- 172.21.15.27:0/2580279994 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd1f920bf0 con 0x55fd1f9c95e0 2024-04-25T01:26:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.851+0000 7f7f73fff640 10 client.15858 handle_command_reply: tid=0 2024-04-25T01:26:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.851+0000 7f7f9dfba740 1 client.15858 shutdown 2024-04-25T01:26:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.851+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2580279994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 msgr2=0x55fd1f9b6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b6a40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400bf80 tx=0x7f7f840118d0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2580279994 shutdown_connections 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fd1f9c95e0 0x55fd1f9cba10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b6a40 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f73fff640 0 client.15858 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2580279994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2580279994 >> 172.21.15.27:0/2580279994 conn(0x55fd1f9aa950 msgr2=0x55fd1f9b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2580279994 shutdown_connections 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2580279994 wait complete. 2024-04-25T01:26:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.852+0000 7f7f9dfba740 20 client.15858 trim_cache size 0 max 16384 2024-04-25T01:26:05.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.870+0000 7f7f9dfba740 1 Processor -- start 2024-04-25T01:26:05.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.870+0000 7f7f9dfba740 1 -- start start 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.870+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9ae630 0x55fd1f9c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.870+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.870+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.870+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941b1530 con 0x55fd1f9ae630 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.870+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941aee00 con 0x55fd1f9b9620 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.870+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941acc50 con 0x55fd1f9aef50 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9ae630 0x55fd1f9c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9ae630 0x55fd1f9c4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55894/0 (socket says 172.21.15.27:55894) 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 -- 172.21.15.27:0/375650361 learned_addr learned my addr 172.21.15.27:0/375650361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f9976b640 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 -- 172.21.15.27:0/375650361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9aef50 msgr2=0x55fd1f9b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 -- 172.21.15.27:0/375650361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 msgr2=0x55fd1f9b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 -- 172.21.15.27:0/375650361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd1f99c090 con 0x55fd1f9ae630 2024-04-25T01:26:05.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9ae630 0x55fd1f9c4f70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7f90000c00 tx=0x7f7f900b8380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dc75a4ffedcbd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f7a7fc640 1 -- 172.21.15.27:0/375650361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90049040 con 0x55fd1f9ae630 2024-04-25T01:26:05.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f7a7fc640 1 -- 172.21.15.27:0/375650361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f900bc6b0 con 0x55fd1f9ae630 2024-04-25T01:26:05.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.871+0000 7f7f7a7fc640 1 -- 172.21.15.27:0/375650361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900c0360 con 0x55fd1f9ae630 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 -- 172.21.15.27:0/375650361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9ae630 msgr2=0x55fd1f9c4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9ae630 0x55fd1f9c4f70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7f90000c00 tx=0x7f7f900b8380 comp rx=0 tx=0).stop 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 -- 172.21.15.27:0/375650361 shutdown_connections 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9b9620 0x55fd1f9b8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/375650361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9ae630 0x55fd1f9c4f70 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 -- 172.21.15.27:0/375650361 >> 172.21.15.27:0/375650361 conn(0x55fd1f9aa950 msgr2=0x55fd1f9b7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 -- 172.21.15.27:0/375650361 shutdown_connections 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.872+0000 7f7f9dfba740 1 -- 172.21.15.27:0/375650361 wait complete. 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f9dfba740 1 -- start start 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f9dfba740 1 Processor -- start 2024-04-25T01:26:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f9dfba740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9b9620 0x55fd1fa2eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941ad2c0 con 0x55fd1f9aef50 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941ae730 con 0x55fd1f9ae630 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f9dfba740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f941ad990 con 0x55fd1f9b9620 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.873+0000 7f7f98f6a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f98f6a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:57596/0 (socket says 172.21.15.27:57596) 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f9976b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9b9620 0x55fd1fa2eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f98f6a640 1 -- 172.21.15.27:0/2441431187 learned_addr learned my addr 172.21.15.27:0/2441431187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f9dfba740 10 client.0 _ll_register_callbacks cb 0x7f7f8400bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f98f6a640 1 -- 172.21.15.27:0/2441431187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9b9620 msgr2=0x55fd1fa2eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9b9620 0x55fd1fa2eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f98f6a640 1 -- 172.21.15.27:0/2441431187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 msgr2=0x55fd1f9b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f98f6a640 1 -- 172.21.15.27:0/2441431187 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd1fa2f0d0 con 0x55fd1f9ae630 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f98f6a640 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b3860 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f900c46f0 tx=0x7f7f900f0220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1cba714db6172c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.874+0000 7f7f73fff640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f73fff640 1 -- 172.21.15.27:0/2441431187 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90049070 con 0x55fd1f9ae630 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f73fff640 1 -- 172.21.15.27:0/2441431187 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7f900c4790 con 0x55fd1f9ae630 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2441431187 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f900bc6b0 con 0x55fd1f9ae630 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f73fff640 1 -- 172.21.15.27:0/2441431187 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7f900bb910 con 0x55fd1f9ae630 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 10 client.25412 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2441431187 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fd1fa30320 con 0x55fd1f9ae630 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f73fff640 1 -- 172.21.15.27:0/2441431187 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f900c0360 con 0x55fd1f9ae630 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f73fff640 1 -- 172.21.15.27:0/2441431187 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7f9003bbb0 con 0x55fd1f9ae630 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 10 client.25412 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 10 client.25412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 20 client.25412 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fd1fa30820 0x55fd1fa32c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 4 client.25412 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:26:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.875+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2441431187 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f7f900bc6b0 con 0x55fd1fa30820 2024-04-25T01:26:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.876+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fd1fa30820 0x55fd1fa32c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.876+0000 7f7f73fff640 10 client.25412 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.876+0000 7f7f8bfff640 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fd1fa30820 0x55fd1fa32c10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28f0923ff061b3cb server_cookie=584550b97c59a343 in_seq=0 out_seq=0 2024-04-25T01:26:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.876+0000 7f7f73fff640 1 -- 172.21.15.27:0/2441431187 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f7f900bc6b0 con 0x55fd1fa30820 2024-04-25T01:26:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.876+0000 7f7f73fff640 10 client.25412 handle_command_reply: tid=0 2024-04-25T01:26:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.876+0000 7f7f9dfba740 1 client.25412 shutdown 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2441431187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9ae630 msgr2=0x55fd1f9b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b3860 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f900c46f0 tx=0x7f7f900f0220 comp rx=0 tx=0).stop 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2441431187 shutdown_connections 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fd1fa30820 0x55fd1fa32c10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd1f9b9620 0x55fd1fa2eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f73fff640 0 client.25412 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd1f9aef50 0x55fd1f9b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 --2- 172.21.15.27:0/2441431187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fd1f9ae630 0x55fd1f9b3860 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2441431187 >> 172.21.15.27:0/2441431187 conn(0x55fd1f9aa950 msgr2=0x55fd1f9b6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2441431187 shutdown_connections 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 1 -- 172.21.15.27:0/2441431187 wait complete. 2024-04-25T01:26:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.877+0000 7f7f9dfba740 20 client.25412 trim_cache size 0 max 16384 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout: "c1da276a-3301-44b9-814e-bff3319fa7ce": { 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "c1da276a-3301-44b9-814e-bff3319fa7ce", 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:26:05.862 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:26:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f7c0a2f30 msgr2=0x7f7f7c0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f7c0a2f30 0x7f7f7c0a53f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c005c40 tx=0x7f7f8c004ab0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 msgr2=0x7f7f94197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:05.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 0x7f7f94197fe0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7f90006010 tx=0x7f7f900042f0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/1510873321 shutdown_connections 2024-04-25T01:26:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f9410bd00 0x7f7f9419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f941032e0 0x7f7f94198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f7c0a2f30 0x7f7f7c0a53f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 --2- 172.21.15.27:0/1510873321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f941029c0 0x7f7f94197fe0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/1510873321 >> 172.21.15.27:0/1510873321 conn(0x7f7f94077740 msgr2=0x7f7f940fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/1510873321 shutdown_connections 2024-04-25T01:26:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:05.878+0000 7f7f9b1ce640 1 -- 172.21.15.27:0/1510873321 wait complete. 2024-04-25T01:26:05.869 INFO:tasks.cephfs.filesystem:scrub status for tag:c1da276a-3301-44b9-814e-bff3319fa7ce - {'path': '/', 'tag': 'c1da276a-3301-44b9-814e-bff3319fa7ce', 'options': 'recursive,force'} 2024-04-25T01:26:06.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:06 smithi027 ceph-mon[29197]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-25T01:26:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:06 smithi153 ceph-mon[32621]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-25T01:26:06.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:06 smithi139 ceph-mon[32760]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-25T01:26:07.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:07 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:26:07.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:07 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:07.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:07 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:26:07.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:07 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:07.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:07 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:26:07.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:07 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:07.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:07 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:26:07.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:07 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:07.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:07 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:26:07.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:07 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:07.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:07 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:26:07.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:07 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:08.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:08 smithi027 ceph-mon[29197]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-25T01:26:08.346 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:08 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:26:08.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:08 smithi153 ceph-mon[32621]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-25T01:26:08.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:08 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:26:08.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:08 smithi139 ceph-mon[32760]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-25T01:26:08.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:08 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:26:10.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:10 smithi027 ceph-mon[29197]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-25T01:26:10.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:10 smithi153 ceph-mon[32621]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-25T01:26:10.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:10 smithi139 ceph-mon[32760]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-25T01:26:12.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:12 smithi027 ceph-mon[29197]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-25T01:26:12.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:12 smithi153 ceph-mon[32621]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-25T01:26:12.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:12 smithi139 ceph-mon[32760]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-25T01:26:14.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:14 smithi027 ceph-mon[29197]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-25T01:26:14.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:14 smithi153 ceph-mon[32621]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-25T01:26:14.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:14 smithi139 ceph-mon[32760]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-25T01:26:16.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:16 smithi027 ceph-mon[29197]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-25T01:26:16.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:16 smithi153 ceph-mon[32621]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-25T01:26:16.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:16 smithi139 ceph-mon[32760]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-25T01:26:18.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:18 smithi027 ceph-mon[29197]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-25T01:26:18.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:18 smithi153 ceph-mon[32621]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-25T01:26:18.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:18 smithi139 ceph-mon[32760]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-25T01:26:20.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:20 smithi027 ceph-mon[29197]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-25T01:26:20.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:20 smithi153 ceph-mon[32621]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-25T01:26:20.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:20 smithi139 ceph-mon[32760]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-25T01:26:22.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:22 smithi027 ceph-mon[29197]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-25T01:26:22.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:22 smithi153 ceph-mon[32621]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-25T01:26:22.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:22 smithi139 ceph-mon[32760]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-25T01:26:24.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:24 smithi027 ceph-mon[29197]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:24.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:24 smithi153 ceph-mon[32621]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:24.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:24 smithi139 ceph-mon[32760]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:26.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:26 smithi027 ceph-mon[29197]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:26.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:26 smithi153 ceph-mon[32621]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:26.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:26 smithi139 ceph-mon[32760]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:28.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:28 smithi027 ceph-mon[29197]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:28.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:28 smithi153 ceph-mon[32621]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:28.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:28 smithi139 ceph-mon[32760]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:30.343 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:30 smithi139 ceph-mon[32760]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:30.343 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:30 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:30.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:30 smithi027 ceph-mon[29197]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:30.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:30 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:30.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:30 smithi153 ceph-mon[32621]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:30.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:30 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:31.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:31.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:31.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:31.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:31.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:31.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:26:32.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:32 smithi153 ceph-mon[32621]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:32.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:32 smithi139 ceph-mon[32760]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:32.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:32 smithi027 ceph-mon[29197]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:34.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:34 smithi153 ceph-mon[32621]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:34.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:34 smithi139 ceph-mon[32760]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:34.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:34 smithi027 ceph-mon[29197]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:35.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:26:35.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.000+0000 7fc49cf70640 1 -- 172.21.15.27:0/2445275049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100fd0 msgr2=0x7fc49810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.000+0000 7fc49cf70640 1 --2- 172.21.15.27:0/2445275049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100fd0 0x7fc49810d4e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc490009cd0 tx=0x7fc49002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 -- 172.21.15.27:0/2445275049 shutdown_connections 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 --2- 172.21.15.27:0/2445275049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100fd0 0x7fc49810d4e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 --2- 172.21.15.27:0/2445275049 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc498100610 0x7fc498100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 --2- 172.21.15.27:0/2445275049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4981066b0 0x7fc498106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 -- 172.21.15.27:0/2445275049 >> 172.21.15.27:0/2445275049 conn(0x7fc498077740 msgr2=0x7fc498077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 -- 172.21.15.27:0/2445275049 shutdown_connections 2024-04-25T01:26:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 -- 172.21.15.27:0/2445275049 wait complete. 2024-04-25T01:26:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 Processor -- start 2024-04-25T01:26:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.001+0000 7fc49cf70640 1 -- start start 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49cf70640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100610 0x7fc498197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49cf70640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc498100fd0 0x7fc4981984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49cf70640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4981066b0 0x7fc49819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49cf70640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc4981109f0 con 0x7fc498100610 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49cf70640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc498110870 con 0x7fc498100fd0 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49cf70640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc498110b70 con 0x7fc4981066b0 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100610 0x7fc498197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100610 0x7fc498197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39588/0 (socket says 172.21.15.27:39588) 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 -- 172.21.15.27:0/3648719546 learned_addr learned my addr 172.21.15.27:0/3648719546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc48ffff640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc498100fd0 0x7fc4981984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc496d9d640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4981066b0 0x7fc49819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 -- 172.21.15.27:0/3648719546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4981066b0 msgr2=0x7fc49819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4981066b0 0x7fc49819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 -- 172.21.15.27:0/3648719546 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc498100fd0 msgr2=0x7fc4981984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc498100fd0 0x7fc4981984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 -- 172.21.15.27:0/3648719546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49819cf50 con 0x7fc498100610 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc48ffff640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc498100fd0 0x7fc4981984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.002+0000 7fc49659c640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100610 0x7fc498197f60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc48800d950 tx=0x7fc48800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.003+0000 7fc48f7fe640 1 -- 172.21.15.27:0/3648719546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488017020 con 0x7fc498100610 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.003+0000 7fc49cf70640 1 -- 172.21.15.27:0/3648719546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49819d240 con 0x7fc498100610 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.003+0000 7fc49cf70640 1 -- 172.21.15.27:0/3648719546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4981021e0 con 0x7fc498100610 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.003+0000 7fc48f7fe640 1 -- 172.21.15.27:0/3648719546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc488009c30 con 0x7fc498100610 2024-04-25T01:26:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.003+0000 7fc48f7fe640 1 -- 172.21.15.27:0/3648719546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48800f6b0 con 0x7fc498100610 2024-04-25T01:26:35.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc48f7fe640 1 -- 172.21.15.27:0/3648719546 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7fc48801a020 con 0x7fc498100610 2024-04-25T01:26:35.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc49fc80740 1 Processor -- start 2024-04-25T01:26:35.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc49fc80740 1 -- start start 2024-04-25T01:26:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a5da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a5df50 0x563935a680e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a68620 0x563935a6aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc49fc80740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5639359c26b0 con 0x563935a68620 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc49fc80740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56393596a880 con 0x563935a5d630 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc49fc80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5639359be670 con 0x563935a5df50 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.006+0000 7fc48f7fe640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4840a2f30 0x7fc4840a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc48ffff640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4840a2f30 0x7fc4840a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc496d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a68620 0x563935a6aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc496d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a68620 0x563935a6aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39604/0 (socket says 172.21.15.27:39604) 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc496d9d640 1 -- 172.21.15.27:0/123244894 learned_addr learned my addr 172.21.15.27:0/123244894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc49659c640 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a5df50 0x563935a680e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc496d9d640 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a5da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc49659c640 1 -- 172.21.15.27:0/123244894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 msgr2=0x563935a5da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc49659c640 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a5da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc49659c640 1 -- 172.21.15.27:0/123244894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a68620 msgr2=0x563935a6aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc49659c640 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a68620 0x563935a6aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc49659c640 1 -- 172.21.15.27:0/123244894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639357a07d0 con 0x563935a5df50 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc496d9d640 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a68620 0x563935a6aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc48ffff640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4840a2f30 0x7fc4840a53f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4800058c0 tx=0x7fc480005850 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.007+0000 7fc48f7fe640 1 -- 172.21.15.27:0/3648719546 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc4880cf9a0 con 0x7fc498100610 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.008+0000 7fc49659c640 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a5df50 0x563935a680e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc4880ccb20 tx=0x7fc488023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=518342523ae2bd8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.008+0000 7fc48cff9640 1 -- 172.21.15.27:0/123244894 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488007420 con 0x563935a5df50 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.008+0000 7fc48cff9640 1 -- 172.21.15.27:0/123244894 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc48809bff0 con 0x563935a5df50 2024-04-25T01:26:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.008+0000 7fc48cff9640 1 -- 172.21.15.27:0/123244894 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488097b90 con 0x563935a5df50 2024-04-25T01:26:35.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 -- 172.21.15.27:0/123244894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a5df50 msgr2=0x563935a680e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:35.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a5df50 0x563935a680e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc4880ccb20 tx=0x7fc488023000 comp rx=0 tx=0).stop 2024-04-25T01:26:35.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 -- 172.21.15.27:0/123244894 shutdown_connections 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a68620 0x563935a6aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a5df50 0x563935a680e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 --2- 172.21.15.27:0/123244894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a5da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 -- 172.21.15.27:0/123244894 >> 172.21.15.27:0/123244894 conn(0x563935a59950 msgr2=0x563935a59d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 -- 172.21.15.27:0/123244894 shutdown_connections 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.009+0000 7fc49fc80740 1 -- 172.21.15.27:0/123244894 wait complete. 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.010+0000 7fc49fc80740 1 -- start start 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.010+0000 7fc49fc80740 1 Processor -- start 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5d630 0x563935a655f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5df50 0x563935a65b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5d630 0x563935a655f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a68620 0x563935a66070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5d630 0x563935a655f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39616/0 (socket says 172.21.15.27:39616) 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 -- 172.21.15.27:0/2708595808 learned_addr learned my addr 172.21.15.27:0/2708595808 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563935a6b3b0 con 0x563935a5d630 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56393597df50 con 0x563935a5df50 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563935a6b530 con 0x563935a68620 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc48ffff640 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5df50 0x563935a65b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc496d9d640 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a68620 0x563935a66070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49fc80740 10 client.0 _ll_register_callbacks cb 0x5639359b9000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 -- 172.21.15.27:0/2708595808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a68620 msgr2=0x563935a66070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a68620 0x563935a66070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 -- 172.21.15.27:0/2708595808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5df50 msgr2=0x563935a65b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5df50 0x563935a65b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 -- 172.21.15.27:0/2708595808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639356d63f0 con 0x563935a5d630 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc48809bff0 con 0x563935a5d630 2024-04-25T01:26:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc49659c640 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5d630 0x563935a655f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc4880ccc60 tx=0x7fc48809b380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68659fd32e2adf52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.011+0000 7fc47dffb640 10 client.15876 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc47dffb640 1 -- 172.21.15.27:0/2708595808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488097b90 con 0x563935a5d630 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc47dffb640 1 -- 172.21.15.27:0/2708595808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc488097f60 con 0x563935a5d630 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc47dffb640 1 -- 172.21.15.27:0/2708595808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc4880a4560 con 0x563935a5d630 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc49fc80740 10 client.15876 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5639356960f0 con 0x563935a5d630 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc47dffb640 1 -- 172.21.15.27:0/2708595808 <== mon.0 v2:172.21.15.27:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc4880860f0 con 0x563935a5d630 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc49fc80740 10 client.15876 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc49fc80740 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc49fc80740 20 client.15876 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563935a78640 0x563935a7aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc49fc80740 4 client.15876 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.012+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639359cfbf0 con 0x563935a78640 2024-04-25T01:26:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.013+0000 7fc48ffff640 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563935a78640 0x563935a7aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:35.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.013+0000 7fc48ffff640 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563935a78640 0x563935a7aa30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=befc9b73f8941ca0 server_cookie=6dba8fc5af815863 in_seq=0 out_seq=0 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.013+0000 7fc47dffb640 10 client.15876 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc47dffb640 1 -- 172.21.15.27:0/2708595808 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639359cfbf0 con 0x563935a78640 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc47dffb640 10 client.15876 handle_command_reply: tid=0 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 client.15876 shutdown 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5d630 msgr2=0x563935a655f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5d630 0x563935a655f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc4880ccc60 tx=0x7fc48809b380 comp rx=0 tx=0).stop 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 shutdown_connections 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563935a78640 0x563935a7aa30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc47dffb640 0 client.15876 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a68620 0x563935a66070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5df50 0x563935a65b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2708595808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5d630 0x563935a655f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 >> 172.21.15.27:0/2708595808 conn(0x563935a59950 msgr2=0x563935a5f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 shutdown_connections 2024-04-25T01:26:35.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 1 -- 172.21.15.27:0/2708595808 wait complete. 2024-04-25T01:26:35.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.014+0000 7fc49fc80740 20 client.15876 trim_cache size 0 max 16384 2024-04-25T01:26:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.050+0000 7fc49fc80740 1 Processor -- start 2024-04-25T01:26:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.050+0000 7fc49fc80740 1 -- start start 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.050+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a67d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.050+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a742a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a800b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc49fc80740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc4780126d0 con 0x563935a5df50 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc49fc80740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc47800ffa0 con 0x563935a5d630 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc49fc80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc47800ddf0 con 0x563935a59950 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a800b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc49659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a67d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a800b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39624/0 (socket says 172.21.15.27:39624) 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 -- 172.21.15.27:0/1382771456 learned_addr learned my addr 172.21.15.27:0/1382771456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc49659c640 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a67d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:42076/0 (socket says 172.21.15.27:42076) 2024-04-25T01:26:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc48ffff640 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 -- 172.21.15.27:0/1382771456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 msgr2=0x563935a67d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a67d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 -- 172.21.15.27:0/1382771456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 msgr2=0x563935a742a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a742a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 -- 172.21.15.27:0/1382771456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563935a55020 con 0x563935a5df50 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc49659c640 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a67d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.051+0000 7fc496d9d640 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a800b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc49002ae00 tx=0x7fc490005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb19bbfc40d6548 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.052+0000 7fc47ffff640 1 -- 172.21.15.27:0/1382771456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490031b70 con 0x563935a5df50 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.052+0000 7fc47ffff640 1 -- 172.21.15.27:0/1382771456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc490036040 con 0x563935a5df50 2024-04-25T01:26:36.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.052+0000 7fc47ffff640 1 -- 172.21.15.27:0/1382771456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4900305a0 con 0x563935a5df50 2024-04-25T01:26:36.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 -- 172.21.15.27:0/1382771456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 msgr2=0x563935a800b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a800b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc49002ae00 tx=0x7fc490005c50 comp rx=0 tx=0).stop 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 -- 172.21.15.27:0/1382771456 shutdown_connections 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a800b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a742a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 --2- 172.21.15.27:0/1382771456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a67d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 -- 172.21.15.27:0/1382771456 >> 172.21.15.27:0/1382771456 conn(0x563935a64230 msgr2=0x563935a58af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 -- 172.21.15.27:0/1382771456 shutdown_connections 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 -- 172.21.15.27:0/1382771456 wait complete. 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 -- start start 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.053+0000 7fc49fc80740 1 Processor -- start 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a707a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a70ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49fc80740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a6cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49fc80740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc47800e460 con 0x563935a5df50 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49fc80740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc47800f8d0 con 0x563935a5d630 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49fc80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc47800eb30 con 0x563935a59950 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49fc80740 10 client.0 _ll_register_callbacks cb 0x7fc490009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc496d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a6cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc496d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a6cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39632/0 (socket says 172.21.15.27:39632) 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc496d9d640 1 -- 172.21.15.27:0/2194222710 learned_addr learned my addr 172.21.15.27:0/2194222710 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc48ffff640 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a70ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49659c640 1 -- 172.21.15.27:0/2194222710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 msgr2=0x563935a70ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49659c640 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a70ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49659c640 1 -- 172.21.15.27:0/2194222710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 msgr2=0x563935a6cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49659c640 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a6cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc49659c640 1 -- 172.21.15.27:0/2194222710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563935a6d520 con 0x563935a59950 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.054+0000 7fc496d9d640 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a6cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.055+0000 7fc49659c640 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a707a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc4880ccc10 tx=0x7fc488023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b284263839c4ba23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.055+0000 7fc47dffb640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.055+0000 7fc47dffb640 1 -- 172.21.15.27:0/2194222710 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488097b90 con 0x563935a59950 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.055+0000 7fc49fc80740 1 -- 172.21.15.27:0/2194222710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc490036040 con 0x563935a59950 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.055+0000 7fc47dffb640 1 -- 172.21.15.27:0/2194222710 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc4880a42e0 con 0x563935a59950 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.055+0000 7fc47dffb640 1 -- 172.21.15.27:0/2194222710 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc48809b6c0 con 0x563935a59950 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.055+0000 7fc49fc80740 10 client.25492 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.055+0000 7fc49fc80740 1 -- 172.21.15.27:0/2194222710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563935a6d520 con 0x563935a59950 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc47dffb640 1 -- 172.21.15.27:0/2194222710 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc490036040 con 0x563935a59950 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc47dffb640 1 -- 172.21.15.27:0/2194222710 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc48809add0 con 0x563935a59950 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc49fc80740 10 client.25492 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc49fc80740 10 client.25492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc49fc80740 20 client.25492 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563935a68620 0x563935ae1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc49fc80740 4 client.25492 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc49fc80740 1 -- 172.21.15.27:0/2194222710 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639359c9f40 con 0x563935a68620 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.056+0000 7fc48ffff640 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563935a68620 0x563935ae1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc47dffb640 10 client.25492 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc48ffff640 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563935a68620 0x563935ae1830 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14c4af16a673fa5 server_cookie=769fa14179aae35b in_seq=0 out_seq=0 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc47dffb640 1 -- 172.21.15.27:0/2194222710 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5639359c9f40 con 0x563935a68620 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc47dffb640 10 client.25492 handle_command_reply: tid=0 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 client.25492 shutdown 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 -- 172.21.15.27:0/2194222710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 msgr2=0x563935a707a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a707a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc4880ccc10 tx=0x7fc488023000 comp rx=0 tx=0).stop 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 -- 172.21.15.27:0/2194222710 shutdown_connections 2024-04-25T01:26:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563935a68620 0x563935ae1830 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563935a5df50 0x563935a6cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc47dffb640 0 client.25492 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563935a5d630 0x563935a70ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 --2- 172.21.15.27:0/2194222710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563935a59950 0x563935a707a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 -- 172.21.15.27:0/2194222710 >> 172.21.15.27:0/2194222710 conn(0x563935a64230 msgr2=0x563935a58260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 -- 172.21.15.27:0/2194222710 shutdown_connections 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 1 -- 172.21.15.27:0/2194222710 wait complete. 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.057+0000 7fc49fc80740 20 client.25492 trim_cache size 0 max 16384 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:26:36.044 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:26:36.045 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:26:36.045 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:26:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 -- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4840a2f30 msgr2=0x7fc4840a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4840a2f30 0x7fc4840a53f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4800058c0 tx=0x7fc480005850 comp rx=0 tx=0).stop 2024-04-25T01:26:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 -- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100610 msgr2=0x7fc498197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100610 0x7fc498197f60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc48800d950 tx=0x7fc48800de10 comp rx=0 tx=0).stop 2024-04-25T01:26:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 -- 172.21.15.27:0/3648719546 shutdown_connections 2024-04-25T01:26:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4981066b0 0x7fc49819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc498100fd0 0x7fc4981984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4840a2f30 0x7fc4840a53f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 --2- 172.21.15.27:0/3648719546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc498100610 0x7fc498197f60 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 -- 172.21.15.27:0/3648719546 >> 172.21.15.27:0/3648719546 conn(0x7fc498077740 msgr2=0x7fc4980ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 -- 172.21.15.27:0/3648719546 shutdown_connections 2024-04-25T01:26:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.058+0000 7fc49cf70640 1 -- 172.21.15.27:0/3648719546 wait complete. 2024-04-25T01:26:36.053 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:26:36.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:26:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 -- 172.21.15.27:0/4115755151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 msgr2=0x7fceb0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 --2- 172.21.15.27:0/4115755151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 0x7fceb0100aa0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009930 tx=0x7fcea402a420 comp rx=0 tx=0).stop 2024-04-25T01:26:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 -- 172.21.15.27:0/4115755151 shutdown_connections 2024-04-25T01:26:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 --2- 172.21.15.27:0/4115755151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fceb0100fe0 0x7fceb010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 --2- 172.21.15.27:0/4115755151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 0x7fceb0100aa0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 --2- 172.21.15.27:0/4115755151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb0106630 0x7fceb0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 -- 172.21.15.27:0/4115755151 >> 172.21.15.27:0/4115755151 conn(0x7fceb0077740 msgr2=0x7fceb0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 -- 172.21.15.27:0/4115755151 shutdown_connections 2024-04-25T01:26:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.177+0000 7fceb5b70640 1 -- 172.21.15.27:0/4115755151 wait complete. 2024-04-25T01:26:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceb5b70640 1 Processor -- start 2024-04-25T01:26:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceb5b70640 1 -- start start 2024-04-25T01:26:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceb5b70640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 0x7fceb0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceb5b70640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb0100fe0 0x7fceb0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceb5b70640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fceb0106630 0x7fceb019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceb5b70640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fceb0111b40 con 0x7fceb0100620 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceb5b70640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fceb01119c0 con 0x7fceb0106630 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceb5b70640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fceb0111cc0 con 0x7fceb0100fe0 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.178+0000 7fceaf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 0x7fceb0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 0x7fceb0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39658/0 (socket says 172.21.15.27:39658) 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaf7fe640 1 -- 172.21.15.27:0/3464197783 learned_addr learned my addr 172.21.15.27:0/3464197783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaeffd640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb0100fe0 0x7fceb0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaffff640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fceb0106630 0x7fceb019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaf7fe640 1 -- 172.21.15.27:0/3464197783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb0100fe0 msgr2=0x7fceb0198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaf7fe640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb0100fe0 0x7fceb0198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaf7fe640 1 -- 172.21.15.27:0/3464197783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fceb0106630 msgr2=0x7fceb019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaf7fe640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fceb0106630 0x7fceb019c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaf7fe640 1 -- 172.21.15.27:0/3464197783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb019cff0 con 0x7fceb0100620 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceaf7fe640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 0x7fceb0198000 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcea000e9f0 tx=0x7fcea000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceacff9640 1 -- 172.21.15.27:0/3464197783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea000cd80 con 0x7fceb0100620 2024-04-25T01:26:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb019d2e0 con 0x7fceb0100620 2024-04-25T01:26:36.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceacff9640 1 -- 172.21.15.27:0/3464197783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcea0010910 con 0x7fceb0100620 2024-04-25T01:26:36.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.179+0000 7fceacff9640 1 -- 172.21.15.27:0/3464197783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0018990 con 0x7fceb0100620 2024-04-25T01:26:36.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.180+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb01a4b70 con 0x7fceb0100620 2024-04-25T01:26:36.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.181+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb0068560 con 0x7fceb0100620 2024-04-25T01:26:36.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.182+0000 7fceacff9640 1 -- 172.21.15.27:0/3464197783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7fcea0010ac0 con 0x7fceb0100620 2024-04-25T01:26:36.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.182+0000 7fceacff9640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce980a2f30 0x7fce980a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.183+0000 7fceacff9640 1 -- 172.21.15.27:0/3464197783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fcea00cfe00 con 0x7fceb0100620 2024-04-25T01:26:36.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.183+0000 7fceaeffd640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce980a2f30 0x7fce980a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.183+0000 7fceaeffd640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce980a2f30 0x7fce980a53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009900 tx=0x7fcea40058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.187+0000 7fceacff9640 1 -- 172.21.15.27:0/3464197783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcea009c450 con 0x7fceb0100620 2024-04-25T01:26:36.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.427+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fceb0108bb0 con 0x7fceb0100620 2024-04-25T01:26:36.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.428+0000 7fceacff9640 1 -- 172.21.15.27:0/3464197783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7fcea009bd50 con 0x7fceb0100620 2024-04-25T01:26:36.408 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:26:36.408 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:25:23.250708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24505,24517]},"id":1}]} 2024-04-25T01:26:36.408 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-04-25T01:26:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce980a2f30 msgr2=0x7fce980a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce980a2f30 0x7fce980a53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009900 tx=0x7fcea40058d0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 msgr2=0x7fceb0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 0x7fceb0198000 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcea000e9f0 tx=0x7fcea000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 shutdown_connections 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fceb0106630 0x7fceb019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb0100fe0 0x7fceb0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce980a2f30 0x7fce980a53f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 --2- 172.21.15.27:0/3464197783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb0100620 0x7fceb0198000 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 >> 172.21.15.27:0/3464197783 conn(0x7fceb0077740 msgr2=0x7fceb0102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 shutdown_connections 2024-04-25T01:26:36.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.430+0000 7fceb5b70640 1 -- 172.21.15.27:0/3464197783 wait complete. 2024-04-25T01:26:36.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:26:36.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 -- 172.21.15.27:0/2608667641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb701066b0 msgr2=0x7fdb70106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/2608667641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb701066b0 0x7fdb70106a90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009930 tx=0x7fdb6402a420 comp rx=0 tx=0).stop 2024-04-25T01:26:36.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 -- 172.21.15.27:0/2608667641 shutdown_connections 2024-04-25T01:26:36.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/2608667641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb70100fd0 0x7fdb7010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/2608667641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb70100610 0x7fdb70100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/2608667641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb701066b0 0x7fdb70106a90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 -- 172.21.15.27:0/2608667641 >> 172.21.15.27:0/2608667641 conn(0x7fdb70077740 msgr2=0x7fdb70077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 -- 172.21.15.27:0/2608667641 shutdown_connections 2024-04-25T01:26:36.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.550+0000 7fdb76cb3640 1 -- 172.21.15.27:0/2608667641 wait complete. 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb76cb3640 1 Processor -- start 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb76cb3640 1 -- start start 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb76cb3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb70100610 0x7fdb70198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb76cb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb70100fd0 0x7fdb701993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb76cb3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb701066b0 0x7fdb701a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb76cb3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdb701109f0 con 0x7fdb70100610 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb76cb3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdb70110870 con 0x7fdb701066b0 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb76cb3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb70110b70 con 0x7fdb70100fd0 2024-04-25T01:26:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb74a4f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb70100610 0x7fdb70198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb74a4f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb70100610 0x7fdb70198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39674/0 (socket says 172.21.15.27:39674) 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.551+0000 7fdb74a4f640 1 -- 172.21.15.27:0/3635088015 learned_addr learned my addr 172.21.15.27:0/3635088015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb70100fd0 0x7fdb701993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb75250640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb701066b0 0x7fdb701a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb74a4f640 1 -- 172.21.15.27:0/3635088015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb70100fd0 msgr2=0x7fdb701993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb74a4f640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb70100fd0 0x7fdb701993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb74a4f640 1 -- 172.21.15.27:0/3635088015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb701066b0 msgr2=0x7fdb701a3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb74a4f640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb701066b0 0x7fdb701a3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb74a4f640 1 -- 172.21.15.27:0/3635088015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb701a4450 con 0x7fdb70100610 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb70100fd0 0x7fdb701993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb74a4f640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb70100610 0x7fdb70198e90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdb64005940 tx=0x7fdb640040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb6dffb640 1 -- 172.21.15.27:0/3635088015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6403f030 con 0x7fdb70100610 2024-04-25T01:26:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.552+0000 7fdb76cb3640 1 -- 172.21.15.27:0/3635088015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb701a4620 con 0x7fdb70100610 2024-04-25T01:26:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.553+0000 7fdb6dffb640 1 -- 172.21.15.27:0/3635088015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb640044c0 con 0x7fdb70100610 2024-04-25T01:26:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.553+0000 7fdb6dffb640 1 -- 172.21.15.27:0/3635088015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64035070 con 0x7fdb70100610 2024-04-25T01:26:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.553+0000 7fdb76cb3640 1 -- 172.21.15.27:0/3635088015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb701a4ae0 con 0x7fdb70100610 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb799bf740 1 Processor -- start 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb799bf740 1 -- start start 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5580 0x55cda65d5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5ea0 0x55cda65e0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65e2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb799bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cda653a670 con 0x55cda65e0570 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb799bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cda64e2970 con 0x55cda65d5ea0 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb799bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cda6536630 con 0x55cda65d5580 2024-04-25T01:26:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb6dffb640 1 -- 172.21.15.27:0/3635088015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7fdb64042030 con 0x7fdb70100610 2024-04-25T01:26:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.556+0000 7fdb75250640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5580 0x55cda65d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb6ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65e2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb75250640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5580 0x55cda65d5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:42134/0 (socket says 172.21.15.27:42134) 2024-04-25T01:26:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb75250640 1 -- 172.21.15.27:0/1605891358 learned_addr learned my addr 172.21.15.27:0/1605891358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb75250640 1 -- 172.21.15.27:0/1605891358 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5ea0 msgr2=0x55cda65e0030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:26:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb75250640 1 --2- 172.21.15.27:0/1605891358 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5ea0 0x55cda65e0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb75250640 1 -- 172.21.15.27:0/1605891358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 msgr2=0x55cda65e2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb75250640 1 --2- 172.21.15.27:0/1605891358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65e2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb75250640 1 -- 172.21.15.27:0/1605891358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cda6506400 con 0x55cda65d5580 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/1605891358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65e2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb75250640 1 --2- 172.21.15.27:0/1605891358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5580 0x55cda65d5960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009f50 tx=0x7fdb680118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f42cd8bb87e2654c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb6dffb640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdb600a2eb0 0x7fdb600a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.557+0000 7fdb5affd640 1 -- 172.21.15.27:0/1605891358 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68004b20 con 0x55cda65d5580 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.558+0000 7fdb6dffb640 1 -- 172.21.15.27:0/3635088015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdb640efe40 con 0x7fdb70100610 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.558+0000 7fdb74a4f640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdb600a2eb0 0x7fdb600a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.558+0000 7fdb5affd640 1 -- 172.21.15.27:0/1605891358 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb68004cc0 con 0x55cda65d5580 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.558+0000 7fdb5affd640 1 -- 172.21.15.27:0/1605891358 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68005020 con 0x55cda65d5580 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 -- 172.21.15.27:0/1605891358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5580 msgr2=0x55cda65d5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1605891358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5580 0x55cda65d5960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009f50 tx=0x7fdb680118d0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb74a4f640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdb600a2eb0 0x7fdb600a5370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdb640ed020 tx=0x7fdb64009680 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 -- 172.21.15.27:0/1605891358 shutdown_connections 2024-04-25T01:26:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1605891358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65e2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1605891358 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5ea0 0x55cda65e0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1605891358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5580 0x55cda65d5960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 -- 172.21.15.27:0/1605891358 >> 172.21.15.27:0/1605891358 conn(0x55cda65d18a0 msgr2=0x55cda65d1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 -- 172.21.15.27:0/1605891358 shutdown_connections 2024-04-25T01:26:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 -- 172.21.15.27:0/1605891358 wait complete. 2024-04-25T01:26:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 -- start start 2024-04-25T01:26:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.559+0000 7fdb799bf740 1 Processor -- start 2024-04-25T01:26:36.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5580 0x55cda65dfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65dd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb799bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cda65e3360 con 0x55cda65e0570 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb799bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cda64f6050 con 0x55cda65d5580 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb799bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cda65e34e0 con 0x55cda65d5ea0 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb799bf740 10 client.0 _ll_register_callbacks cb 0x55cda624dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65dd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65dd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39690/0 (socket says 172.21.15.27:39690) 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 -- 172.21.15.27:0/3040313363 learned_addr learned my addr 172.21.15.27:0/3040313363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5580 0x55cda65dfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 -- 172.21.15.27:0/3040313363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 msgr2=0x55cda65dd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 -- 172.21.15.27:0/3040313363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5580 msgr2=0x55cda65dfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5580 0x55cda65dfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.560+0000 7fdb75250640 1 -- 172.21.15.27:0/3040313363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cda64c7dc0 con 0x55cda65e0570 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.561+0000 7fdb75250640 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65dd730 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fdb68012fd0 tx=0x7fdb680044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599a93a4003cfbea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.561+0000 7fdb4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.561+0000 7fdb4ffff640 1 -- 172.21.15.27:0/3040313363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68004a40 con 0x55cda65e0570 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.561+0000 7fdb4ffff640 1 -- 172.21.15.27:0/3040313363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdb68016040 con 0x55cda65e0570 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.561+0000 7fdb799bf740 1 -- 172.21.15.27:0/3040313363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb68004cc0 con 0x55cda65e0570 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.561+0000 7fdb4ffff640 1 -- 172.21.15.27:0/3040313363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdb68035050 con 0x55cda65e0570 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.561+0000 7fdb799bf740 10 client.15900 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.561+0000 7fdb799bf740 1 -- 172.21.15.27:0/3040313363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cda624e3f0 con 0x55cda65e0570 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb4ffff640 1 -- 172.21.15.27:0/3040313363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb6802c070 con 0x55cda65e0570 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb4ffff640 1 -- 172.21.15.27:0/3040313363 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdb68027570 con 0x55cda65e0570 2024-04-25T01:26:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb799bf740 10 client.15900 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb799bf740 10 client.15900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb799bf740 20 client.15900 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb799bf740 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55cda65f0590 0x55cda65f2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb799bf740 4 client.15900 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb799bf740 1 -- 172.21.15.27:0/3040313363 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdb68004cc0 con 0x55cda65f0590 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.562+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55cda65f0590 0x55cda65f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.563+0000 7fdb4ffff640 10 client.15900 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.563+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55cda65f0590 0x55cda65f2980 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b951532c2f149ba5 server_cookie=7c92f507b5dfb457 in_seq=0 out_seq=0 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb4ffff640 1 -- 172.21.15.27:0/3040313363 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fdb68004cc0 con 0x55cda65f0590 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb4ffff640 10 client.15900 handle_command_reply: tid=0 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 client.15900 shutdown 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 -- 172.21.15.27:0/3040313363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 msgr2=0x55cda65dd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65dd730 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fdb68012fd0 tx=0x7fdb680044d0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 -- 172.21.15.27:0/3040313363 shutdown_connections 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55cda65f0590 0x55cda65f2980 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65e0570 0x55cda65dd730 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb4ffff640 0 client.15900 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 --2- 172.21.15.27:0/3040313363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65d5580 0x55cda65dfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.564+0000 7fdb799bf740 1 -- 172.21.15.27:0/3040313363 >> 172.21.15.27:0/3040313363 conn(0x55cda65d18a0 msgr2=0x55cda65d76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.565+0000 7fdb799bf740 1 -- 172.21.15.27:0/3040313363 shutdown_connections 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.565+0000 7fdb799bf740 1 -- 172.21.15.27:0/3040313363 wait complete. 2024-04-25T01:26:36.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.565+0000 7fdb799bf740 20 client.15900 trim_cache size 0 max 16384 2024-04-25T01:26:36.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb799bf740 1 Processor -- start 2024-04-25T01:26:36.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb799bf740 1 -- start start 2024-04-25T01:26:36.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda65ec250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 0x55cda65f80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb799bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdb701af3b0 con 0x55cda65d5580 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb799bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdb701acc80 con 0x55cda65e0570 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb799bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb701aaad0 con 0x55cda65d5ea0 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb75250640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda65ec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb75250640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda65ec250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39694/0 (socket says 172.21.15.27:39694) 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb75250640 1 -- 172.21.15.27:0/4057456005 learned_addr learned my addr 172.21.15.27:0/4057456005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.592+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb75250640 1 -- 172.21.15.27:0/4057456005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 msgr2=0x55cda65dea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb75250640 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb75250640 1 -- 172.21.15.27:0/4057456005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 msgr2=0x55cda65f80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb75250640 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 0x55cda65f80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb75250640 1 -- 172.21.15.27:0/4057456005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cda65df8b0 con 0x55cda65d5580 2024-04-25T01:26:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:36.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb75250640 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda65ec250 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdb680095d0 tx=0x7fdb68009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae1856dcc7a569f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb5a7fc640 1 -- 172.21.15.27:0/4057456005 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68016070 con 0x55cda65d5580 2024-04-25T01:26:36.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb5a7fc640 1 -- 172.21.15.27:0/4057456005 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb6802c040 con 0x55cda65d5580 2024-04-25T01:26:36.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.593+0000 7fdb5a7fc640 1 -- 172.21.15.27:0/4057456005 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68005210 con 0x55cda65d5580 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 -- 172.21.15.27:0/4057456005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 msgr2=0x55cda65ec250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda65ec250 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdb680095d0 tx=0x7fdb68009fb0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 -- 172.21.15.27:0/4057456005 shutdown_connections 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 0x55cda65f80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda65dea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 --2- 172.21.15.27:0/4057456005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda65ec250 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 -- 172.21.15.27:0/4057456005 >> 172.21.15.27:0/4057456005 conn(0x55cda65d18a0 msgr2=0x55cda65d7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 -- 172.21.15.27:0/4057456005 shutdown_connections 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.594+0000 7fdb799bf740 1 -- 172.21.15.27:0/4057456005 wait complete. 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 1 -- start start 2024-04-25T01:26:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 1 Processor -- start 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda664ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda664f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 0x55cda65e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdb701ab140 con 0x55cda65d5580 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdb701ac5b0 con 0x55cda65e0570 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb701ab810 con 0x55cda65d5ea0 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb799bf740 10 client.0 _ll_register_callbacks cb 0x7fdb5c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.595+0000 7fdb75250640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda664ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda664ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39706/0 (socket says 172.21.15.27:39706) 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 -- 172.21.15.27:0/1847935871 learned_addr learned my addr 172.21.15.27:0/1847935871 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 0x55cda65e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda664f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 -- 172.21.15.27:0/1847935871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 msgr2=0x55cda664f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda664f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 -- 172.21.15.27:0/1847935871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 msgr2=0x55cda65e5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 0x55cda65e5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 -- 172.21.15.27:0/1847935871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cda65d5cd0 con 0x55cda65d5580 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb75250640 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda664ed60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fdb680095a0 tx=0x7fdb68004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e056701214595374 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.596+0000 7fdb4ffff640 1 -- 172.21.15.27:0/1847935871 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68016070 con 0x55cda65d5580 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb4ffff640 1 -- 172.21.15.27:0/1847935871 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdb68005210 con 0x55cda65d5580 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 1 -- 172.21.15.27:0/1847935871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb6802c040 con 0x55cda65d5580 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb4ffff640 1 -- 172.21.15.27:0/1847935871 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdb68004380 con 0x55cda65d5580 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 10 client.15906 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 1 -- 172.21.15.27:0/1847935871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cda63187d0 con 0x55cda65d5580 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb4ffff640 1 -- 172.21.15.27:0/1847935871 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb6802c040 con 0x55cda65d5580 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb4ffff640 1 -- 172.21.15.27:0/1847935871 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdb68005860 con 0x55cda65d5580 2024-04-25T01:26:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 10 client.15906 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 10 client.15906 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 20 client.15906 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55cda6656700 0x55cda6658af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 4 client.15906 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.597+0000 7fdb799bf740 1 -- 172.21.15.27:0/1847935871 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cda6547bb0 con 0x55cda6656700 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55cda6656700 0x55cda6658af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb6ffff640 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55cda6656700 0x55cda6658af0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f217019f407ee111 server_cookie=8dd73dfda4f300d0 in_seq=0 out_seq=0 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb4ffff640 10 client.15906 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb4ffff640 1 -- 172.21.15.27:0/1847935871 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cda6547bb0 con 0x55cda6656700 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb4ffff640 10 client.15906 handle_command_reply: tid=0 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb799bf740 1 client.15906 shutdown 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb799bf740 1 -- 172.21.15.27:0/1847935871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 msgr2=0x55cda664ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda664ed60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fdb680095a0 tx=0x7fdb68004a70 comp rx=0 tx=0).stop 2024-04-25T01:26:36.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb799bf740 1 -- 172.21.15.27:0/1847935871 shutdown_connections 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.598+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55cda6656700 0x55cda6658af0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cda65e0570 0x55cda65e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cda65d5ea0 0x55cda664f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb799bf740 1 --2- 172.21.15.27:0/1847935871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cda65d5580 0x55cda664ed60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb799bf740 1 -- 172.21.15.27:0/1847935871 >> 172.21.15.27:0/1847935871 conn(0x55cda65d18a0 msgr2=0x55cda65eb240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb4ffff640 0 client.15906 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb799bf740 1 -- 172.21.15.27:0/1847935871 shutdown_connections 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb799bf740 1 -- 172.21.15.27:0/1847935871 wait complete. 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb799bf740 20 client.15906 trim_cache size 0 max 16384 2024-04-25T01:26:36.583 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb76cb3640 1 -- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdb600a2eb0 msgr2=0x7fdb600a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdb600a2eb0 0x7fdb600a5370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdb640ed020 tx=0x7fdb64009680 comp rx=0 tx=0).stop 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.599+0000 7fdb76cb3640 1 -- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb70100610 msgr2=0x7fdb70198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb70100610 0x7fdb70198e90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdb64005940 tx=0x7fdb640040c0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 -- 172.21.15.27:0/3635088015 shutdown_connections 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb701066b0 0x7fdb701a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb70100fd0 0x7fdb701993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdb600a2eb0 0x7fdb600a5370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 --2- 172.21.15.27:0/3635088015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb70100610 0x7fdb70198e90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 -- 172.21.15.27:0/3635088015 >> 172.21.15.27:0/3635088015 conn(0x7fdb70077740 msgr2=0x7fdb700ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 -- 172.21.15.27:0/3635088015 shutdown_connections 2024-04-25T01:26:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.600+0000 7fdb76cb3640 1 -- 172.21.15.27:0/3635088015 wait complete. 2024-04-25T01:26:36.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:26:36.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:36 smithi027 ceph-mon[29197]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:36.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:36 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3464197783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 -- 172.21.15.27:0/4287304494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 msgr2=0x7ff6bc10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 --2- 172.21.15.27:0/4287304494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 0x7ff6bc10d4e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009cd0 tx=0x7ff6b402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 -- 172.21.15.27:0/4287304494 shutdown_connections 2024-04-25T01:26:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 --2- 172.21.15.27:0/4287304494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 0x7ff6bc10d4e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 --2- 172.21.15.27:0/4287304494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100610 0x7ff6bc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 --2- 172.21.15.27:0/4287304494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6bc1066b0 0x7ff6bc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 -- 172.21.15.27:0/4287304494 >> 172.21.15.27:0/4287304494 conn(0x7ff6bc077740 msgr2=0x7ff6bc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 -- 172.21.15.27:0/4287304494 shutdown_connections 2024-04-25T01:26:36.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.753+0000 7ff6c4148640 1 -- 172.21.15.27:0/4287304494 wait complete. 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c4148640 1 Processor -- start 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c4148640 1 -- start start 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c4148640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6bc100610 0x7ff6bc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c4148640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 0x7ff6bc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c4148640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc1066b0 0x7ff6bc19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c4148640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc110a80 con 0x7ff6bc100fd0 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c4148640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc110900 con 0x7ff6bc100610 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c4148640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc110c00 con 0x7ff6bc1066b0 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c16e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 0x7ff6bc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c16e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 0x7ff6bc198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39726/0 (socket says 172.21.15.27:39726) 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c16e3640 1 -- 172.21.15.27:0/1057080196 learned_addr learned my addr 172.21.15.27:0/1057080196 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6bc100610 0x7ff6bc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc1066b0 0x7ff6bc19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c16e3640 1 -- 172.21.15.27:0/1057080196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc1066b0 msgr2=0x7ff6bc19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc1066b0 0x7ff6bc19c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.754+0000 7ff6c16e3640 1 -- 172.21.15.27:0/1057080196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6bc100610 msgr2=0x7ff6bc197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6bc100610 0x7ff6bc197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6c16e3640 1 -- 172.21.15.27:0/1057080196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6bc19cf20 con 0x7ff6bc100fd0 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc1066b0 0x7ff6bc19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 0x7ff6bc198470 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00e9f0 tx=0x7ff6ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6b2ffd640 1 -- 172.21.15.27:0/1057080196 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac00cd80 con 0x7ff6bc100fd0 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6c4148640 1 -- 172.21.15.27:0/1057080196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6bc19d210 con 0x7ff6bc100fd0 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6c4148640 1 -- 172.21.15.27:0/1057080196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6bc1a4aa0 con 0x7ff6bc100fd0 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6b2ffd640 1 -- 172.21.15.27:0/1057080196 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6ac005500 con 0x7ff6bc100fd0 2024-04-25T01:26:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.755+0000 7ff6b2ffd640 1 -- 172.21.15.27:0/1057080196 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac010690 con 0x7ff6bc100fd0 2024-04-25T01:26:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.758+0000 7ff6b2ffd640 1 -- 172.21.15.27:0/1057080196 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7ff6ac0046c0 con 0x7ff6bc100fd0 2024-04-25T01:26:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.758+0000 7ff6c6e4b740 1 Processor -- start 2024-04-25T01:26:36.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6c6e4b740 1 -- start start 2024-04-25T01:26:36.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6b2ffd640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6a80a2f30 0x7ff6a80a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a22840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a31440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a31980 0x55f5b0a33d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f5b0987670 con 0x55f5b0a22d80 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f5b092f970 con 0x55f5b0a22460 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f5b0983630 con 0x55f5b0a31980 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6a80a2f30 0x7ff6a80a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.759+0000 7ff6b2ffd640 1 -- 172.21.15.27:0/1057080196 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff6ac0cfef0 con 0x7ff6bc100fd0 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a31440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a31440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39730/0 (socket says 172.21.15.27:39730) 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 -- 172.21.15.27:0/4228841959 learned_addr learned my addr 172.21.15.27:0/4228841959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/4228841959 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a22840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 -- 172.21.15.27:0/4228841959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a31980 msgr2=0x55f5b0a33d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/4228841959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a31980 0x55f5b0a33d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 -- 172.21.15.27:0/4228841959 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 msgr2=0x55f5b0a22840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/4228841959 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a22840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 -- 172.21.15.27:0/4228841959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5b0953400 con 0x55f5b0a22d80 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6a80a2f30 0x7ff6a80a53f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4005e30 tx=0x7ff6b4009800 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/4228841959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a31440 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac09bb40 tx=0x7ff6ac09b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0f75e47bcfd536 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.760+0000 7ff6a7fff640 1 -- 172.21.15.27:0/4228841959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac09e070 con 0x55f5b0a22d80 2024-04-25T01:26:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6a7fff640 1 -- 172.21.15.27:0/4228841959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6ac0a01f0 con 0x55f5b0a22d80 2024-04-25T01:26:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6a7fff640 1 -- 172.21.15.27:0/4228841959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac09b4f0 con 0x55f5b0a22d80 2024-04-25T01:26:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/4228841959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 msgr2=0x55f5b0a31440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/4228841959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a31440 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac09bb40 tx=0x7ff6ac09b7a0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/4228841959 shutdown_connections 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/4228841959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a31980 0x55f5b0a33d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/4228841959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a31440 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/4228841959 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a22840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.761+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/4228841959 >> 172.21.15.27:0/4228841959 conn(0x55f5b0a1e8a0 msgr2=0x55f5b0a1ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.762+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/4228841959 shutdown_connections 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.762+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/4228841959 wait complete. 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.762+0000 7ff6c6e4b740 1 -- start start 2024-04-25T01:26:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.762+0000 7ff6c6e4b740 1 Processor -- start 2024-04-25T01:26:36.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a31300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a2e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a31980 0x55f5b0a2eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f5b0a347c0 con 0x55f5b0a31980 2024-04-25T01:26:36.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f5b0943050 con 0x55f5b0a22460 2024-04-25T01:26:36.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f5b0a34940 con 0x55f5b0a22d80 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c6e4b740 10 client.0 _ll_register_callbacks cb 0x55f5b069abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a31980 0x55f5b0a2eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a31980 0x55f5b0a2eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39746/0 (socket says 172.21.15.27:39746) 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 -- 172.21.15.27:0/3501283005 learned_addr learned my addr 172.21.15.27:0/3501283005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a2e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a31300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 -- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 msgr2=0x55f5b0a2e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a2e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 -- 172.21.15.27:0/3501283005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 msgr2=0x55f5b0a31300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a31300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 -- 172.21.15.27:0/3501283005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5b0914dc0 con 0x55f5b0a31980 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a2e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a31300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a31980 0x55f5b0a2eb70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4031b40 tx=0x7ff6b40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c280c67420285e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.763+0000 7ff6a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6a57fa640 1 -- 172.21.15.27:0/3501283005 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b404b070 con 0x55f5b0a31980 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6a57fa640 1 -- 172.21.15.27:0/3501283005 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff6b4004740 con 0x55f5b0a31980 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/3501283005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6ac0a01f0 con 0x55f5b0a31980 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6a57fa640 1 -- 172.21.15.27:0/3501283005 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6b4031d80 con 0x55f5b0a31980 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6c6e4b740 10 client.15918 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/3501283005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f5b069b3f0 con 0x55f5b0a31980 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6a57fa640 1 -- 172.21.15.27:0/3501283005 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b4036040 con 0x55f5b0a31980 2024-04-25T01:26:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6a57fa640 1 -- 172.21.15.27:0/3501283005 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff6b4030070 con 0x55f5b0a31980 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6c6e4b740 10 client.15918 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6c6e4b740 10 client.15918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6c6e4b740 20 client.15918 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.764+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55f5b0a41ab0 0x55f5b0a43ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.765+0000 7ff6c6e4b740 4 client.15918 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.765+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/3501283005 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff6ac0a01f0 con 0x55f5b0a41ab0 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.765+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55f5b0a41ab0 0x55f5b0a43ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.765+0000 7ff6a57fa640 10 client.15918 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:26:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.765+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55f5b0a41ab0 0x55f5b0a43ee0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7cb801e23651bfa server_cookie=37bb114fd5efad97 in_seq=0 out_seq=0 2024-04-25T01:26:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6a57fa640 1 -- 172.21.15.27:0/3501283005 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ff6ac0a01f0 con 0x55f5b0a41ab0 2024-04-25T01:26:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6a57fa640 10 client.15918 handle_command_reply: tid=0 2024-04-25T01:26:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 client.15918 shutdown 2024-04-25T01:26:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/3501283005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a31980 msgr2=0x55f5b0a2eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a31980 0x55f5b0a2eb70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4031b40 tx=0x7ff6b40095c0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/3501283005 shutdown_connections 2024-04-25T01:26:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55f5b0a41ab0 0x55f5b0a43ee0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a31980 0x55f5b0a2eb70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a2e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/3501283005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a22460 0x55f5b0a31300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6a57fa640 0 client.15918 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:26:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/3501283005 >> 172.21.15.27:0/3501283005 conn(0x55f5b0a1e8a0 msgr2=0x55f5b0a24650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/3501283005 shutdown_connections 2024-04-25T01:26:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/3501283005 wait complete. 2024-04-25T01:26:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.766+0000 7ff6c6e4b740 20 client.15918 trim_cache size 0 max 16384 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c6e4b740 1 Processor -- start 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c6e4b740 1 -- start start 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22460 0x55f5b0a3d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a1ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a1db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc1af2c0 con 0x55f5b0a22d80 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc1acb90 con 0x55f5b0a31980 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c6e4b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc1aa9e0 con 0x55f5b0a22460 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c16e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a1ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c16e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a1ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39760/0 (socket says 172.21.15.27:39760) 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c16e3640 1 -- 172.21.15.27:0/94868442 learned_addr learned my addr 172.21.15.27:0/94868442 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22460 0x55f5b0a3d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a1db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c16e3640 1 -- 172.21.15.27:0/94868442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22460 msgr2=0x55f5b0a3d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22460 0x55f5b0a3d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c16e3640 1 -- 172.21.15.27:0/94868442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 msgr2=0x55f5b0a1db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a1db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.780+0000 7ff6c16e3640 1 -- 172.21.15.27:0/94868442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5b07657d0 con 0x55f5b0a22d80 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.781+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a1db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.781+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a1ec80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac09bdd0 tx=0x7ff6ac0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5b317ffab3a079 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.781+0000 7ff6a77fe640 1 -- 172.21.15.27:0/94868442 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac09e040 con 0x55f5b0a22d80 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.781+0000 7ff6a77fe640 1 -- 172.21.15.27:0/94868442 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6ac0a01f0 con 0x55f5b0a22d80 2024-04-25T01:26:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.781+0000 7ff6a77fe640 1 -- 172.21.15.27:0/94868442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac09af60 con 0x55f5b0a22d80 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/94868442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 msgr2=0x55f5b0a1ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a1ec80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac09bdd0 tx=0x7ff6ac0057b0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/94868442 shutdown_connections 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a1db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22d80 0x55f5b0a1ec80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/94868442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22460 0x55f5b0a3d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/94868442 >> 172.21.15.27:0/94868442 conn(0x55f5b0a1e8a0 msgr2=0x55f5b0a2de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/94868442 shutdown_connections 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/94868442 wait complete. 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 -- start start 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.782+0000 7ff6c6e4b740 1 Processor -- start 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c6e4b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22460 0x55f5b0a39b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22460 0x55f5b0a39b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22460 0x55f5b0a39b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39776/0 (socket says 172.21.15.27:39776) 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 -- 172.21.15.27:0/273799022 learned_addr learned my addr 172.21.15.27:0/273799022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a3a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a350c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc1ab050 con 0x55f5b0a22460 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc1ac4c0 con 0x55f5b0a31980 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc1ab720 con 0x55f5b0a22d80 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c6e4b740 10 client.0 _ll_register_callbacks cb 0x7ff6b403f050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a3a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c26e5640 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 -- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 msgr2=0x55f5b0a3a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a3a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 -- 172.21.15.27:0/273799022 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 msgr2=0x55f5b0a350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 -- 172.21.15.27:0/273799022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5b0a358c0 con 0x55f5b0a22460 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6c1ee4640 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22460 0x55f5b0a39b60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55f5b0a44720 tx=0x7ff6b8012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa28eea94215c19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.783+0000 7ff6a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6a57fa640 1 -- 172.21.15.27:0/273799022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8023070 con 0x55f5b0a22460 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6a57fa640 1 -- 172.21.15.27:0/273799022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff6b8005220 con 0x55f5b0a22460 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6ac0a01f0 con 0x55f5b0a22460 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6a57fa640 1 -- 172.21.15.27:0/273799022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6b802e050 con 0x55f5b0a22460 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 10 client.15924 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f5b0aa7bb0 con 0x55f5b0a22460 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6a57fa640 1 -- 172.21.15.27:0/273799022 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6ac0a01f0 con 0x55f5b0a22460 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6a57fa640 1 -- 172.21.15.27:0/273799022 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff6b8004950 con 0x55f5b0a22460 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 10 client.15924 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 10 client.15924 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 20 client.15924 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55f5b0aa8310 0x55f5b0aaa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 4 client.15924 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.784+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f5b0994bb0 con 0x55f5b0aa8310 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55f5b0aa8310 0x55f5b0aaa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c16e3640 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55f5b0aa8310 0x55f5b0aaa730 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f84a4ce9cae033b server_cookie=d8375f53d5ccf88e in_seq=0 out_seq=0 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6a57fa640 10 client.15924 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6a57fa640 1 -- 172.21.15.27:0/273799022 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f5b0994bb0 con 0x55f5b0aa8310 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6a57fa640 10 client.15924 handle_command_reply: tid=0 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 client.15924 shutdown 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22460 msgr2=0x55f5b0a39b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22460 0x55f5b0a39b60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55f5b0a44720 tx=0x7ff6b8012ae0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 shutdown_connections 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55f5b0aa8310 0x55f5b0aaa730 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f5b0a31980 0x55f5b0a350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5b0a22d80 0x55f5b0a3a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 --2- 172.21.15.27:0/273799022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5b0a22460 0x55f5b0a39b60 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.785+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 >> 172.21.15.27:0/273799022 conn(0x55f5b0a1e8a0 msgr2=0x55f5b0a1db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6a57fa640 0 client.15924 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:26:36.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 shutdown_connections 2024-04-25T01:26:36.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c6e4b740 1 -- 172.21.15.27:0/273799022 wait complete. 2024-04-25T01:26:36.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c6e4b740 20 client.15924 trim_cache size 0 max 16384 2024-04-25T01:26:36.771 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 -- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6a80a2f30 msgr2=0x7ff6a80a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6a80a2f30 0x7ff6a80a53f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4005e30 tx=0x7ff6b4009800 comp rx=0 tx=0).stop 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 -- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 msgr2=0x7ff6bc198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 0x7ff6bc198470 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00e9f0 tx=0x7ff6ac00edf0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 -- 172.21.15.27:0/1057080196 shutdown_connections 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc1066b0 0x7ff6bc19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6a80a2f30 0x7ff6a80a53f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6bc100fd0 0x7ff6bc198470 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 --2- 172.21.15.27:0/1057080196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6bc100610 0x7ff6bc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 -- 172.21.15.27:0/1057080196 >> 172.21.15.27:0/1057080196 conn(0x7ff6bc077740 msgr2=0x7ff6bc0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 -- 172.21.15.27:0/1057080196 shutdown_connections 2024-04-25T01:26:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.786+0000 7ff6c4148640 1 -- 172.21.15.27:0/1057080196 wait complete. 2024-04-25T01:26:36.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:36 smithi153 ceph-mon[32621]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:36.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:36 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3464197783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:36.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:26:36.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:36 smithi139 ceph-mon[32760]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:36.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:36 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3464197783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 -- 172.21.15.27:0/1438948692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae701066b0 msgr2=0x7fae70106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 --2- 172.21.15.27:0/1438948692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae701066b0 0x7fae70106a90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009930 tx=0x7fae6c02a420 comp rx=0 tx=0).stop 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 -- 172.21.15.27:0/1438948692 shutdown_connections 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 --2- 172.21.15.27:0/1438948692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae70100fd0 0x7fae7010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 --2- 172.21.15.27:0/1438948692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae70100610 0x7fae70100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 --2- 172.21.15.27:0/1438948692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae701066b0 0x7fae70106a90 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 -- 172.21.15.27:0/1438948692 >> 172.21.15.27:0/1438948692 conn(0x7fae70077740 msgr2=0x7fae70077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 -- 172.21.15.27:0/1438948692 shutdown_connections 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.903+0000 7fae77563640 1 -- 172.21.15.27:0/1438948692 wait complete. 2024-04-25T01:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.904+0000 7fae77563640 1 Processor -- start 2024-04-25T01:26:36.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.904+0000 7fae77563640 1 -- start start 2024-04-25T01:26:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.904+0000 7fae77563640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae70100610 0x7fae70196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.904+0000 7fae77563640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae70100fd0 0x7fae70197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae77563640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae701066b0 0x7fae7019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae77563640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae70110a30 con 0x7fae70100610 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae77563640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae701108b0 con 0x7fae70100fd0 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae77563640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae70110bb0 con 0x7fae701066b0 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae70100610 0x7fae70196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae70100610 0x7fae70196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39786/0 (socket says 172.21.15.27:39786) 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 -- 172.21.15.27:0/2615447060 learned_addr learned my addr 172.21.15.27:0/2615447060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae74afe640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae70100fd0 0x7fae70197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae75b00640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae701066b0 0x7fae7019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 -- 172.21.15.27:0/2615447060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae701066b0 msgr2=0x7fae7019b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae701066b0 0x7fae7019b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 -- 172.21.15.27:0/2615447060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae70100fd0 msgr2=0x7fae70197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae70100fd0 0x7fae70197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 -- 172.21.15.27:0/2615447060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7019be70 con 0x7fae70100610 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae74afe640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae70100fd0 0x7fae70197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.905+0000 7fae752ff640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae70100610 0x7fae70196ed0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fae6c005940 tx=0x7fae6c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.906+0000 7fae667fc640 1 -- 172.21.15.27:0/2615447060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c03f030 con 0x7fae70100610 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.906+0000 7fae77563640 1 -- 172.21.15.27:0/2615447060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae7019c100 con 0x7fae70100610 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.906+0000 7fae667fc640 1 -- 172.21.15.27:0/2615447060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae6c003ce0 con 0x7fae70100610 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.906+0000 7fae77563640 1 -- 172.21.15.27:0/2615447060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae700688d0 con 0x7fae70100610 2024-04-25T01:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.906+0000 7fae667fc640 1 -- 172.21.15.27:0/2615447060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c035070 con 0x7fae70100610 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.908+0000 7fae7a277740 1 Processor -- start 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.908+0000 7fae7a277740 1 -- start start 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae7a277740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afe8f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae7a277740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afe9a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae7a277740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe9c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae7a277740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5566afdf4670 con 0x5566afe8fea0 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae7a277740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5566afd9c970 con 0x5566afe9a570 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae7a277740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5566afdf0630 con 0x5566afe8f580 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afe8f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afe8f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:42252/0 (socket says 172.21.15.27:42252) 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 -- 172.21.15.27:0/3157531056 learned_addr learned my addr 172.21.15.27:0/3157531056 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae74afe640 1 --2- 172.21.15.27:0/3157531056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe9c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 -- 172.21.15.27:0/3157531056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 msgr2=0x5566afe9c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 --2- 172.21.15.27:0/3157531056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe9c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 -- 172.21.15.27:0/3157531056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 msgr2=0x5566afe9a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 --2- 172.21.15.27:0/3157531056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afe9a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 -- 172.21.15.27:0/3157531056 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566afdc0400 con 0x5566afe8f580 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.909+0000 7fae75b00640 1 --2- 172.21.15.27:0/3157531056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afe8f960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae68009ee0 tx=0x7fae68010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e32955d7d1062ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.910+0000 7fae5b7fe640 1 -- 172.21.15.27:0/3157531056 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68016070 con 0x5566afe8f580 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.910+0000 7fae5b7fe640 1 -- 172.21.15.27:0/3157531056 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae68004d20 con 0x5566afe8f580 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.910+0000 7fae667fc640 1 -- 172.21.15.27:0/2615447060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7fae6c042030 con 0x7fae70100610 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.910+0000 7fae5b7fe640 1 -- 172.21.15.27:0/3157531056 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae680127b0 con 0x5566afe8f580 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.910+0000 7fae7a277740 1 -- 172.21.15.27:0/3157531056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 msgr2=0x5566afe8f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.910+0000 7fae7a277740 1 --2- 172.21.15.27:0/3157531056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afe8f960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae68009ee0 tx=0x7fae68010740 comp rx=0 tx=0).stop 2024-04-25T01:26:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 -- 172.21.15.27:0/3157531056 shutdown_connections 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 --2- 172.21.15.27:0/3157531056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe9c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 --2- 172.21.15.27:0/3157531056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afe9a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 --2- 172.21.15.27:0/3157531056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afe8f960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 -- 172.21.15.27:0/3157531056 >> 172.21.15.27:0/3157531056 conn(0x5566afe8b8a0 msgr2=0x5566afe8bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 -- 172.21.15.27:0/3157531056 shutdown_connections 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae667fc640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fae5c0a2dc0 0x7fae5c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 -- 172.21.15.27:0/3157531056 wait complete. 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae74afe640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fae5c0a2dc0 0x7fae5c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 -- start start 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae667fc640 1 -- 172.21.15.27:0/2615447060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fae6c0ef7a0 con 0x7fae70100610 2024-04-25T01:26:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.911+0000 7fae7a277740 1 Processor -- start 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae74afe640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fae5c0a2dc0 0x7fae5c0a5280 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5566afe8ca30 tx=0x7fae6000b390 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae7a277740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8f580 0x5566afe99ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae7a277740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8fea0 0x5566afe97190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae7a277740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe976d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae7a277740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5566afe9d3b0 con 0x5566afe8f580 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae7a277740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5566afdb0050 con 0x5566afe9a570 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae7a277740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5566afe9d530 con 0x5566afe8fea0 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae7a277740 10 client.0 _ll_register_callbacks cb 0x5566afb07be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae752ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8f580 0x5566afe99ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae75b00640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae75b00640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:45904/0 (socket says 172.21.15.27:45904) 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae75b00640 1 -- 172.21.15.27:0/1332441129 learned_addr learned my addr 172.21.15.27:0/1332441129 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae74afe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8fea0 0x5566afe97190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae75b00640 1 -- 172.21.15.27:0/1332441129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8fea0 msgr2=0x5566afe97190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae75b00640 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8fea0 0x5566afe97190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.912+0000 7fae75b00640 1 -- 172.21.15.27:0/1332441129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8f580 msgr2=0x5566afe99ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae75b00640 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8f580 0x5566afe99ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae75b00640 1 -- 172.21.15.27:0/1332441129 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566afd81dc0 con 0x5566afe9a570 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae752ff640 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8f580 0x5566afe99ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae75b00640 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe976d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fae68005b10 tx=0x7fae68004a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2621fbdf69b6d785 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae58ff9640 1 -- 172.21.15.27:0/1332441129 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68016050 con 0x5566afe9a570 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae7a277740 1 -- 172.21.15.27:0/1332441129 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae68004d20 con 0x5566afe9a570 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae58ff9640 1 -- 172.21.15.27:0/1332441129 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fae680050e0 con 0x5566afe9a570 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae58ff9640 1 -- 172.21.15.27:0/1332441129 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fae6802f050 con 0x5566afe9a570 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae7a277740 10 client.25472 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:36.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.913+0000 7fae7a277740 1 -- 172.21.15.27:0/1332441129 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5566afb083f0 con 0x5566afe9a570 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.914+0000 7fae58ff9640 1 -- 172.21.15.27:0/1332441129 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae68004d20 con 0x5566afe9a570 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.914+0000 7fae58ff9640 1 -- 172.21.15.27:0/1332441129 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fae6800f040 con 0x5566afe9a570 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.914+0000 7fae7a277740 10 client.25472 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.914+0000 7fae7a277740 10 client.25472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.914+0000 7fae7a277740 20 client.25472 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.914+0000 7fae7a277740 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566afeb2da0 0x5566afeb5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.914+0000 7fae7a277740 4 client.25472 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.914+0000 7fae7a277740 1 -- 172.21.15.27:0/1332441129 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566afe01bb0 con 0x5566afeb2da0 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.915+0000 7fae752ff640 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566afeb2da0 0x5566afeb5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.915+0000 7fae58ff9640 10 client.25472 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.915+0000 7fae752ff640 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566afeb2da0 0x5566afeb5190 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f81125517b114ea6 server_cookie=468e615ec0fccfc9 in_seq=0 out_seq=0 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae58ff9640 1 -- 172.21.15.27:0/1332441129 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566afe01bb0 con 0x5566afeb2da0 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae58ff9640 10 client.25472 handle_command_reply: tid=0 2024-04-25T01:26:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 client.25472 shutdown 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 -- 172.21.15.27:0/1332441129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 msgr2=0x5566afe976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe976d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fae68005b10 tx=0x7fae68004a60 comp rx=0 tx=0).stop 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 -- 172.21.15.27:0/1332441129 shutdown_connections 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566afeb2da0 0x5566afeb5190 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe9a570 0x5566afe976d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae58ff9640 0 client.25472 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8fea0 0x5566afe97190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 --2- 172.21.15.27:0/1332441129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8f580 0x5566afe99ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 -- 172.21.15.27:0/1332441129 >> 172.21.15.27:0/1332441129 conn(0x5566afe8b8a0 msgr2=0x5566afe916a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 -- 172.21.15.27:0/1332441129 shutdown_connections 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 1 -- 172.21.15.27:0/1332441129 wait complete. 2024-04-25T01:26:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.916+0000 7fae7a277740 20 client.25472 trim_cache size 0 max 16384 2024-04-25T01:26:36.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae7a277740 1 Processor -- start 2024-04-25T01:26:36.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae7a277740 1 -- start start 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae7a277740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8f580 0x5566afeba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae7a277740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afebacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae7a277740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe9a570 0x5566afe8a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae7a277740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae701af200 con 0x5566afe8fea0 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae7a277740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae701acad0 con 0x5566afe8f580 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae7a277740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae701aa920 con 0x5566afe9a570 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae74afe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afebacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae74afe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afebacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39794/0 (socket says 172.21.15.27:39794) 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae74afe640 1 -- 172.21.15.27:0/632909771 learned_addr learned my addr 172.21.15.27:0/632909771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae752ff640 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8f580 0x5566afeba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae75b00640 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe9a570 0x5566afe8a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae74afe640 1 -- 172.21.15.27:0/632909771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe9a570 msgr2=0x5566afe8a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae74afe640 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe9a570 0x5566afe8a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae74afe640 1 -- 172.21.15.27:0/632909771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8f580 msgr2=0x5566afeba790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae74afe640 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8f580 0x5566afeba790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae74afe640 1 -- 172.21.15.27:0/632909771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566afe97100 con 0x5566afe8fea0 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.930+0000 7fae75b00640 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe9a570 0x5566afe8a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.931+0000 7fae74afe640 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afebacd0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fae6001f4a0 tx=0x7fae60019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a855ec04a09eff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.931+0000 7fae5affd640 1 -- 172.21.15.27:0/632909771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae60004020 con 0x5566afe8fea0 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.931+0000 7fae5affd640 1 -- 172.21.15.27:0/632909771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae600041c0 con 0x5566afe8fea0 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.931+0000 7fae5affd640 1 -- 172.21.15.27:0/632909771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae60028820 con 0x5566afe8fea0 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 -- 172.21.15.27:0/632909771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 msgr2=0x5566afebacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afebacd0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fae6001f4a0 tx=0x7fae60019000 comp rx=0 tx=0).stop 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 -- 172.21.15.27:0/632909771 shutdown_connections 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe9a570 0x5566afe8a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe8fea0 0x5566afebacd0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 --2- 172.21.15.27:0/632909771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8f580 0x5566afeba790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 -- 172.21.15.27:0/632909771 >> 172.21.15.27:0/632909771 conn(0x5566afe8b8a0 msgr2=0x5566afe8a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 -- 172.21.15.27:0/632909771 shutdown_connections 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 -- 172.21.15.27:0/632909771 wait complete. 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 -- start start 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.932+0000 7fae7a277740 1 Processor -- start 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae7a277740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afeab120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae7a277740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8fea0 0x5566afea85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae7a277740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe9a570 0x5566afea8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae7a277740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae701aaf90 con 0x5566afe9a570 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae7a277740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fae701ac400 con 0x5566afe8fea0 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae7a277740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae701ab660 con 0x5566afe8f580 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae7a277740 10 client.0 _ll_register_callbacks cb 0x7fae6000bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae74afe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8fea0 0x5566afea85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae74afe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8fea0 0x5566afea85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:45916/0 (socket says 172.21.15.27:45916) 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae74afe640 1 -- 172.21.15.27:0/4087107566 learned_addr learned my addr 172.21.15.27:0/4087107566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae75b00640 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe9a570 0x5566afea8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae752ff640 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afeab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae74afe640 1 -- 172.21.15.27:0/4087107566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 msgr2=0x5566afeab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae74afe640 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afeab120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae74afe640 1 -- 172.21.15.27:0/4087107566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe9a570 msgr2=0x5566afea8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae74afe640 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe9a570 0x5566afea8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.933+0000 7fae74afe640 1 -- 172.21.15.27:0/4087107566 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566afbd27d0 con 0x5566afe8fea0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae75b00640 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe9a570 0x5566afea8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae74afe640 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8fea0 0x5566afea85e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fae6000bfd0 tx=0x7fae60004090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b83ffb34000d0337 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae58ff9640 1 -- 172.21.15.27:0/4087107566 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae60037060 con 0x5566afe8fea0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae7a277740 1 -- 172.21.15.27:0/4087107566 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae600041c0 con 0x5566afe8fea0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae58ff9640 1 -- 172.21.15.27:0/4087107566 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fae60003800 con 0x5566afe8fea0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae58ff9640 1 -- 172.21.15.27:0/4087107566 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fae6003e050 con 0x5566afe8fea0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae7a277740 10 client.25478 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae7a277740 1 -- 172.21.15.27:0/4087107566 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5566aff18b60 con 0x5566afe8fea0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.934+0000 7fae58ff9640 1 -- 172.21.15.27:0/4087107566 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae600041c0 con 0x5566afe8fea0 2024-04-25T01:26:36.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae58ff9640 1 -- 172.21.15.27:0/4087107566 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fae60028bd0 con 0x5566afe8fea0 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae7a277740 10 client.25478 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae7a277740 10 client.25478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae7a277740 20 client.25478 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae7a277740 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566aff19250 0x5566aff1b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae7a277740 4 client.25478 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae7a277740 1 -- 172.21.15.27:0/4087107566 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566afdfbf00 con 0x5566aff19250 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae752ff640 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566aff19250 0x5566aff1b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae58ff9640 10 client.25478 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.935+0000 7fae752ff640 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566aff19250 0x5566aff1b670 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=266f9000b2f08c5e server_cookie=b968e8a5c762fd5c in_seq=0 out_seq=0 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae58ff9640 1 -- 172.21.15.27:0/4087107566 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566afdfbf00 con 0x5566aff19250 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae58ff9640 10 client.25478 handle_command_reply: tid=0 2024-04-25T01:26:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 client.25478 shutdown 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 -- 172.21.15.27:0/4087107566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8fea0 msgr2=0x5566afea85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8fea0 0x5566afea85e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fae6000bfd0 tx=0x7fae60004090 comp rx=0 tx=0).stop 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 -- 172.21.15.27:0/4087107566 shutdown_connections 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566aff19250 0x5566aff1b670 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566afe9a570 0x5566afea8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566afe8fea0 0x5566afea85e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae58ff9640 0 client.25478 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 --2- 172.21.15.27:0/4087107566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566afe8f580 0x5566afeab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 -- 172.21.15.27:0/4087107566 >> 172.21.15.27:0/4087107566 conn(0x5566afe8b8a0 msgr2=0x5566afe8ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 -- 172.21.15.27:0/4087107566 shutdown_connections 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 1 -- 172.21.15.27:0/4087107566 wait complete. 2024-04-25T01:26:36.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.936+0000 7fae7a277740 20 client.25478 trim_cache size 0 max 16384 2024-04-25T01:26:36.924 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 -- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fae5c0a2dc0 msgr2=0x7fae5c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fae5c0a2dc0 0x7fae5c0a5280 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5566afe8ca30 tx=0x7fae6000b390 comp rx=0 tx=0).stop 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 -- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae70100610 msgr2=0x7fae70196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae70100610 0x7fae70196ed0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fae6c005940 tx=0x7fae6c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 -- 172.21.15.27:0/2615447060 shutdown_connections 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae701066b0 0x7fae7019b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae70100fd0 0x7fae70197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fae5c0a2dc0 0x7fae5c0a5280 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 --2- 172.21.15.27:0/2615447060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae70100610 0x7fae70196ed0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 -- 172.21.15.27:0/2615447060 >> 172.21.15.27:0/2615447060 conn(0x7fae70077740 msgr2=0x7fae700ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 -- 172.21.15.27:0/2615447060 shutdown_connections 2024-04-25T01:26:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:36.937+0000 7fae77563640 1 -- 172.21.15.27:0/2615447060 wait complete. 2024-04-25T01:26:37.929 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:26:37.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:26:37.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.116+0000 7f5800ae1640 1 -- 172.21.15.27:0/3914826064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57fc102150 msgr2=0x7f57fc10e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.116+0000 7f5800ae1640 1 --2- 172.21.15.27:0/3914826064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57fc102150 0x7f57fc10e660 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009cd0 tx=0x7f57f402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.117+0000 7f5800ae1640 1 -- 172.21.15.27:0/3914826064 shutdown_connections 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.117+0000 7f5800ae1640 1 --2- 172.21.15.27:0/3914826064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57fc102150 0x7f57fc10e660 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.117+0000 7f5800ae1640 1 --2- 172.21.15.27:0/3914826064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57fc101790 0x7f57fc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.117+0000 7f5800ae1640 1 --2- 172.21.15.27:0/3914826064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57fc1077a0 0x7f57fc107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.117+0000 7f5800ae1640 1 -- 172.21.15.27:0/3914826064 >> 172.21.15.27:0/3914826064 conn(0x7f57fc0fd2a0 msgr2=0x7f57fc0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.117+0000 7f5800ae1640 1 -- 172.21.15.27:0/3914826064 shutdown_connections 2024-04-25T01:26:38.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.117+0000 7f5800ae1640 1 -- 172.21.15.27:0/3914826064 wait complete. 2024-04-25T01:26:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f5800ae1640 1 Processor -- start 2024-04-25T01:26:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f5800ae1640 1 -- start start 2024-04-25T01:26:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f5800ae1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57fc101790 0x7f57fc1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f5800ae1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57fc102150 0x7f57fc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f5800ae1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57fc1077a0 0x7f57fc19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f5800ae1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc111bb0 con 0x7f57fc1077a0 2024-04-25T01:26:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f5800ae1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc111a30 con 0x7f57fc102150 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f5800ae1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc111d30 con 0x7f57fc101790 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f57fa59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57fc101790 0x7f57fc1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f57f9d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57fc102150 0x7f57fc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f57fa59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57fc101790 0x7f57fc1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:48976/0 (socket says 172.21.15.27:48976) 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f57fa59c640 1 -- 172.21.15.27:0/442373833 learned_addr learned my addr 172.21.15.27:0/442373833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.118+0000 7f57fad9d640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57fc1077a0 0x7f57fc19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57fa59c640 1 -- 172.21.15.27:0/442373833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57fc102150 msgr2=0x7f57fc1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57fa59c640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57fc102150 0x7f57fc1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57fa59c640 1 -- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57fc1077a0 msgr2=0x7f57fc19c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57fa59c640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57fc1077a0 0x7f57fc19c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57fa59c640 1 -- 172.21.15.27:0/442373833 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57fc19d0a0 con 0x7f57fc101790 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57fc102150 0x7f57fc1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57fa59c640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57fc101790 0x7f57fc1980b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00e9f0 tx=0x7f57ec00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57eb7fe640 1 -- 172.21.15.27:0/442373833 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec00cd80 con 0x7f57fc101790 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f5800ae1640 1 -- 172.21.15.27:0/442373833 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57fc19d390 con 0x7f57fc101790 2024-04-25T01:26:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57eb7fe640 1 -- 172.21.15.27:0/442373833 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57ec005500 con 0x7f57fc101790 2024-04-25T01:26:38.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f57eb7fe640 1 -- 172.21.15.27:0/442373833 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec010690 con 0x7f57fc101790 2024-04-25T01:26:38.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.119+0000 7f5800ae1640 1 -- 172.21.15.27:0/442373833 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57fc1a4c20 con 0x7f57fc101790 2024-04-25T01:26:38.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.122+0000 7f57eb7fe640 1 -- 172.21.15.27:0/442373833 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7f57ec0046c0 con 0x7f57fc101790 2024-04-25T01:26:38.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.122+0000 7f57eb7fe640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57e40a2eb0 0x7f57e40a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.123+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57e40a2eb0 0x7f57e40a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.123+0000 7f57eb7fe640 1 -- 172.21.15.27:0/442373833 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f57ec0cfe70 con 0x7f57fc101790 2024-04-25T01:26:38.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.123+0000 7f58037f5740 1 Processor -- start 2024-04-25T01:26:38.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.123+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57e40a2eb0 0x7f57e40a5370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0ff910 tx=0x7f57f0005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.123+0000 7f58037f5740 1 -- start start 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.123+0000 7f58037f5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c20cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.123+0000 7f58037f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c21290 0x559cb7c21710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f58037f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c2e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c21290 0x559cb7c21710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c21290 0x559cb7c21710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41338/0 (socket says 172.21.15.27:41338) 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 -- 172.21.15.27:0/3515406201 learned_addr learned my addr 172.21.15.27:0/3515406201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f58037f5740 1 -- 172.21.15.27:0/3515406201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559cb7b852c0 con 0x559cb7c21290 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f58037f5740 1 -- 172.21.15.27:0/3515406201 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559cb7b2d910 con 0x559cb7c22980 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f58037f5740 1 -- 172.21.15.27:0/3515406201 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559cb7b81280 con 0x559cb7c2c010 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57fa59c640 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c20cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57fad9d640 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c2e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 -- 172.21.15.27:0/3515406201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 msgr2=0x559cb7c2e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c2e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 -- 172.21.15.27:0/3515406201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 msgr2=0x559cb7c20cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c20cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 -- 172.21.15.27:0/3515406201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb7b3efb0 con 0x559cb7c21290 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57fad9d640 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c2e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.124+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c21290 0x559cb7c21710 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009ba0 tx=0x7f57f0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f891ba1aec5f6de9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.125+0000 7f57e8ff9640 1 -- 172.21.15.27:0/3515406201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f00044c0 con 0x559cb7c21290 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.125+0000 7f57e8ff9640 1 -- 172.21.15.27:0/3515406201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f000f960 con 0x559cb7c21290 2024-04-25T01:26:38.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.125+0000 7f57e8ff9640 1 -- 172.21.15.27:0/3515406201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f001cb00 con 0x559cb7c21290 2024-04-25T01:26:38.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 -- 172.21.15.27:0/3515406201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c21290 msgr2=0x559cb7c21710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c21290 0x559cb7c21710 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009ba0 tx=0x7f57f0014000 comp rx=0 tx=0).stop 2024-04-25T01:26:38.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 -- 172.21.15.27:0/3515406201 shutdown_connections 2024-04-25T01:26:38.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c2e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c21290 0x559cb7c21710 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 --2- 172.21.15.27:0/3515406201 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c20cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 -- 172.21.15.27:0/3515406201 >> 172.21.15.27:0/3515406201 conn(0x559cb7be9950 msgr2=0x559cb7c1d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 -- 172.21.15.27:0/3515406201 shutdown_connections 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.126+0000 7f58037f5740 1 -- 172.21.15.27:0/3515406201 wait complete. 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.127+0000 7f58037f5740 1 -- start start 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.127+0000 7f58037f5740 1 Processor -- start 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.127+0000 7f58037f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c21290 0x559cb7c28f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.127+0000 7f58037f5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c294b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.127+0000 7f58037f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c299f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57fad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c299f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57fa59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c21290 0x559cb7c28f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57fad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c299f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41352/0 (socket says 172.21.15.27:41352) 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57fad9d640 1 -- 172.21.15.27:0/136016425 learned_addr learned my addr 172.21.15.27:0/136016425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559cb7c2eea0 con 0x559cb7c2c010 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559cb7b40e90 con 0x559cb7c22980 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c294b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559cb7c2f020 con 0x559cb7c21290 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57f9d9b640 1 -- 172.21.15.27:0/136016425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c21290 msgr2=0x559cb7c28f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c21290 0x559cb7c28f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57f9d9b640 1 -- 172.21.15.27:0/136016425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 msgr2=0x559cb7c299f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c299f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57f9d9b640 1 -- 172.21.15.27:0/136016425 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb79617e0 con 0x559cb7c22980 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57fad9d640 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c299f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f58037f5740 10 client.0 _ll_register_callbacks cb 0x559cb7b7c970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:38.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57f000f960 con 0x559cb7c22980 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c294b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009e80 tx=0x7f57f0005510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb8946c554e94954 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.128+0000 7f57e1ffb640 10 client.25490 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f57e1ffb640 1 -- 172.21.15.27:0/136016425 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f0034060 con 0x559cb7c22980 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f57e1ffb640 1 -- 172.21.15.27:0/136016425 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f57f000f690 con 0x559cb7c22980 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f57e1ffb640 1 -- 172.21.15.27:0/136016425 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f57f000fb10 con 0x559cb7c22980 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f58037f5740 10 client.25490 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({fsmap=25}) -- 0x559cb79c1f70 con 0x559cb7c22980 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f57e1ffb640 1 -- 172.21.15.27:0/136016425 <== mon.2 v2:172.21.15.139:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f57f001a070 con 0x559cb7c22980 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f58037f5740 10 client.25490 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f58037f5740 10 client.25490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f58037f5740 20 client.25490 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f58037f5740 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559cb7c3bbc0 0x559cb7c3dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f58037f5740 4 client.25490 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.129+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cb7b595a0 con 0x559cb7c3bbc0 2024-04-25T01:26:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.130+0000 7f57fa59c640 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559cb7c3bbc0 0x559cb7c3dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.130+0000 7f57fa59c640 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559cb7c3bbc0 0x559cb7c3dfb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fb05478043286e7 server_cookie=3de2cf64f3b26ba in_seq=0 out_seq=0 2024-04-25T01:26:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.130+0000 7f57e1ffb640 10 client.25490 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f57e1ffb640 1 -- 172.21.15.27:0/136016425 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559cb7b595a0 con 0x559cb7c3bbc0 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f57e1ffb640 10 client.25490 handle_command_reply: tid=0 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 client.25490 shutdown 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 msgr2=0x559cb7c294b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c294b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009e80 tx=0x7f57f0005510 comp rx=0 tx=0).stop 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 shutdown_connections 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559cb7c3bbc0 0x559cb7c3dfb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c299f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c22980 0x559cb7c294b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 --2- 172.21.15.27:0/136016425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c21290 0x559cb7c28f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f57e1ffb640 0 client.25490 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 >> 172.21.15.27:0/136016425 conn(0x559cb7be9950 msgr2=0x559cb7c23660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 shutdown_connections 2024-04-25T01:26:38.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 1 -- 172.21.15.27:0/136016425 wait complete. 2024-04-25T01:26:38.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.131+0000 7f58037f5740 20 client.25490 trim_cache size 0 max 16384 2024-04-25T01:26:38.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.167+0000 7f58037f5740 1 Processor -- start 2024-04-25T01:26:38.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.167+0000 7f58037f5740 1 -- start start 2024-04-25T01:26:38.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.167+0000 7f58037f5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c37dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.167+0000 7f58037f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c22980 0x559cb7c1c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.167+0000 7f58037f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c435b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.167+0000 7f58037f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc1b14c0 con 0x559cb7c2c010 2024-04-25T01:26:38.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.167+0000 7f58037f5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc1aed90 con 0x559cb7c21290 2024-04-25T01:26:38.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.167+0000 7f58037f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc1acbe0 con 0x559cb7c22980 2024-04-25T01:26:38.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c435b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41356/0 (socket says 172.21.15.27:41356) 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fad9d640 1 -- 172.21.15.27:0/1890507704 learned_addr learned my addr 172.21.15.27:0/1890507704 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fa59c640 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c37dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c22980 0x559cb7c1c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fa59c640 1 -- 172.21.15.27:0/1890507704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c22980 msgr2=0x559cb7c1c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fa59c640 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c22980 0x559cb7c1c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fa59c640 1 -- 172.21.15.27:0/1890507704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 msgr2=0x559cb7c435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fa59c640 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fa59c640 1 -- 172.21.15.27:0/1890507704 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb7859170 con 0x559cb7c21290 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57fa59c640 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c37dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0ccff0 tx=0x7f57ec0cf040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf828c1e89ad13c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57e3fff640 1 -- 172.21.15.27:0/1890507704 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec0a9070 con 0x559cb7c21290 2024-04-25T01:26:38.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57e3fff640 1 -- 172.21.15.27:0/1890507704 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57ec0b8070 con 0x559cb7c21290 2024-04-25T01:26:38.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.168+0000 7f57e3fff640 1 -- 172.21.15.27:0/1890507704 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec0a4800 con 0x559cb7c21290 2024-04-25T01:26:38.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.169+0000 7f58037f5740 1 -- 172.21.15.27:0/1890507704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 msgr2=0x559cb7c37dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.169+0000 7f58037f5740 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c37dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0ccff0 tx=0x7f57ec0cf040 comp rx=0 tx=0).stop 2024-04-25T01:26:38.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.169+0000 7f58037f5740 1 -- 172.21.15.27:0/1890507704 shutdown_connections 2024-04-25T01:26:38.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.169+0000 7f58037f5740 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c2c010 0x559cb7c435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.169+0000 7f58037f5740 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c22980 0x559cb7c1c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.169+0000 7f58037f5740 1 --2- 172.21.15.27:0/1890507704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c37dd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.169+0000 7f58037f5740 1 -- 172.21.15.27:0/1890507704 >> 172.21.15.27:0/1890507704 conn(0x559cb7be9950 msgr2=0x559cb7c1c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 -- 172.21.15.27:0/1890507704 shutdown_connections 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 -- 172.21.15.27:0/1890507704 wait complete. 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 -- start start 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 Processor -- start 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c9d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c22980 0x559cb7c342e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c34820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc1ad250 con 0x559cb7c22980 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc1ae6c0 con 0x559cb7c21290 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc1ad920 con 0x559cb7c2c010 2024-04-25T01:26:38.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.170+0000 7f58037f5740 10 client.0 _ll_register_callbacks cb 0x7f57f00041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c22980 0x559cb7c342e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c22980 0x559cb7c342e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41362/0 (socket says 172.21.15.27:41362) 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 -- 172.21.15.27:0/3277696871 learned_addr learned my addr 172.21.15.27:0/3277696871 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57fad9d640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c34820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57fa59c640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c9d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 -- 172.21.15.27:0/3277696871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 msgr2=0x559cb7c34820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c34820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 -- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 msgr2=0x559cb7c9d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c9d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 -- 172.21.15.27:0/3277696871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb7c9a7e0 con 0x559cb7c22980 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57fa59c640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c9d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57fad9d640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c34820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57f9d9b640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c22980 0x559cb7c342e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f57f00058c0 tx=0x7f57f0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6650b2025e6c2bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:38.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57e1ffb640 1 -- 172.21.15.27:0/3277696871 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f000fd50 con 0x559cb7c22980 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f58037f5740 1 -- 172.21.15.27:0/3277696871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57ec0b8070 con 0x559cb7c22980 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.171+0000 7f57e1ffb640 1 -- 172.21.15.27:0/3277696871 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f57f001c710 con 0x559cb7c22980 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f57e1ffb640 1 -- 172.21.15.27:0/3277696871 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f57f0049050 con 0x559cb7c22980 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f58037f5740 10 client.15957 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f58037f5740 1 -- 172.21.15.27:0/3277696871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559cb7ca64f0 con 0x559cb7c22980 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f57e1ffb640 1 -- 172.21.15.27:0/3277696871 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57ec0b8070 con 0x559cb7c22980 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f57e1ffb640 1 -- 172.21.15.27:0/3277696871 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f57f001a040 con 0x559cb7c22980 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f58037f5740 10 client.15957 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f58037f5740 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f58037f5740 20 client.15957 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f58037f5740 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559cb7ca6bb0 0x559cb7ca8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f58037f5740 4 client.15957 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.172+0000 7f58037f5740 1 -- 172.21.15.27:0/3277696871 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559cb7b92800 con 0x559cb7ca6bb0 2024-04-25T01:26:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.173+0000 7f57fa59c640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559cb7ca6bb0 0x559cb7ca8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.173+0000 7f57e1ffb640 10 client.15957 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:38.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.173+0000 7f57fa59c640 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559cb7ca6bb0 0x559cb7ca8fd0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fe865c4054315d0 server_cookie=e0351321f825644c in_seq=0 out_seq=0 2024-04-25T01:26:38.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.174+0000 7f57e1ffb640 1 -- 172.21.15.27:0/3277696871 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559cb7b92800 con 0x559cb7ca6bb0 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.174+0000 7f57e1ffb640 10 client.15957 handle_command_reply: tid=0 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.174+0000 7f58037f5740 1 client.15957 shutdown 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 -- 172.21.15.27:0/3277696871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c22980 msgr2=0x559cb7c342e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c22980 0x559cb7c342e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f57f00058c0 tx=0x7f57f0005850 comp rx=0 tx=0).stop 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 -- 172.21.15.27:0/3277696871 shutdown_connections 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559cb7ca6bb0 0x559cb7ca8fd0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559cb7c2c010 0x559cb7c34820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559cb7c22980 0x559cb7c342e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 --2- 172.21.15.27:0/3277696871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559cb7c21290 0x559cb7c9d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f57e1ffb640 0 client.15957 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 -- 172.21.15.27:0/3277696871 >> 172.21.15.27:0/3277696871 conn(0x559cb7be9950 msgr2=0x559cb7c1c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 -- 172.21.15.27:0/3277696871 shutdown_connections 2024-04-25T01:26:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 1 -- 172.21.15.27:0/3277696871 wait complete. 2024-04-25T01:26:38.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.175+0000 7f58037f5740 20 client.15957 trim_cache size 0 max 16384 2024-04-25T01:26:38.159 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:26:38.159 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:26:38.159 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "89002673-7ec9-4026-89ac-456f9f56750e", 2024-04-25T01:26:38.159 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:26:38.159 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 -- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57e40a2eb0 msgr2=0x7f57e40a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57e40a2eb0 0x7f57e40a5370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0ff910 tx=0x7f57f0005f90 comp rx=0 tx=0).stop 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 -- 172.21.15.27:0/442373833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57fc101790 msgr2=0x7f57fc1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57fc101790 0x7f57fc1980b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00e9f0 tx=0x7f57ec00edf0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 -- 172.21.15.27:0/442373833 shutdown_connections 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57e40a2eb0 0x7f57e40a5370 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57fc1077a0 0x7f57fc19c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57fc102150 0x7f57fc1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 --2- 172.21.15.27:0/442373833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57fc101790 0x7f57fc1980b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 -- 172.21.15.27:0/442373833 >> 172.21.15.27:0/442373833 conn(0x7f57fc0fd2a0 msgr2=0x7f57fc0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 -- 172.21.15.27:0/442373833 shutdown_connections 2024-04-25T01:26:38.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.176+0000 7f5800ae1640 1 -- 172.21.15.27:0/442373833 wait complete. 2024-04-25T01:26:38.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:26:38.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.303+0000 7ff13d55a640 1 -- 172.21.15.27:0/1696882278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 msgr2=0x7ff13810dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.303+0000 7ff13d55a640 1 --2- 172.21.15.27:0/1696882278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 0x7ff13810dbb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009900 tx=0x7ff12c02a410 comp rx=0 tx=0).stop 2024-04-25T01:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.303+0000 7ff13d55a640 1 -- 172.21.15.27:0/1696882278 shutdown_connections 2024-04-25T01:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.303+0000 7ff13d55a640 1 --2- 172.21.15.27:0/1696882278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff13810e0f0 0x7ff1381104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.303+0000 7ff13d55a640 1 --2- 172.21.15.27:0/1696882278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 0x7ff13810dbb0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.303+0000 7ff13d55a640 1 --2- 172.21.15.27:0/1696882278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff138100540 0x7ff138100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.303+0000 7ff13d55a640 1 -- 172.21.15.27:0/1696882278 >> 172.21.15.27:0/1696882278 conn(0x7ff138077740 msgr2=0x7ff138077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.303+0000 7ff13d55a640 1 -- 172.21.15.27:0/1696882278 shutdown_connections 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 -- 172.21.15.27:0/1696882278 wait complete. 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 Processor -- start 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 -- start start 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff138100540 0x7ff13819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 0x7ff13819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff13810e0f0 0x7ff13819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff138113d40 con 0x7ff138100e60 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff138113bc0 con 0x7ff138100540 2024-04-25T01:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.304+0000 7ff13d55a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff138113ec0 con 0x7ff13810e0f0 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1367fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 0x7ff13819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1367fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 0x7ff13819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41380/0 (socket says 172.21.15.27:41380) 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1367fc640 1 -- 172.21.15.27:0/131323158 learned_addr learned my addr 172.21.15.27:0/131323158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1377fe640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff13810e0f0 0x7ff13819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff136ffd640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff138100540 0x7ff13819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1377fe640 1 -- 172.21.15.27:0/131323158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff138100540 msgr2=0x7ff13819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1377fe640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff138100540 0x7ff13819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1377fe640 1 -- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 msgr2=0x7ff13819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1377fe640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 0x7ff13819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1377fe640 1 -- 172.21.15.27:0/131323158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13819f190 con 0x7ff13810e0f0 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff136ffd640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff138100540 0x7ff13819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.305+0000 7ff1377fe640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff13810e0f0 0x7ff13819ea60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff130005fd0 tx=0x7ff13000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.306+0000 7ff11ffff640 1 -- 172.21.15.27:0/131323158 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130004d20 con 0x7ff13810e0f0 2024-04-25T01:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.306+0000 7ff13d55a640 1 -- 172.21.15.27:0/131323158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13819f480 con 0x7ff13810e0f0 2024-04-25T01:26:38.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.306+0000 7ff13d55a640 1 -- 172.21.15.27:0/131323158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1381a6d10 con 0x7ff13810e0f0 2024-04-25T01:26:38.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.306+0000 7ff11ffff640 1 -- 172.21.15.27:0/131323158 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff130005020 con 0x7ff13810e0f0 2024-04-25T01:26:38.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.306+0000 7ff11ffff640 1 -- 172.21.15.27:0/131323158 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1300116b0 con 0x7ff13810e0f0 2024-04-25T01:26:38.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.308+0000 7ff11ffff640 1 -- 172.21.15.27:0/131323158 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) ==== 138117+0+0 (secure 0 0 0) 0x7ff13000ca60 con 0x7ff13810e0f0 2024-04-25T01:26:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.309+0000 7ff11ffff640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1200a2f30 0x7ff1200a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.309+0000 7ff11ffff640 1 -- 172.21.15.27:0/131323158 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff1300cf930 con 0x7ff13810e0f0 2024-04-25T01:26:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.309+0000 7ff136ffd640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1200a2f30 0x7ff1200a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.309+0000 7ff140268740 1 Processor -- start 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.309+0000 7ff136ffd640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1200a2f30 0x7ff1200a53f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff12800afb0 tx=0x7ff12800a2d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.309+0000 7ff140268740 1 -- start start 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff140268740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca859380 0x55fcca859760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff140268740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca86c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff140268740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff140268740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fcca7be6b0 con 0x55fcca859380 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff140268740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fcca766880 con 0x55fcca859d30 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff140268740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fcca7ba670 con 0x55fcca86ce30 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca859380 0x55fcca859760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca859380 0x55fcca859760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41386/0 (socket says 172.21.15.27:41386) 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 -- 172.21.15.27:0/798370636 learned_addr learned my addr 172.21.15.27:0/798370636 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff1377fe640 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff1367fc640 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca86c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 -- 172.21.15.27:0/798370636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 msgr2=0x55fcca86f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 -- 172.21.15.27:0/798370636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 msgr2=0x55fcca86c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca86c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 -- 172.21.15.27:0/798370636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcca59c7d0 con 0x55fcca859380 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff1367fc640 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca86c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.310+0000 7ff136ffd640 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca859380 0x55fcca859760 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff12800a5c0 tx=0x7ff128012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcb07f755c4b8c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff11d7fa640 1 -- 172.21.15.27:0/798370636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128016070 con 0x55fcca859380 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff11d7fa640 1 -- 172.21.15.27:0/798370636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12801bb40 con 0x55fcca859380 2024-04-25T01:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff11d7fa640 1 -- 172.21.15.27:0/798370636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128019600 con 0x55fcca859380 2024-04-25T01:26:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 -- 172.21.15.27:0/798370636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca859380 msgr2=0x55fcca859760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca859380 0x55fcca859760 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff12800a5c0 tx=0x7ff128012000 comp rx=0 tx=0).stop 2024-04-25T01:26:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 -- 172.21.15.27:0/798370636 shutdown_connections 2024-04-25T01:26:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca86c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 --2- 172.21.15.27:0/798370636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca859380 0x55fcca859760 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 -- 172.21.15.27:0/798370636 >> 172.21.15.27:0/798370636 conn(0x55fcca855950 msgr2=0x55fcca855d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 -- 172.21.15.27:0/798370636 shutdown_connections 2024-04-25T01:26:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.311+0000 7ff140268740 1 -- 172.21.15.27:0/798370636 wait complete. 2024-04-25T01:26:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 1 -- start start 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 1 Processor -- start 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca869f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca8790d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fcca86fbf0 con 0x55fcca86a9f0 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fcca779f50 con 0x55fcca859d30 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fcca86fd70 con 0x55fcca86ce30 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff140268740 10 client.0 _ll_register_callbacks cb 0x55fcca7b5000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff1377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca8790d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff1377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca8790d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41394/0 (socket says 172.21.15.27:41394) 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff1377fe640 1 -- 172.21.15.27:0/2677359026 learned_addr learned my addr 172.21.15.27:0/2677359026 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.312+0000 7ff136ffd640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca869f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff1367fc640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff1377fe640 1 -- 172.21.15.27:0/2677359026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 msgr2=0x55fcca86a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff1377fe640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff1377fe640 1 -- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 msgr2=0x55fcca869f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff1377fe640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca869f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff1377fe640 1 -- 172.21.15.27:0/2677359026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcca4d23f0 con 0x55fcca86a9f0 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff136ffd640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca869f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff1367fc640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff1377fe640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca8790d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff13009b580 tx=0x7ff13009c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84ad36a2fd3b894 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff11aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:26:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff11aff5640 1 -- 172.21.15.27:0/2677359026 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13001f050 con 0x55fcca86a9f0 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff11aff5640 1 -- 172.21.15.27:0/2677359026 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff13009cbb0 con 0x55fcca86a9f0 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff140268740 1 -- 172.21.15.27:0/2677359026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff12801bb40 con 0x55fcca86a9f0 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff11aff5640 1 -- 172.21.15.27:0/2677359026 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff1300a44f0 con 0x55fcca86a9f0 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff140268740 10 client.15969 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff140268740 1 -- 172.21.15.27:0/2677359026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fcca4920f0 con 0x55fcca86a9f0 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.313+0000 7ff11aff5640 1 -- 172.21.15.27:0/2677359026 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12801bb40 con 0x55fcca86a9f0 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff11aff5640 1 -- 172.21.15.27:0/2677359026 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff13000f040 con 0x55fcca86a9f0 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff140268740 10 client.15969 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff140268740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff140268740 20 client.15969 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff140268740 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fcca87d7f0 0x55fcca87fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff140268740 4 client.15969 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff140268740 1 -- 172.21.15.27:0/2677359026 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcca7cbbf0 con 0x55fcca87d7f0 2024-04-25T01:26:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff136ffd640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fcca87d7f0 0x55fcca87fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff136ffd640 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fcca87d7f0 0x55fcca87fc10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4c0808d4173ef2 server_cookie=f025f90ad96e66a6 in_seq=0 out_seq=0 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.314+0000 7ff11aff5640 10 client.15969 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff11aff5640 1 -- 172.21.15.27:0/2677359026 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fcca7cbbf0 con 0x55fcca87d7f0 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff11aff5640 10 client.15969 handle_command_reply: tid=0 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff140268740 1 client.15969 shutdown 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff140268740 1 -- 172.21.15.27:0/2677359026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 msgr2=0x55fcca8790d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff140268740 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca8790d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff13009b580 tx=0x7ff13009c040 comp rx=0 tx=0).stop 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff140268740 1 -- 172.21.15.27:0/2677359026 shutdown_connections 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff140268740 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fcca87d7f0 0x55fcca87fc10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff140268740 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca8790d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff11aff5640 0 client.15969 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.315+0000 7ff140268740 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca86a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.316+0000 7ff140268740 1 --2- 172.21.15.27:0/2677359026 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca869f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.316+0000 7ff140268740 1 -- 172.21.15.27:0/2677359026 >> 172.21.15.27:0/2677359026 conn(0x55fcca855950 msgr2=0x55fcca85b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.316+0000 7ff140268740 1 -- 172.21.15.27:0/2677359026 shutdown_connections 2024-04-25T01:26:38.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.316+0000 7ff140268740 1 -- 172.21.15.27:0/2677359026 wait complete. 2024-04-25T01:26:38.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.316+0000 7ff140268740 20 client.15969 trim_cache size 0 max 16384 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.333+0000 7ff140268740 1 Processor -- start 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.333+0000 7ff140268740 1 -- start start 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff140268740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca859d30 0x55fcca855d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff140268740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca86a9f0 0x55fcca854970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff140268740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86ce30 0x55fcca8854e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff140268740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1240127b0 con 0x55fcca86ce30 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff140268740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff124010080 con 0x55fcca86a9f0 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff140268740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff12400ded0 con 0x55fcca859d30 2024-04-25T01:26:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff1377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86ce30 0x55fcca8854e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff1377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86ce30 0x55fcca8854e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41406/0 (socket says 172.21.15.27:41406) 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff1377fe640 1 -- 172.21.15.27:0/3534656566 learned_addr learned my addr 172.21.15.27:0/3534656566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff136ffd640 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca859d30 0x55fcca855d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff1367fc640 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca86a9f0 0x55fcca854970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff136ffd640 1 -- 172.21.15.27:0/3534656566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca86a9f0 msgr2=0x55fcca854970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff136ffd640 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca86a9f0 0x55fcca854970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff136ffd640 1 -- 172.21.15.27:0/3534656566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86ce30 msgr2=0x55fcca8854e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff136ffd640 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86ce30 0x55fcca8854e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff136ffd640 1 -- 172.21.15.27:0/3534656566 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcca86f590 con 0x55fcca859d30 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.334+0000 7ff1377fe640 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86ce30 0x55fcca8854e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.335+0000 7ff136ffd640 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca859d30 0x55fcca855d30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff12801f000 tx=0x7ff12801cda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68e0da3ee0af0709 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.335+0000 7ff11cff9640 1 -- 172.21.15.27:0/3534656566 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128016040 con 0x55fcca859d30 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.335+0000 7ff11cff9640 1 -- 172.21.15.27:0/3534656566 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12801bd40 con 0x55fcca859d30 2024-04-25T01:26:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.335+0000 7ff11cff9640 1 -- 172.21.15.27:0/3534656566 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128027480 con 0x55fcca859d30 2024-04-25T01:26:38.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 -- 172.21.15.27:0/3534656566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca859d30 msgr2=0x55fcca855d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca859d30 0x55fcca855d30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff12801f000 tx=0x7ff12801cda0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 -- 172.21.15.27:0/3534656566 shutdown_connections 2024-04-25T01:26:38.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86ce30 0x55fcca8854e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca86a9f0 0x55fcca854970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 --2- 172.21.15.27:0/3534656566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca859d30 0x55fcca855d30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 -- 172.21.15.27:0/3534656566 >> 172.21.15.27:0/3534656566 conn(0x55fcca855950 msgr2=0x55fcca854260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 -- 172.21.15.27:0/3534656566 shutdown_connections 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 -- 172.21.15.27:0/3534656566 wait complete. 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 -- start start 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.336+0000 7ff140268740 1 Processor -- start 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff140268740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca875220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff140268740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca866800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff140268740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca866d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff140268740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff12400e540 con 0x55fcca86a9f0 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff140268740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff12400f9b0 con 0x55fcca859d30 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff140268740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff12400ec10 con 0x55fcca86ce30 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff140268740 10 client.0 _ll_register_callbacks cb 0x7ff130004ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff1367fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca866800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff1367fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca866800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41420/0 (socket says 172.21.15.27:41420) 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff1367fc640 1 -- 172.21.15.27:0/4031033458 learned_addr learned my addr 172.21.15.27:0/4031033458 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff1377fe640 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca866d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff136ffd640 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca875220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff136ffd640 1 -- 172.21.15.27:0/4031033458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 msgr2=0x55fcca866d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff136ffd640 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca866d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff136ffd640 1 -- 172.21.15.27:0/4031033458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 msgr2=0x55fcca866800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff136ffd640 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca866800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff136ffd640 1 -- 172.21.15.27:0/4031033458 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcca867380 con 0x55fcca859d30 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.337+0000 7ff1367fc640 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca866800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff136ffd640 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca875220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff128010880 tx=0x7ff12800fd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=880617b2efe5c9d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff11aff5640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff11aff5640 1 -- 172.21.15.27:0/4031033458 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128016040 con 0x55fcca859d30 2024-04-25T01:26:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff11aff5640 1 -- 172.21.15.27:0/4031033458 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff12801c7e0 con 0x55fcca859d30 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff140268740 1 -- 172.21.15.27:0/4031033458 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff12801bd40 con 0x55fcca859d30 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff11aff5640 1 -- 172.21.15.27:0/4031033458 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff12804f050 con 0x55fcca859d30 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff140268740 10 client.25508 fetch_fsmap learned FSMap version 25 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff140268740 1 -- 172.21.15.27:0/4031033458 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fcca8e3b90 con 0x55fcca859d30 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.338+0000 7ff11aff5640 1 -- 172.21.15.27:0/4031033458 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12801bd40 con 0x55fcca859d30 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff11aff5640 1 -- 172.21.15.27:0/4031033458 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff128027580 con 0x55fcca859d30 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff140268740 10 client.25508 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff140268740 10 client.25508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff140268740 20 client.25508 populate_metadata read hostname 'smithi027' 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff140268740 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fcca87d7f0 0x55fcca8e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff140268740 4 client.25508 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff140268740 1 -- 172.21.15.27:0/4031033458 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fcca7c5f40 con 0x55fcca87d7f0 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff1367fc640 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fcca87d7f0 0x55fcca8e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff11aff5640 10 client.25508 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.339+0000 7ff1367fc640 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fcca87d7f0 0x55fcca8e5fc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53bf48af977e2dbb server_cookie=3c7ca5e544edeb3 in_seq=0 out_seq=0 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff11aff5640 1 -- 172.21.15.27:0/4031033458 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fcca7c5f40 con 0x55fcca87d7f0 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff11aff5640 10 client.25508 handle_command_reply: tid=0 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 client.25508 shutdown 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 -- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 msgr2=0x55fcca875220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca875220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff128010880 tx=0x7ff12800fd70 comp rx=0 tx=0).stop 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 -- 172.21.15.27:0/4031033458 shutdown_connections 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fcca87d7f0 0x55fcca8e5fc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcca86ce30 0x55fcca866d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff11aff5640 0 client.25508 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcca86a9f0 0x55fcca866800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 --2- 172.21.15.27:0/4031033458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcca859d30 0x55fcca875220 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 -- 172.21.15.27:0/4031033458 >> 172.21.15.27:0/4031033458 conn(0x55fcca855950 msgr2=0x55fcca854af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 -- 172.21.15.27:0/4031033458 shutdown_connections 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 1 -- 172.21.15.27:0/4031033458 wait complete. 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.340+0000 7ff140268740 20 client.25508 trim_cache size 0 max 16384 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stdout: "89002673-7ec9-4026-89ac-456f9f56750e": { 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "89002673-7ec9-4026-89ac-456f9f56750e", 2024-04-25T01:26:38.326 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:26:38.327 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:26:38.327 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:26:38.327 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 -- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1200a2f30 msgr2=0x7ff1200a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1200a2f30 0x7ff1200a53f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff12800afb0 tx=0x7ff12800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 -- 172.21.15.27:0/131323158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff13810e0f0 msgr2=0x7ff13819ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff13810e0f0 0x7ff13819ea60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff130005fd0 tx=0x7ff13000ed10 comp rx=0 tx=0).stop 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 -- 172.21.15.27:0/131323158 shutdown_connections 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff13810e0f0 0x7ff13819ea60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1200a2f30 0x7ff1200a53f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff138100e60 0x7ff13819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 --2- 172.21.15.27:0/131323158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff138100540 0x7ff13819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:38.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 -- 172.21.15.27:0/131323158 >> 172.21.15.27:0/131323158 conn(0x7ff138077740 msgr2=0x7ff13810f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:38.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 -- 172.21.15.27:0/131323158 shutdown_connections 2024-04-25T01:26:38.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:26:38.341+0000 7ff13d55a640 1 -- 172.21.15.27:0/131323158 wait complete. 2024-04-25T01:26:38.333 INFO:tasks.cephfs.filesystem:scrub status for tag:89002673-7ec9-4026-89ac-456f9f56750e - {'path': '/', 'tag': '89002673-7ec9-4026-89ac-456f9f56750e', 'options': 'recursive,force'} 2024-04-25T01:26:38.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:38 smithi027 ceph-mon[29197]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:38.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:38 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:26:38.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:38 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:38.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:38 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:26:38.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:38 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:38.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:38 smithi153 ceph-mon[32621]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:38.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:38 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:26:38.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:38 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:38.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:38 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:26:38.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:38 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:38.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:38 smithi139 ceph-mon[32760]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:38.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:38 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:26:38.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:38 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:38.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:38 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:26:38.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:38 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:26:40.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:40 smithi153 ceph-mon[32621]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:40.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:40 smithi139 ceph-mon[32760]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:40.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:40 smithi027 ceph-mon[29197]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:26:42.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:42 smithi139 ceph-mon[32760]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T01:26:42.794 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:42 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:26:42.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:42 smithi027 ceph-mon[29197]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T01:26:42.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:42 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:26:43.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:42 smithi153 ceph-mon[32621]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T01:26:43.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:42 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:26:44.393 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 95% (11909/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-25T01:26:44.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:44 smithi139 ceph-mon[32760]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:44.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:44 smithi027 ceph-mon[29197]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:44.959 INFO:tasks.workunit.client.0.smithi027.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-25T01:26:44.959 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T01:26:44.959 INFO:tasks.workunit.client.0.smithi027.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr: git switch -c 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr:Or undo this operation with: 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr: git switch - 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T01:26:44.960 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-04-25T01:26:44.961 INFO:tasks.workunit.client.0.smithi027.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-25T01:26:44.964 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T01:26:45.006 INFO:tasks.workunit.client.0.smithi027.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T01:26:45.009 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T01:26:45.009 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T01:26:45.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:44 smithi153 ceph-mon[32621]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:45.580 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T01:26:45.637 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T01:26:45.678 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T01:26:45.680 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T01:26:45.680 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T01:26:45.720 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T01:26:45.723 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:26:45.723 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T01:26:45.779 INFO:tasks.workunit:Running workunits matching fs/test_o_trunc.sh on client.0... 2024-04-25T01:26:45.781 INFO:tasks.workunit:Running workunit fs/test_o_trunc.sh... 2024-04-25T01:26:45.781 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/test_o_trunc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-04-25T01:26:45.841 INFO:tasks.workunit.client.0.smithi027.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-04-25T01:26:45.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs 2024-04-25T01:26:45.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc trunc.foo 600 2024-04-25T01:26:46.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:46 smithi139 ceph-mon[32760]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:46.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:46 smithi027 ceph-mon[29197]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:47.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:46 smithi153 ceph-mon[32621]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:48.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:48 smithi139 ceph-mon[32760]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T01:26:48.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:48 smithi139 ceph-mon[32760]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:26:48.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:48 smithi027 ceph-mon[29197]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T01:26:48.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:48 smithi027 ceph-mon[29197]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:26:49.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:48 smithi153 ceph-mon[32621]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T01:26:49.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:48 smithi153 ceph-mon[32621]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:26:50.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:50 smithi027 ceph-mon[29197]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:51.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:50 smithi153 ceph-mon[32621]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:51.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:50 smithi139 ceph-mon[32760]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T01:26:52.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:52 smithi027 ceph-mon[29197]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.3 KiB/s wr, 10 op/s 2024-04-25T01:26:53.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:52 smithi153 ceph-mon[32621]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.3 KiB/s wr, 10 op/s 2024-04-25T01:26:53.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:52 smithi139 ceph-mon[32760]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.3 KiB/s wr, 10 op/s 2024-04-25T01:26:54.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:54 smithi027 ceph-mon[29197]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-25T01:26:55.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:54 smithi153 ceph-mon[32621]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-25T01:26:55.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:54 smithi139 ceph-mon[32760]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-25T01:26:56.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:56 smithi027 ceph-mon[29197]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-25T01:26:57.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:56 smithi153 ceph-mon[32621]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-25T01:26:57.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:56 smithi139 ceph-mon[32760]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2024-04-25T01:26:58.827 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:58 smithi153 ceph-mon[32621]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-04-25T01:26:58.827 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:26:58 smithi153 ceph-mon[32621]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-25T01:26:58.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:58 smithi027 ceph-mon[29197]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-04-25T01:26:58.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:26:58 smithi027 ceph-mon[29197]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-25T01:26:59.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:58 smithi139 ceph-mon[32760]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-04-25T01:26:59.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:26:58 smithi139 ceph-mon[32760]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-25T01:27:00.843 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:00 smithi139 ceph-mon[32760]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:27:00.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:00 smithi027 ceph-mon[29197]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:27:01.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:00 smithi153 ceph-mon[32621]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:27:02.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:02 smithi027 ceph-mon[29197]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-04-25T01:27:03.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:02 smithi153 ceph-mon[32621]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-04-25T01:27:03.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:02 smithi139 ceph-mon[32760]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-04-25T01:27:04.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:04 smithi027 ceph-mon[29197]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T01:27:05.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:04 smithi153 ceph-mon[32621]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T01:27:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:04 smithi139 ceph-mon[32760]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T01:27:06.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:06 smithi027 ceph-mon[29197]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T01:27:07.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:06 smithi153 ceph-mon[32621]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T01:27:07.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:06 smithi139 ceph-mon[32760]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T01:27:08.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:27:08.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:27:08.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.472+0000 7fe23f4f5640 1 -- 172.21.15.27:0/2165458911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100620 msgr2=0x7fe238100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.472+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/2165458911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100620 0x7fe238100aa0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe228009930 tx=0x7fe22802a420 comp rx=0 tx=0).stop 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.472+0000 7fe23f4f5640 1 -- 172.21.15.27:0/2165458911 shutdown_connections 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.472+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/2165458911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe238100fe0 0x7fe23810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.472+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/2165458911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100620 0x7fe238100aa0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.472+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/2165458911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe238106630 0x7fe238106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.472+0000 7fe23f4f5640 1 -- 172.21.15.27:0/2165458911 >> 172.21.15.27:0/2165458911 conn(0x7fe2380776a0 msgr2=0x7fe238077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 -- 172.21.15.27:0/2165458911 shutdown_connections 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 -- 172.21.15.27:0/2165458911 wait complete. 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 Processor -- start 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 -- start start 2024-04-25T01:27:08.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe238100620 0x7fe238197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100fe0 0x7fe238198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe238106630 0x7fe23819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe238110a40 con 0x7fe238100fe0 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe2381108c0 con 0x7fe238106630 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.473+0000 7fe23f4f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe238110bc0 con 0x7fe238100620 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100fe0 0x7fe238198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100fe0 0x7fe238198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36176/0 (socket says 172.21.15.27:36176) 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 -- 172.21.15.27:0/3366795383 learned_addr learned my addr 172.21.15.27:0/3366795383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23d291640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe238100620 0x7fe238197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23da92640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe238106630 0x7fe23819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 -- 172.21.15.27:0/3366795383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe238100620 msgr2=0x7fe238197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe238100620 0x7fe238197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 -- 172.21.15.27:0/3366795383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe238106630 msgr2=0x7fe23819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe238106630 0x7fe23819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 -- 172.21.15.27:0/3366795383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe23819cfc0 con 0x7fe238100fe0 2024-04-25T01:27:08.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23ca90640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100fe0 0x7fe238198510 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe228009230 tx=0x7fe228004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe22e7fc640 1 -- 172.21.15.27:0/3366795383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22803f030 con 0x7fe238100fe0 2024-04-25T01:27:08.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe22e7fc640 1 -- 172.21.15.27:0/3366795383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe228003ad0 con 0x7fe238100fe0 2024-04-25T01:27:08.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe22e7fc640 1 -- 172.21.15.27:0/3366795383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228035070 con 0x7fe238100fe0 2024-04-25T01:27:08.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23f4f5640 1 -- 172.21.15.27:0/3366795383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe23819d250 con 0x7fe238100fe0 2024-04-25T01:27:08.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.474+0000 7fe23f4f5640 1 -- 172.21.15.27:0/3366795383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2381a4b40 con 0x7fe238100fe0 2024-04-25T01:27:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.477+0000 7fe22e7fc640 1 -- 172.21.15.27:0/3366795383 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe228042030 con 0x7fe238100fe0 2024-04-25T01:27:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.477+0000 7fe242204740 1 Processor -- start 2024-04-25T01:27:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.477+0000 7fe242204740 1 -- start start 2024-04-25T01:27:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.477+0000 7fe242204740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344f630 0x56191344fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.477+0000 7fe242204740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x56191345a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.477+0000 7fe242204740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe242204740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5619133b46b0 con 0x56191344f630 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe242204740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56191335c880 con 0x56191345a620 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe242204740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5619133b0670 con 0x56191344ff50 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe22e7fc640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe2240a2d00 0x7fe2240a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344f630 0x56191344fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344f630 0x56191344fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36178/0 (socket says 172.21.15.27:36178) 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 -- 172.21.15.27:0/1744293352 learned_addr learned my addr 172.21.15.27:0/1744293352 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe2240a2d00 0x7fe2240a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe22e7fc640 1 -- 172.21.15.27:0/3366795383 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe2280efe20 con 0x7fe238100fe0 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23d291640 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x56191345a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23ca90640 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 -- 172.21.15.27:0/1744293352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 msgr2=0x56191345a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x56191345a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 -- 172.21.15.27:0/1744293352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 msgr2=0x56191345ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 -- 172.21.15.27:0/1744293352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619131927d0 con 0x56191344f630 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23ca90640 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344f630 0x56191344fa10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe230012da0 tx=0x7fe2300107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95dc83bf295505 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.478+0000 7fe23da92640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe2240a2d00 0x7fe2240a51c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe230005c40 tx=0x7fe23001d000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe2237fe640 1 -- 172.21.15.27:0/1744293352 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23002a070 con 0x56191344f630 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe2237fe640 1 -- 172.21.15.27:0/1744293352 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe230003e50 con 0x56191344f630 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe2237fe640 1 -- 172.21.15.27:0/1744293352 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe230004270 con 0x56191344f630 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 -- 172.21.15.27:0/1744293352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344f630 msgr2=0x56191344fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344f630 0x56191344fa10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe230012da0 tx=0x7fe2300107a0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 -- 172.21.15.27:0/1744293352 shutdown_connections 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x56191345a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 --2- 172.21.15.27:0/1744293352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344f630 0x56191344fa10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 -- 172.21.15.27:0/1744293352 >> 172.21.15.27:0/1744293352 conn(0x56191344b950 msgr2=0x56191344bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 -- 172.21.15.27:0/1744293352 shutdown_connections 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.479+0000 7fe242204740 1 -- 172.21.15.27:0/1744293352 wait complete. 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 1 -- start start 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 1 Processor -- start 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 0x561913457580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344ff50 0x561913457ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913458000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56191345d3b0 con 0x56191345a620 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56191336ff50 con 0x56191344ff50 2024-04-25T01:27:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56191345d530 con 0x56191344f630 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe242204740 10 client.0 _ll_register_callbacks cb 0x5619133ab000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe23da92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913458000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe23da92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913458000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36184/0 (socket says 172.21.15.27:36184) 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.480+0000 7fe23da92640 1 -- 172.21.15.27:0/3140883592 learned_addr learned my addr 172.21.15.27:0/3140883592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23d291640 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 0x561913457580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23ca90640 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344ff50 0x561913457ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23da92640 1 -- 172.21.15.27:0/3140883592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 msgr2=0x561913457580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23da92640 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 0x561913457580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23da92640 1 -- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344ff50 msgr2=0x561913457ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23da92640 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344ff50 0x561913457ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23da92640 1 -- 172.21.15.27:0/3140883592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619130c83f0 con 0x56191345a620 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23ca90640 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344ff50 0x561913457ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe23da92640 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913458000 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe23000e010 tx=0x7fe23000dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46087aea9564d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe220ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe220ff9640 1 -- 172.21.15.27:0/3140883592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23002a070 con 0x56191345a620 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe220ff9640 1 -- 172.21.15.27:0/3140883592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe230009dc0 con 0x56191345a620 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe242204740 1 -- 172.21.15.27:0/3140883592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe230003e50 con 0x56191345a620 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe220ff9640 1 -- 172.21.15.27:0/3140883592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe2300052d0 con 0x56191345a620 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe242204740 10 client.15987 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe242204740 1 -- 172.21.15.27:0/3140883592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5619130880f0 con 0x56191345a620 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.481+0000 7fe220ff9640 1 -- 172.21.15.27:0/3140883592 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe230003e50 con 0x56191345a620 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe220ff9640 1 -- 172.21.15.27:0/3140883592 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fe230024030 con 0x56191345a620 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe242204740 10 client.15987 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe242204740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe242204740 20 client.15987 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe242204740 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56191346a7e0 0x56191346cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe242204740 4 client.15987 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe242204740 1 -- 172.21.15.27:0/3140883592 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619133c1bf0 con 0x56191346a7e0 2024-04-25T01:27:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe23d291640 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56191346a7e0 0x56191346cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe220ff9640 10 client.15987 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.482+0000 7fe23d291640 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56191346a7e0 0x56191346cbd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=384290ba5350515 server_cookie=9660da19079c3c05 in_seq=0 out_seq=0 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe220ff9640 1 -- 172.21.15.27:0/3140883592 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5619133c1bf0 con 0x56191346a7e0 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe220ff9640 10 client.15987 handle_command_reply: tid=0 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 client.15987 shutdown 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 -- 172.21.15.27:0/3140883592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 msgr2=0x561913458000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913458000 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe23000e010 tx=0x7fe23000dd20 comp rx=0 tx=0).stop 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 -- 172.21.15.27:0/3140883592 shutdown_connections 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56191346a7e0 0x56191346cbd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913458000 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344ff50 0x561913457ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 --2- 172.21.15.27:0/3140883592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 0x561913457580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe242204740 1 -- 172.21.15.27:0/3140883592 >> 172.21.15.27:0/3140883592 conn(0x56191344b950 msgr2=0x561913451750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.483+0000 7fe220ff9640 0 client.15987 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.484+0000 7fe242204740 1 -- 172.21.15.27:0/3140883592 shutdown_connections 2024-04-25T01:27:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.484+0000 7fe242204740 1 -- 172.21.15.27:0/3140883592 wait complete. 2024-04-25T01:27:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.484+0000 7fe242204740 20 client.15987 trim_cache size 0 max 16384 2024-04-25T01:27:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.501+0000 7fe242204740 1 Processor -- start 2024-04-25T01:27:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.501+0000 7fe242204740 1 -- start start 2024-04-25T01:27:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe242204740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344f630 0x561913466270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe242204740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x5619134566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe242204740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913457e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe242204740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe2381b1470 con 0x56191345a620 2024-04-25T01:27:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe242204740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe2381aed40 con 0x56191344f630 2024-04-25T01:27:08.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe242204740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe2381acb90 con 0x56191344ff50 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913457e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913457e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36192/0 (socket says 172.21.15.27:36192) 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 -- 172.21.15.27:0/4244762046 learned_addr learned my addr 172.21.15.27:0/4244762046 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23ca90640 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x5619134566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 -- 172.21.15.27:0/4244762046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 msgr2=0x5619134566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x5619134566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 -- 172.21.15.27:0/4244762046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344f630 msgr2=0x561913466270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344f630 0x561913466270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 -- 172.21.15.27:0/4244762046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56191344b780 con 0x56191345a620 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23ca90640 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x5619134566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe23da92640 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913457e20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe2300107e0 tx=0x7fe230012c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ff480809a99fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe222ffd640 1 -- 172.21.15.27:0/4244762046 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23002a040 con 0x56191345a620 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe222ffd640 1 -- 172.21.15.27:0/4244762046 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe230009480 con 0x56191345a620 2024-04-25T01:27:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.502+0000 7fe222ffd640 1 -- 172.21.15.27:0/4244762046 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23001ea30 con 0x56191345a620 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 -- 172.21.15.27:0/4244762046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 msgr2=0x561913457e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913457e20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe2300107e0 tx=0x7fe230012c20 comp rx=0 tx=0).stop 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 -- 172.21.15.27:0/4244762046 shutdown_connections 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191345a620 0x561913457e20 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344ff50 0x5619134566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 --2- 172.21.15.27:0/4244762046 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191344f630 0x561913466270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 -- 172.21.15.27:0/4244762046 >> 172.21.15.27:0/4244762046 conn(0x56191344b950 msgr2=0x56191344a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 -- 172.21.15.27:0/4244762046 shutdown_connections 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 -- 172.21.15.27:0/4244762046 wait complete. 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 -- start start 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.503+0000 7fe242204740 1 Processor -- start 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe242204740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 0x56191345d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe242204740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344ff50 0x56191345dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe242204740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe242204740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe2381ad200 con 0x56191344ff50 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe242204740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe2381ae670 con 0x56191345a620 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe242204740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe2381ad8d0 con 0x56191344f630 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe242204740 10 client.0 _ll_register_callbacks cb 0x7fe230028070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344ff50 0x56191345dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344ff50 0x56191345dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36204/0 (socket says 172.21.15.27:36204) 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 -- 172.21.15.27:0/734041241 learned_addr learned my addr 172.21.15.27:0/734041241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23d291640 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 0x56191345d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23da92640 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 -- 172.21.15.27:0/734041241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 msgr2=0x56191345d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 0x56191345d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 -- 172.21.15.27:0/734041241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 msgr2=0x56191345e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 -- 172.21.15.27:0/734041241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619134ceee0 con 0x56191344ff50 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe23ca90640 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344ff50 0x56191345dc80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe228009900 tx=0x7fe2280bb9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1bcca7b9c640c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.504+0000 7fe220ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe220ff9640 1 -- 172.21.15.27:0/734041241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228049040 con 0x56191344ff50 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe220ff9640 1 -- 172.21.15.27:0/734041241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe228003c80 con 0x56191344ff50 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 1 -- 172.21.15.27:0/734041241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe230009480 con 0x56191344ff50 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe220ff9640 1 -- 172.21.15.27:0/734041241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe2280c0360 con 0x56191344ff50 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 10 client.15993 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 1 -- 172.21.15.27:0/734041241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5619134d0160 con 0x56191344ff50 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe220ff9640 1 -- 172.21.15.27:0/734041241 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe230009480 con 0x56191344ff50 2024-04-25T01:27:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe220ff9640 1 -- 172.21.15.27:0/734041241 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fe2280d9070 con 0x56191344ff50 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 10 client.15993 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 10 client.15993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 20 client.15993 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5619134d08e0 0x5619134d2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 4 client.15993 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.505+0000 7fe242204740 1 -- 172.21.15.27:0/734041241 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5619133bbf40 con 0x5619134d08e0 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.506+0000 7fe23d291640 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5619134d08e0 0x5619134d2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.506+0000 7fe23d291640 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5619134d08e0 0x5619134d2d00 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60d705a4060c38fc server_cookie=d6a3f540f1e89472 in_seq=0 out_seq=0 2024-04-25T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.506+0000 7fe220ff9640 10 client.15993 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe220ff9640 1 -- 172.21.15.27:0/734041241 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5619133bbf40 con 0x5619134d08e0 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe220ff9640 10 client.15993 handle_command_reply: tid=0 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 client.15993 shutdown 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 -- 172.21.15.27:0/734041241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344ff50 msgr2=0x56191345dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344ff50 0x56191345dc80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe228009900 tx=0x7fe2280bb9d0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 -- 172.21.15.27:0/734041241 shutdown_connections 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5619134d08e0 0x5619134d2d00 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56191345a620 0x56191345e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56191344ff50 0x56191345dc80 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe220ff9640 0 client.15993 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 --2- 172.21.15.27:0/734041241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56191344f630 0x56191345d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 -- 172.21.15.27:0/734041241 >> 172.21.15.27:0/734041241 conn(0x56191344b950 msgr2=0x561913450dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 -- 172.21.15.27:0/734041241 shutdown_connections 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 1 -- 172.21.15.27:0/734041241 wait complete. 2024-04-25T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.507+0000 7fe242204740 20 client.15993 trim_cache size 0 max 16384 2024-04-25T01:27:08.493 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:27:08.493 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:27:08.493 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:27:08.493 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 -- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe2240a2d00 msgr2=0x7fe2240a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe2240a2d00 0x7fe2240a51c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe230005c40 tx=0x7fe23001d000 comp rx=0 tx=0).stop 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 -- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100fe0 msgr2=0x7fe238198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100fe0 0x7fe238198510 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe228009230 tx=0x7fe228004270 comp rx=0 tx=0).stop 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 -- 172.21.15.27:0/3366795383 shutdown_connections 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe238106630 0x7fe23819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe2240a2d00 0x7fe2240a51c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe238100fe0 0x7fe238198510 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 --2- 172.21.15.27:0/3366795383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe238100620 0x7fe238197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 -- 172.21.15.27:0/3366795383 >> 172.21.15.27:0/3366795383 conn(0x7fe2380776a0 msgr2=0x7fe2380ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 -- 172.21.15.27:0/3366795383 shutdown_connections 2024-04-25T01:27:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.508+0000 7fe23f4f5640 1 -- 172.21.15.27:0/3366795383 wait complete. 2024-04-25T01:27:08.497 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:27:08.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:27:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 -- 172.21.15.27:0/1343044633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 msgr2=0x7f84f0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 --2- 172.21.15.27:0/1343044633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 0x7f84f0101c20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009900 tx=0x7f84e002a480 comp rx=0 tx=0).stop 2024-04-25T01:27:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 -- 172.21.15.27:0/1343044633 shutdown_connections 2024-04-25T01:27:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 --2- 172.21.15.27:0/1343044633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84f0102160 0x7f84f010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 --2- 172.21.15.27:0/1343044633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 0x7f84f0101c20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 --2- 172.21.15.27:0/1343044633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84f0107750 0x7f84f0107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 -- 172.21.15.27:0/1343044633 >> 172.21.15.27:0/1343044633 conn(0x7f84f00788d0 msgr2=0x7f84f0078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 -- 172.21.15.27:0/1343044633 shutdown_connections 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.622+0000 7f84f4c76640 1 -- 172.21.15.27:0/1343044633 wait complete. 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84f4c76640 1 Processor -- start 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84f4c76640 1 -- start start 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84f4c76640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 0x7f84f0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84f4c76640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84f0102160 0x7f84f0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84f4c76640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84f0107750 0x7f84f019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84f4c76640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84f0111b20 con 0x7f84f01017c0 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84f4c76640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01119a0 con 0x7f84f0102160 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84f4c76640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84f0111ca0 con 0x7f84f0107750 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 0x7f84f0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 0x7f84f0198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36220/0 (socket says 172.21.15.27:36220) 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 -- 172.21.15.27:0/388296181 learned_addr learned my addr 172.21.15.27:0/388296181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84edd9b640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84f0102160 0x7f84f0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 -- 172.21.15.27:0/388296181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84f0107750 msgr2=0x7f84f019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84f0107750 0x7f84f019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 -- 172.21.15.27:0/388296181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84f0102160 msgr2=0x7f84f0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84f0102160 0x7f84f0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 -- 172.21.15.27:0/388296181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f019cfe0 con 0x7f84f01017c0 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84edd9b640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84f0102160 0x7f84f0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.623+0000 7f84ee59c640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 0x7f84f0198040 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f84d800d950 tx=0x7f84d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.624+0000 7f84e77fe640 1 -- 172.21.15.27:0/388296181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d8017030 con 0x7f84f01017c0 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.624+0000 7f84e77fe640 1 -- 172.21.15.27:0/388296181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d8004020 con 0x7f84f01017c0 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.624+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f019d2d0 con 0x7f84f01017c0 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.624+0000 7f84e77fe640 1 -- 172.21.15.27:0/388296181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d80047a0 con 0x7f84f01017c0 2024-04-25T01:27:08.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.624+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f01a4b60 con 0x7f84f01017c0 2024-04-25T01:27:08.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.625+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f0102e70 con 0x7f84f01017c0 2024-04-25T01:27:08.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.626+0000 7f84e77fe640 1 -- 172.21.15.27:0/388296181 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f84d80041d0 con 0x7f84f01017c0 2024-04-25T01:27:08.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.627+0000 7f84e77fe640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f84dc0a2d80 0x7f84dc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.627+0000 7f84e77fe640 1 -- 172.21.15.27:0/388296181 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f84d80cf7e0 con 0x7f84f01017c0 2024-04-25T01:27:08.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.627+0000 7f84edd9b640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f84dc0a2d80 0x7f84dc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.627+0000 7f84edd9b640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f84dc0a2d80 0x7f84dc0a5240 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f84e0005ab0 tx=0x7f84e0000bc0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.630+0000 7f84e77fe640 1 -- 172.21.15.27:0/388296181 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84d809c0f0 con 0x7f84f01017c0 2024-04-25T01:27:08.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:08 smithi027 ceph-mon[29197]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 33 KiB/s wr, 30 op/s 2024-04-25T01:27:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.854+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84f0199340 con 0x7f84f01017c0 2024-04-25T01:27:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.855+0000 7f84e77fe640 1 -- 172.21.15.27:0/388296181 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11883 (secure 0 0 0) 0x7f84f0199340 con 0x7f84f01017c0 2024-04-25T01:27:08.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:27:08.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:26:47.433054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24517,24521,24505]},"id":1}]} 2024-04-25T01:27:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f84dc0a2d80 msgr2=0x7f84dc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f84dc0a2d80 0x7f84dc0a5240 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f84e0005ab0 tx=0x7f84e0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 msgr2=0x7f84f0198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 0x7f84f0198040 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f84d800d950 tx=0x7f84d800de10 comp rx=0 tx=0).stop 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 shutdown_connections 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84f0107750 0x7f84f019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84f0102160 0x7f84f0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f84dc0a2d80 0x7f84dc0a5240 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 --2- 172.21.15.27:0/388296181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f01017c0 0x7f84f0198040 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 >> 172.21.15.27:0/388296181 conn(0x7f84f00788d0 msgr2=0x7f84f0100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 shutdown_connections 2024-04-25T01:27:08.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.857+0000 7f84f4c76640 1 -- 172.21.15.27:0/388296181 wait complete. 2024-04-25T01:27:08.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:27:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 -- 172.21.15.27:0/1225318843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100620 msgr2=0x7f5214100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 --2- 172.21.15.27:0/1225318843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100620 0x7f5214100aa0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009930 tx=0x7f51fc02a420 comp rx=0 tx=0).stop 2024-04-25T01:27:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 -- 172.21.15.27:0/1225318843 shutdown_connections 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 --2- 172.21.15.27:0/1225318843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214100fe0 0x7f521410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 --2- 172.21.15.27:0/1225318843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100620 0x7f5214100aa0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 --2- 172.21.15.27:0/1225318843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5214106630 0x7f5214106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 -- 172.21.15.27:0/1225318843 >> 172.21.15.27:0/1225318843 conn(0x7f52140776a0 msgr2=0x7f5214077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 -- 172.21.15.27:0/1225318843 shutdown_connections 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.972+0000 7f5218c38640 1 -- 172.21.15.27:0/1225318843 wait complete. 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f5218c38640 1 Processor -- start 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f5218c38640 1 -- start start 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f5218c38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214100620 0x7f5214198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f5218c38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100fe0 0x7f5214198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f5218c38640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5214106630 0x7f521419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f5218c38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5214110a00 con 0x7f5214100fe0 2024-04-25T01:27:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f5218c38640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5214110880 con 0x7f5214106630 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f5218c38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5214110b80 con 0x7f5214100620 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f520bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100fe0 0x7f5214198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f520bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100fe0 0x7f5214198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36240/0 (socket says 172.21.15.27:36240) 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f520bfff640 1 -- 172.21.15.27:0/813109742 learned_addr learned my addr 172.21.15.27:0/813109742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.973+0000 7f521259c640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214100620 0x7f5214198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f5212d9d640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5214106630 0x7f521419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520bfff640 1 -- 172.21.15.27:0/813109742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214100620 msgr2=0x7f5214198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520bfff640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214100620 0x7f5214198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520bfff640 1 -- 172.21.15.27:0/813109742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5214106630 msgr2=0x7f521419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520bfff640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5214106630 0x7f521419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520bfff640 1 -- 172.21.15.27:0/813109742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521419d020 con 0x7f5214100fe0 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f5212d9d640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5214106630 0x7f521419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520bfff640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100fe0 0x7f5214198570 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009230 tx=0x7f51fc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520b7fe640 1 -- 172.21.15.27:0/813109742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc03f030 con 0x7f5214100fe0 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f5218c38640 1 -- 172.21.15.27:0/813109742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f521419d2b0 con 0x7f5214100fe0 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520b7fe640 1 -- 172.21.15.27:0/813109742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51fc003ad0 con 0x7f5214100fe0 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f520b7fe640 1 -- 172.21.15.27:0/813109742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc035070 con 0x7f5214100fe0 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.974+0000 7f5218c38640 1 -- 172.21.15.27:0/813109742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52141a4ba0 con 0x7f5214100fe0 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.976+0000 7f520b7fe640 1 -- 172.21.15.27:0/813109742 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f51fc042030 con 0x7f5214100fe0 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.976+0000 7f521ba24740 1 Processor -- start 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521ba24740 1 -- start start 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521ba24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d482860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521ba24740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d491440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521ba24740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d493d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521ba24740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55614d3e7670 con 0x55614d482da0 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521ba24740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55614d38f970 con 0x55614d491980 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521ba24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55614d3e3630 con 0x55614d482480 2024-04-25T01:27:08.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d491440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d491440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36254/0 (socket says 172.21.15.27:36254) 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521259c640 1 -- 172.21.15.27:0/3557192054 learned_addr learned my addr 172.21.15.27:0/3557192054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f520bfff640 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d493d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f520b7fe640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52000a2d80 0x7f52000a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f5212d9d640 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d482860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521259c640 1 -- 172.21.15.27:0/3557192054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 msgr2=0x55614d482860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521259c640 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d482860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521259c640 1 -- 172.21.15.27:0/3557192054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 msgr2=0x55614d493d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521259c640 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d493d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f521259c640 1 -- 172.21.15.27:0/3557192054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55614d3b3400 con 0x55614d482da0 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f5212d9d640 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d482860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f520bfff640 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d493d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f5212d9d640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52000a2d80 0x7f52000a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.977+0000 7f520b7fe640 1 -- 172.21.15.27:0/813109742 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f51fc0efc30 con 0x7f5214100fe0 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.978+0000 7f521259c640 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d491440 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f520400aba0 tx=0x7f520400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df1cba22319c750 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.978+0000 7f5208ff9640 1 -- 172.21.15.27:0/3557192054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520401d070 con 0x55614d482da0 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.978+0000 7f5208ff9640 1 -- 172.21.15.27:0/3557192054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5204018b20 con 0x55614d482da0 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.978+0000 7f5208ff9640 1 -- 172.21.15.27:0/3557192054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204010b20 con 0x55614d482da0 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.978+0000 7f5212d9d640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52000a2d80 0x7f52000a5240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f520c009270 tx=0x7f520c004a40 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.978+0000 7f521ba24740 1 -- 172.21.15.27:0/3557192054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 msgr2=0x55614d491440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.978+0000 7f521ba24740 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d491440 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f520400aba0 tx=0x7f520400f790 comp rx=0 tx=0).stop 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 -- 172.21.15.27:0/3557192054 shutdown_connections 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d493d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d491440 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 --2- 172.21.15.27:0/3557192054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d482860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 -- 172.21.15.27:0/3557192054 >> 172.21.15.27:0/3557192054 conn(0x55614d47e8a0 msgr2=0x55614d47ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 -- 172.21.15.27:0/3557192054 shutdown_connections 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 -- 172.21.15.27:0/3557192054 wait complete. 2024-04-25T01:27:08.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 -- start start 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 Processor -- start 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482480 0x55614d491320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d48e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d48ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55614d494740 con 0x55614d482da0 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55614d3a3050 con 0x55614d482480 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55614d4948c0 con 0x55614d491980 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f521ba24740 10 client.0 _ll_register_callbacks cb 0x55614d0fabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f520bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d48e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f520bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d48e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36262/0 (socket says 172.21.15.27:36262) 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.979+0000 7f520bfff640 1 -- 172.21.15.27:0/935380375 learned_addr learned my addr 172.21.15.27:0/935380375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f5212d9d640 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d48ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f521259c640 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482480 0x55614d491320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f520bfff640 1 -- 172.21.15.27:0/935380375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 msgr2=0x55614d48ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f520bfff640 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d48ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f520bfff640 1 -- 172.21.15.27:0/935380375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482480 msgr2=0x55614d491320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f520bfff640 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482480 0x55614d491320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f520bfff640 1 -- 172.21.15.27:0/935380375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55614d374dc0 con 0x55614d482da0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f520bfff640 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d48e610 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0b8220 tx=0x7f51fc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa9ea79a81a9245 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f51f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f51f9ffb640 1 -- 172.21.15.27:0/935380375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc049070 con 0x55614d482da0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f51f9ffb640 1 -- 172.21.15.27:0/935380375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f51fc003c80 con 0x55614d482da0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f521ba24740 1 -- 172.21.15.27:0/935380375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5204018b20 con 0x55614d482da0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f51f9ffb640 1 -- 172.21.15.27:0/935380375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f51fc0c0170 con 0x55614d482da0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f521ba24740 10 client.16011 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f521ba24740 1 -- 172.21.15.27:0/935380375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55614d0fb3f0 con 0x55614d482da0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f51f9ffb640 1 -- 172.21.15.27:0/935380375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5204018b20 con 0x55614d482da0 2024-04-25T01:27:08.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.980+0000 7f51f9ffb640 1 -- 172.21.15.27:0/935380375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f51fc09abe0 con 0x55614d482da0 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f521ba24740 10 client.16011 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f521ba24740 10 client.16011 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f521ba24740 20 client.16011 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f521ba24740 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55614d4a19c0 0x55614d4a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f521ba24740 4 client.16011 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f521ba24740 1 -- 172.21.15.27:0/935380375 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55614d3f4bb0 con 0x55614d4a19c0 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f521259c640 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55614d4a19c0 0x55614d4a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f521259c640 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55614d4a19c0 0x55614d4a3e10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=477633a7249ab32c server_cookie=609e0124c84430c in_seq=0 out_seq=0 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.981+0000 7f51f9ffb640 10 client.16011 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f51f9ffb640 1 -- 172.21.15.27:0/935380375 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55614d3f4bb0 con 0x55614d4a19c0 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f51f9ffb640 10 client.16011 handle_command_reply: tid=0 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 client.16011 shutdown 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 -- 172.21.15.27:0/935380375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 msgr2=0x55614d48e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d48e610 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0b8220 tx=0x7f51fc009260 comp rx=0 tx=0).stop 2024-04-25T01:27:08.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 -- 172.21.15.27:0/935380375 shutdown_connections 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55614d4a19c0 0x55614d4a3e10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d48ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d48e610 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f51f9ffb640 0 client.16011 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 --2- 172.21.15.27:0/935380375 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482480 0x55614d491320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 -- 172.21.15.27:0/935380375 >> 172.21.15.27:0/935380375 conn(0x55614d47e8a0 msgr2=0x55614d4846d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 -- 172.21.15.27:0/935380375 shutdown_connections 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 1 -- 172.21.15.27:0/935380375 wait complete. 2024-04-25T01:27:08.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.982+0000 7f521ba24740 20 client.16011 trim_cache size 0 max 16384 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521ba24740 1 Processor -- start 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521ba24740 1 -- start start 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521ba24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d49d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521ba24740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d47ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521ba24740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d490e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521ba24740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52141af2e0 con 0x55614d482da0 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521ba24740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f52141acbb0 con 0x55614d491980 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521ba24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52141aaa00 con 0x55614d482480 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f520bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d47ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f520bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d47ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36272/0 (socket says 172.21.15.27:36272) 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f520bfff640 1 -- 172.21.15.27:0/773029768 learned_addr learned my addr 172.21.15.27:0/773029768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f521259c640 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d49d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f520bfff640 1 -- 172.21.15.27:0/773029768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 msgr2=0x55614d49d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f520bfff640 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d49d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f5212d9d640 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d490e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f520bfff640 1 -- 172.21.15.27:0/773029768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 msgr2=0x55614d490e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f520bfff640 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d490e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.996+0000 7f520bfff640 1 -- 172.21.15.27:0/773029768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55614d1c57d0 con 0x55614d482da0 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f520bfff640 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d47ec80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0bbac0 tx=0x7f51fc0ba2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38f0ebafbe6a804 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f51fbfff640 1 -- 172.21.15.27:0/773029768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc049070 con 0x55614d482da0 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f51fbfff640 1 -- 172.21.15.27:0/773029768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51fc0f2070 con 0x55614d482da0 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f51fbfff640 1 -- 172.21.15.27:0/773029768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc09abe0 con 0x55614d482da0 2024-04-25T01:27:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 -- 172.21.15.27:0/773029768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 msgr2=0x55614d47ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d47ec80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0bbac0 tx=0x7f51fc0ba2d0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 -- 172.21.15.27:0/773029768 shutdown_connections 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d491980 0x55614d490e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482da0 0x55614d47ec80 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 --2- 172.21.15.27:0/773029768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d482480 0x55614d49d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 -- 172.21.15.27:0/773029768 >> 172.21.15.27:0/773029768 conn(0x55614d47e8a0 msgr2=0x55614d489c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 -- 172.21.15.27:0/773029768 shutdown_connections 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.997+0000 7f521ba24740 1 -- 172.21.15.27:0/773029768 wait complete. 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 1 -- start start 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 1 Processor -- start 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482480 0x55614d499fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482da0 0x55614d495320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d4958a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52141ab070 con 0x55614d482480 2024-04-25T01:27:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f52141ac4e0 con 0x55614d482da0 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52141ab740 con 0x55614d491980 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521ba24740 10 client.0 _ll_register_callbacks cb 0x7f51fc009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482480 0x55614d499fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482480 0x55614d499fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36282/0 (socket says 172.21.15.27:36282) 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 -- 172.21.15.27:0/863208664 learned_addr learned my addr 172.21.15.27:0/863208664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f520bfff640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482da0 0x55614d495320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f5212d9d640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d4958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 -- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 msgr2=0x55614d4958a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d4958a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 -- 172.21.15.27:0/863208664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482da0 msgr2=0x55614d495320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482da0 0x55614d495320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 -- 172.21.15.27:0/863208664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55614d4960d0 con 0x55614d482480 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f520bfff640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482da0 0x55614d495320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f5212d9d640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d4958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f521259c640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482480 0x55614d499fb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f520400a310 tx=0x7f520400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8e1a0cf869d60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.998+0000 7f51f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f51f9ffb640 1 -- 172.21.15.27:0/863208664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204028e60 con 0x55614d482480 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f51f9ffb640 1 -- 172.21.15.27:0/863208664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5204005450 con 0x55614d482480 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 1 -- 172.21.15.27:0/863208664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51fc0f2070 con 0x55614d482480 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f51f9ffb640 1 -- 172.21.15.27:0/863208664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f520401d040 con 0x55614d482480 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 10 client.16017 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 1 -- 172.21.15.27:0/863208664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55614d508000 con 0x55614d482480 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f51f9ffb640 1 -- 172.21.15.27:0/863208664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51fc0f2070 con 0x55614d482480 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f51f9ffb640 1 -- 172.21.15.27:0/863208664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f520402f070 con 0x55614d482480 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 10 client.16017 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 10 client.16017 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 20 client.16017 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55614d508570 0x55614d50a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 4 client.16017 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f521ba24740 1 -- 172.21.15.27:0/863208664 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55614d3eef00 con 0x55614d508570 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:08.999+0000 7f520bfff640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55614d508570 0x55614d50a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f520bfff640 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55614d508570 0x55614d50a990 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37e28d1315ae3d1d server_cookie=a2b4effd1f1e8a38 in_seq=0 out_seq=0 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f51f9ffb640 10 client.16017 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f51f9ffb640 1 -- 172.21.15.27:0/863208664 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55614d3eef00 con 0x55614d508570 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f51f9ffb640 10 client.16017 handle_command_reply: tid=0 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 client.16017 shutdown 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 -- 172.21.15.27:0/863208664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482480 msgr2=0x55614d499fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482480 0x55614d499fb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f520400a310 tx=0x7f520400a620 comp rx=0 tx=0).stop 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 -- 172.21.15.27:0/863208664 shutdown_connections 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55614d508570 0x55614d50a990 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55614d491980 0x55614d4958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55614d482da0 0x55614d495320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f51f9ffb640 0 client.16017 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 --2- 172.21.15.27:0/863208664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55614d482480 0x55614d499fb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 -- 172.21.15.27:0/863208664 >> 172.21.15.27:0/863208664 conn(0x55614d47e8a0 msgr2=0x55614d489c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 -- 172.21.15.27:0/863208664 shutdown_connections 2024-04-25T01:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 1 -- 172.21.15.27:0/863208664 wait complete. 2024-04-25T01:27:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.000+0000 7f521ba24740 20 client.16017 trim_cache size 0 max 16384 2024-04-25T01:27:08.986 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 -- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52000a2d80 msgr2=0x7f52000a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52000a2d80 0x7f52000a5240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f520c009270 tx=0x7f520c004a40 comp rx=0 tx=0).stop 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 -- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100fe0 msgr2=0x7f5214198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100fe0 0x7f5214198570 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009230 tx=0x7f51fc004270 comp rx=0 tx=0).stop 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 -- 172.21.15.27:0/813109742 shutdown_connections 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5214106630 0x7f521419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52000a2d80 0x7f52000a5240 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5214100fe0 0x7f5214198570 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 --2- 172.21.15.27:0/813109742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5214100620 0x7f5214198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 -- 172.21.15.27:0/813109742 >> 172.21.15.27:0/813109742 conn(0x7f52140776a0 msgr2=0x7f521406a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 -- 172.21.15.27:0/813109742 shutdown_connections 2024-04-25T01:27:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.001+0000 7f5218c38640 1 -- 172.21.15.27:0/813109742 wait complete. 2024-04-25T01:27:08.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:27:09.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:08 smithi153 ceph-mon[32621]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 33 KiB/s wr, 30 op/s 2024-04-25T01:27:09.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:08 smithi139 ceph-mon[32760]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 33 KiB/s wr, 30 op/s 2024-04-25T01:27:09.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 -- 172.21.15.27:0/2615291841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c107720 msgr2=0x7f803c107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 --2- 172.21.15.27:0/2615291841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c107720 0x7f803c107b00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f802c009930 tx=0x7f802c02a420 comp rx=0 tx=0).stop 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 -- 172.21.15.27:0/2615291841 shutdown_connections 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 --2- 172.21.15.27:0/2615291841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f803c1020d0 0x7f803c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 --2- 172.21.15.27:0/2615291841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f803c101710 0x7f803c101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 --2- 172.21.15.27:0/2615291841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c107720 0x7f803c107b00 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 -- 172.21.15.27:0/2615291841 >> 172.21.15.27:0/2615291841 conn(0x7f803c0fd180 msgr2=0x7f803c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 -- 172.21.15.27:0/2615291841 shutdown_connections 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 -- 172.21.15.27:0/2615291841 wait complete. 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 Processor -- start 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.116+0000 7f804136a640 1 -- start start 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f804136a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f803c101710 0x7f803c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f804136a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c1020d0 0x7f803c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f804136a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f803c107720 0x7f803c19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f804136a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f803c111bb0 con 0x7f803c1020d0 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f804136a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f803c111a30 con 0x7f803c107720 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f804136a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f803c111d30 con 0x7f803c101710 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c1020d0 0x7f803c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c1020d0 0x7f803c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36290/0 (socket says 172.21.15.27:36290) 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 -- 172.21.15.27:0/4050219979 learned_addr learned my addr 172.21.15.27:0/4050219979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f803b7fe640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f803c107720 0x7f803c19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f803affd640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f803c101710 0x7f803c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 -- 172.21.15.27:0/4050219979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f803c101710 msgr2=0x7f803c198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f803c101710 0x7f803c198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 -- 172.21.15.27:0/4050219979 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f803c107720 msgr2=0x7f803c19ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f803c107720 0x7f803c19ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 -- 172.21.15.27:0/4050219979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803c19d160 con 0x7f803c1020d0 2024-04-25T01:27:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8033fff640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c1020d0 0x7f803c1986b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f802400b4a0 tx=0x7f802400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.117+0000 7f8038ff9640 1 -- 172.21.15.27:0/4050219979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024014070 con 0x7f803c1020d0 2024-04-25T01:27:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.118+0000 7f8038ff9640 1 -- 172.21.15.27:0/4050219979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8024010af0 con 0x7f803c1020d0 2024-04-25T01:27:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.118+0000 7f804136a640 1 -- 172.21.15.27:0/4050219979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803c19d450 con 0x7f803c1020d0 2024-04-25T01:27:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.118+0000 7f8038ff9640 1 -- 172.21.15.27:0/4050219979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024017d10 con 0x7f803c1020d0 2024-04-25T01:27:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.118+0000 7f804136a640 1 -- 172.21.15.27:0/4050219979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f803c1a4ce0 con 0x7f803c1020d0 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8038ff9640 1 -- 172.21.15.27:0/4050219979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8024017070 con 0x7f803c1020d0 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8044071740 1 Processor -- start 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8044071740 1 -- start start 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8044071740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082460 0x56151a082840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8044071740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 0x56151a091440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8044071740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a091980 0x56151a093d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8044071740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561519fe7670 con 0x56151a091980 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8044071740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561519f8f970 con 0x56151a082d80 2024-04-25T01:27:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8044071740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561519fe3630 con 0x56151a082460 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.120+0000 7f8038ff9640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80280a2d80 0x7f80280a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f8033fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a091980 0x56151a093d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f8033fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a091980 0x56151a093d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36306/0 (socket says 172.21.15.27:36306) 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082460 0x56151a082840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f8033fff640 1 -- 172.21.15.27:0/248186488 learned_addr learned my addr 172.21.15.27:0/248186488 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803affd640 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 0x56151a091440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80280a2d80 0x7f80280a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f8038ff9640 1 -- 172.21.15.27:0/4050219979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f80240ce910 con 0x7f803c1020d0 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 -- 172.21.15.27:0/248186488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 msgr2=0x56151a091440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 0x56151a091440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 -- 172.21.15.27:0/248186488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a091980 msgr2=0x56151a093d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a091980 0x56151a093d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 -- 172.21.15.27:0/248186488 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561519fb3400 con 0x56151a082460 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f8033fff640 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a091980 0x56151a093d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80280a2d80 0x7f80280a5240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8034009900 tx=0x7f803401b000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f803b7fe640 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082460 0x56151a082840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f803400bf80 tx=0x7f8034008750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1502c5a27e29944 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.121+0000 7f80317fa640 1 -- 172.21.15.27:0/248186488 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8034027070 con 0x56151a082460 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f80317fa640 1 -- 172.21.15.27:0/248186488 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8034009d90 con 0x56151a082460 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f80317fa640 1 -- 172.21.15.27:0/248186488 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803401e070 con 0x56151a082460 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 -- 172.21.15.27:0/248186488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082460 msgr2=0x56151a082840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082460 0x56151a082840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f803400bf80 tx=0x7f8034008750 comp rx=0 tx=0).stop 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 -- 172.21.15.27:0/248186488 shutdown_connections 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a091980 0x56151a093d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 0x56151a091440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 --2- 172.21.15.27:0/248186488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082460 0x56151a082840 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 -- 172.21.15.27:0/248186488 >> 172.21.15.27:0/248186488 conn(0x56151a07e8a0 msgr2=0x56151a07ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 -- 172.21.15.27:0/248186488 shutdown_connections 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.122+0000 7f8044071740 1 -- 172.21.15.27:0/248186488 wait complete. 2024-04-25T01:27:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 1 -- start start 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 1 Processor -- start 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a091250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 0x56151a08e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 0x56151a08ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56151a094760 con 0x56151a082460 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561519fa3050 con 0x56151a082d80 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56151a0948e0 con 0x56151a091980 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8044071740 10 client.0 _ll_register_callbacks cb 0x561519cfabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a091250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a091250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36312/0 (socket says 172.21.15.27:36312) 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803affd640 1 -- 172.21.15.27:0/1398482668 learned_addr learned my addr 172.21.15.27:0/1398482668 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f8033fff640 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 0x56151a08e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803b7fe640 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 0x56151a08ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803affd640 1 -- 172.21.15.27:0/1398482668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 msgr2=0x56151a08ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803affd640 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 0x56151a08ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803affd640 1 -- 172.21.15.27:0/1398482668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 msgr2=0x56151a08e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803affd640 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 0x56151a08e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.123+0000 7f803affd640 1 -- 172.21.15.27:0/1398482668 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561519f74dc0 con 0x56151a082460 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f803affd640 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a091250 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f802c009930 tx=0x7f802c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9f55b2bae9bbb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f80227fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f80227fc640 1 -- 172.21.15.27:0/1398482668 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c035040 con 0x56151a082460 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f80227fc640 1 -- 172.21.15.27:0/1398482668 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f802c004380 con 0x56151a082460 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 1 -- 172.21.15.27:0/1398482668 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8034009d90 con 0x56151a082460 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f80227fc640 1 -- 172.21.15.27:0/1398482668 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f802c04f050 con 0x56151a082460 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 10 client.16029 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 1 -- 172.21.15.27:0/1398482668 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561519cfb3f0 con 0x56151a082460 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f80227fc640 1 -- 172.21.15.27:0/1398482668 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8034009d90 con 0x56151a082460 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f80227fc640 1 -- 172.21.15.27:0/1398482668 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f802c046810 con 0x56151a082460 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 10 client.16029 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 10 client.16029 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 20 client.16029 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:09.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56151a0a19a0 0x56151a0a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 4 client.16029 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.124+0000 7f8044071740 1 -- 172.21.15.27:0/1398482668 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561519ff4bb0 con 0x56151a0a19a0 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.125+0000 7f8033fff640 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56151a0a19a0 0x56151a0a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.125+0000 7f80227fc640 10 client.16029 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.125+0000 7f8033fff640 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56151a0a19a0 0x56151a0a3d90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f95b13f5af96ef7 server_cookie=2a00c43c88a45d5f in_seq=0 out_seq=0 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f80227fc640 1 -- 172.21.15.27:0/1398482668 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561519ff4bb0 con 0x56151a0a19a0 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f80227fc640 10 client.16029 handle_command_reply: tid=0 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 client.16029 shutdown 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 -- 172.21.15.27:0/1398482668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 msgr2=0x56151a091250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a091250 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f802c009930 tx=0x7f802c031a20 comp rx=0 tx=0).stop 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 -- 172.21.15.27:0/1398482668 shutdown_connections 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56151a0a19a0 0x56151a0a3d90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 0x56151a08ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082d80 0x56151a08e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f80227fc640 0 client.16029 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:27:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 --2- 172.21.15.27:0/1398482668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a091250 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 -- 172.21.15.27:0/1398482668 >> 172.21.15.27:0/1398482668 conn(0x56151a07e8a0 msgr2=0x56151a084650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 -- 172.21.15.27:0/1398482668 shutdown_connections 2024-04-25T01:27:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 1 -- 172.21.15.27:0/1398482668 wait complete. 2024-04-25T01:27:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.126+0000 7f8044071740 20 client.16029 trim_cache size 0 max 16384 2024-04-25T01:27:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.139+0000 7f8044071740 1 Processor -- start 2024-04-25T01:27:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f8044071740 1 -- start start 2024-04-25T01:27:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f8044071740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a08ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f8044071740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082d80 0x56151a09d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f8044071740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a091980 0x56151a083dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f8044071740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f803c1af350 con 0x56151a082460 2024-04-25T01:27:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f8044071740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f803c1acc20 con 0x56151a091980 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f8044071740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f803c1aaa70 con 0x56151a082d80 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a08ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a08ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36320/0 (socket says 172.21.15.27:36320) 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803affd640 1 -- 172.21.15.27:0/3016522514 learned_addr learned my addr 172.21.15.27:0/3016522514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f8033fff640 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082d80 0x56151a09d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803b7fe640 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a091980 0x56151a083dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803affd640 1 -- 172.21.15.27:0/3016522514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082d80 msgr2=0x56151a09d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803affd640 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082d80 0x56151a09d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803affd640 1 -- 172.21.15.27:0/3016522514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a091980 msgr2=0x56151a083dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803affd640 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a091980 0x56151a083dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803affd640 1 -- 172.21.15.27:0/3016522514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561519dc57d0 con 0x56151a082460 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.140+0000 7f803b7fe640 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a091980 0x56151a083dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f803affd640 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a08ea20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f802c000c00 tx=0x7f802c031640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657f16f15f7ba504 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8030ff9640 1 -- 172.21.15.27:0/3016522514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c035030 con 0x56151a082460 2024-04-25T01:27:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8030ff9640 1 -- 172.21.15.27:0/3016522514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f802c046960 con 0x56151a082460 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8030ff9640 1 -- 172.21.15.27:0/3016522514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c03db40 con 0x56151a082460 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8044071740 1 -- 172.21.15.27:0/3016522514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 msgr2=0x56151a08ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8044071740 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a08ea20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f802c000c00 tx=0x7f802c031640 comp rx=0 tx=0).stop 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8044071740 1 -- 172.21.15.27:0/3016522514 shutdown_connections 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8044071740 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a091980 0x56151a083dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8044071740 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a082d80 0x56151a09d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.141+0000 7f8044071740 1 --2- 172.21.15.27:0/3016522514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082460 0x56151a08ea20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 -- 172.21.15.27:0/3016522514 >> 172.21.15.27:0/3016522514 conn(0x56151a07e8a0 msgr2=0x56151a08e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 -- 172.21.15.27:0/3016522514 shutdown_connections 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 -- 172.21.15.27:0/3016522514 wait complete. 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 -- start start 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 Processor -- start 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082460 0x56151a099ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082d80 0x56151a09a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 0x56151a096b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f803c1ab0e0 con 0x56151a082d80 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f803c1ac550 con 0x56151a082460 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f803c1ab7b0 con 0x56151a091980 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8044071740 10 client.0 _ll_register_callbacks cb 0x7f802c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8033fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082d80 0x56151a09a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8033fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082d80 0x56151a09a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36334/0 (socket says 172.21.15.27:36334) 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8033fff640 1 -- 172.21.15.27:0/3791300343 learned_addr learned my addr 172.21.15.27:0/3791300343 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f803affd640 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082460 0x56151a099ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f803b7fe640 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 0x56151a096b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8033fff640 1 -- 172.21.15.27:0/3791300343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 msgr2=0x56151a096b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8033fff640 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 0x56151a096b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8033fff640 1 -- 172.21.15.27:0/3791300343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082460 msgr2=0x56151a099ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.142+0000 7f8033fff640 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082460 0x56151a099ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8033fff640 1 -- 172.21.15.27:0/3791300343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56151a097260 con 0x56151a082d80 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8033fff640 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082d80 0x56151a09a0e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f80240970c0 tx=0x7f80240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b44b635cb1c144c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f80227fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f80227fc640 1 -- 172.21.15.27:0/3791300343 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024017360 con 0x56151a082d80 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f80227fc640 1 -- 172.21.15.27:0/3791300343 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8024010ca0 con 0x56151a082d80 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 1 -- 172.21.15.27:0/3791300343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f802c046960 con 0x56151a082d80 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f80227fc640 1 -- 172.21.15.27:0/3791300343 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f80240cbf50 con 0x56151a082d80 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 10 client.16035 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 1 -- 172.21.15.27:0/3791300343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56151a108780 con 0x56151a082d80 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f80227fc640 1 -- 172.21.15.27:0/3791300343 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f802c046960 con 0x56151a082d80 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f80227fc640 1 -- 172.21.15.27:0/3791300343 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f80240b8030 con 0x56151a082d80 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 10 client.16035 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 10 client.16035 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 20 client.16035 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56151a108f00 0x56151a10b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 4 client.16035 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.143+0000 7f8044071740 1 -- 172.21.15.27:0/3791300343 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561519feef00 con 0x56151a108f00 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f803affd640 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56151a108f00 0x56151a10b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f80227fc640 10 client.16035 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f803affd640 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56151a108f00 0x56151a10b320 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c6c385ab01ee80cd server_cookie=9633031c05589fb4 in_seq=0 out_seq=0 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f80227fc640 1 -- 172.21.15.27:0/3791300343 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561519feef00 con 0x56151a108f00 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f80227fc640 10 client.16035 handle_command_reply: tid=0 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 client.16035 shutdown 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 -- 172.21.15.27:0/3791300343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082d80 msgr2=0x56151a09a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082d80 0x56151a09a0e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f80240970c0 tx=0x7f80240046c0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 -- 172.21.15.27:0/3791300343 shutdown_connections 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56151a108f00 0x56151a10b320 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56151a091980 0x56151a096b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56151a082d80 0x56151a09a0e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f80227fc640 0 client.16035 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:27:09.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 --2- 172.21.15.27:0/3791300343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56151a082460 0x56151a099ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 -- 172.21.15.27:0/3791300343 >> 172.21.15.27:0/3791300343 conn(0x56151a07e8a0 msgr2=0x56151a09c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.144+0000 7f8044071740 1 -- 172.21.15.27:0/3791300343 shutdown_connections 2024-04-25T01:27:09.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f8044071740 1 -- 172.21.15.27:0/3791300343 wait complete. 2024-04-25T01:27:09.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f8044071740 20 client.16035 trim_cache size 0 max 16384 2024-04-25T01:27:09.130 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:27:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 -- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80280a2d80 msgr2=0x7f80280a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80280a2d80 0x7f80280a5240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8034009900 tx=0x7f803401b000 comp rx=0 tx=0).stop 2024-04-25T01:27:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 -- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c1020d0 msgr2=0x7f803c1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c1020d0 0x7f803c1986b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f802400b4a0 tx=0x7f802400b960 comp rx=0 tx=0).stop 2024-04-25T01:27:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 -- 172.21.15.27:0/4050219979 shutdown_connections 2024-04-25T01:27:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f803c107720 0x7f803c19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f80280a2d80 0x7f80280a5240 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c1020d0 0x7f803c1986b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 --2- 172.21.15.27:0/4050219979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f803c101710 0x7f803c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 -- 172.21.15.27:0/4050219979 >> 172.21.15.27:0/4050219979 conn(0x7f803c0fd180 msgr2=0x7f803c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 -- 172.21.15.27:0/4050219979 shutdown_connections 2024-04-25T01:27:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.145+0000 7f804136a640 1 -- 172.21.15.27:0/4050219979 wait complete. 2024-04-25T01:27:09.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 -- 172.21.15.27:0/1322842043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 msgr2=0x7f41c0101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 --2- 172.21.15.27:0/1322842043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 0x7f41c0101b90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009930 tx=0x7f41b002a440 comp rx=0 tx=0).stop 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 -- 172.21.15.27:0/1322842043 shutdown_connections 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 --2- 172.21.15.27:0/1322842043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c01020d0 0x7f41c010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 --2- 172.21.15.27:0/1322842043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 0x7f41c0101b90 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 --2- 172.21.15.27:0/1322842043 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41c0107720 0x7f41c0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 -- 172.21.15.27:0/1322842043 >> 172.21.15.27:0/1322842043 conn(0x7f41c0078b90 msgr2=0x7f41c0078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 -- 172.21.15.27:0/1322842043 shutdown_connections 2024-04-25T01:27:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 -- 172.21.15.27:0/1322842043 wait complete. 2024-04-25T01:27:09.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.260+0000 7f41c6261640 1 Processor -- start 2024-04-25T01:27:09.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41c6261640 1 -- start start 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41c6261640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 0x7f41c01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41c6261640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41c01020d0 0x7f41c0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41c6261640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c0107720 0x7f41c019cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41c6261640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41c0111bd0 con 0x7f41c0101710 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41c6261640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41c0111a50 con 0x7f41c01020d0 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41c6261640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41c0111d50 con 0x7f41c0107720 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 0x7f41c01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 0x7f41c01981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36358/0 (socket says 172.21.15.27:36358) 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bf7fe640 1 -- 172.21.15.27:0/1216972813 learned_addr learned my addr 172.21.15.27:0/1216972813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41beffd640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41c01020d0 0x7f41c0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bffff640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c0107720 0x7f41c019cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bf7fe640 1 -- 172.21.15.27:0/1216972813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c0107720 msgr2=0x7f41c019cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c0107720 0x7f41c019cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bf7fe640 1 -- 172.21.15.27:0/1216972813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41c01020d0 msgr2=0x7f41c0198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41c01020d0 0x7f41c0198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.261+0000 7f41bf7fe640 1 -- 172.21.15.27:0/1216972813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c019d1e0 con 0x7f41c0101710 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.262+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 0x7f41c01981f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f41b400b450 tx=0x7f41b400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.262+0000 7f41bcff9640 1 -- 172.21.15.27:0/1216972813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4015020 con 0x7f41c0101710 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.262+0000 7f41bcff9640 1 -- 172.21.15.27:0/1216972813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b4011d70 con 0x7f41c0101710 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.262+0000 7f41c6261640 1 -- 172.21.15.27:0/1216972813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c019d4d0 con 0x7f41c0101710 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.262+0000 7f41bcff9640 1 -- 172.21.15.27:0/1216972813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b400f340 con 0x7f41c0101710 2024-04-25T01:27:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.262+0000 7f41c6261640 1 -- 172.21.15.27:0/1216972813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c01a4d60 con 0x7f41c0101710 2024-04-25T01:27:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.264+0000 7f41bcff9640 1 -- 172.21.15.27:0/1216972813 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f41b4018020 con 0x7f41c0101710 2024-04-25T01:27:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.264+0000 7f41c8f68740 1 Processor -- start 2024-04-25T01:27:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.264+0000 7f41c8f68740 1 -- start start 2024-04-25T01:27:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 0x564cc2ac4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4ea0 0x564cc2acf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2ad1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41c8f68740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564cc2a29670 con 0x564cc2acf570 2024-04-25T01:27:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41c8f68740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564cc29d1970 con 0x564cc2ac4ea0 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41c8f68740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564cc2a25630 con 0x564cc2ac4580 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41bcff9640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41a80a2d00 0x7f41a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2ad1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2ad1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36372/0 (socket says 172.21.15.27:36372) 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 -- 172.21.15.27:0/36735705 learned_addr learned my addr 172.21.15.27:0/36735705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41bffff640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41a80a2d00 0x7f41a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41bcff9640 1 -- 172.21.15.27:0/1216972813 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f41b40cf170 con 0x7f41c0101710 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41bffff640 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 0x564cc2ac4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 -- 172.21.15.27:0/36735705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 msgr2=0x564cc2ac4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 0x564cc2ac4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 -- 172.21.15.27:0/36735705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4ea0 msgr2=0x564cc2acf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4ea0 0x564cc2acf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4ea0 0x564cc2acf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 -- 172.21.15.27:0/36735705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cc29f5400 con 0x564cc2acf570 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41bffff640 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 0x564cc2ac4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4ea0 0x564cc2acf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41beffd640 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2ad1960 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009930 tx=0x7f41b00316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862faeb7b032bcba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.265+0000 7f41bffff640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41a80a2d00 0x7f41a80a51c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009fd0 tx=0x7f41b8009540 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41a5ffb640 1 -- 172.21.15.27:0/36735705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0040070 con 0x564cc2acf570 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41a5ffb640 1 -- 172.21.15.27:0/36735705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b0005020 con 0x564cc2acf570 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41a5ffb640 1 -- 172.21.15.27:0/36735705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0035040 con 0x564cc2acf570 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 -- 172.21.15.27:0/36735705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 msgr2=0x564cc2ad1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2ad1960 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009930 tx=0x7f41b00316e0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 -- 172.21.15.27:0/36735705 shutdown_connections 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2ad1960 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4ea0 0x564cc2acf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 --2- 172.21.15.27:0/36735705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 0x564cc2ac4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 -- 172.21.15.27:0/36735705 >> 172.21.15.27:0/36735705 conn(0x564cc2ac08a0 msgr2=0x564cc2ac0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 -- 172.21.15.27:0/36735705 shutdown_connections 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.266+0000 7f41c8f68740 1 -- 172.21.15.27:0/36735705 wait complete. 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 1 -- start start 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 1 Processor -- start 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2aceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4ea0 0x564cc2acc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2acc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564cc2ad2300 con 0x564cc2acf570 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564cc29e5050 con 0x564cc2ac4580 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564cc2ad2480 con 0x564cc2ac4ea0 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41c8f68740 10 client.0 _ll_register_callbacks cb 0x564cc273cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2acc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2acc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36382/0 (socket says 172.21.15.27:36382) 2024-04-25T01:27:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.267+0000 7f41bffff640 1 -- 172.21.15.27:0/3128843675 learned_addr learned my addr 172.21.15.27:0/3128843675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2aceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41bffff640 1 -- 172.21.15.27:0/3128843675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4ea0 msgr2=0x564cc2acc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41bffff640 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4ea0 0x564cc2acc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41bffff640 1 -- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 msgr2=0x564cc2aceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41bffff640 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2aceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41bffff640 1 -- 172.21.15.27:0/3128843675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cc29b6dc0 con 0x564cc2acf570 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2aceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41bffff640 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2acc720 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f41b800af50 tx=0x7f41b8004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d0cdc685ab7dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f419effd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f419effd640 1 -- 172.21.15.27:0/3128843675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8029060 con 0x564cc2acf570 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f419effd640 1 -- 172.21.15.27:0/3128843675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f41b8019920 con 0x564cc2acf570 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41c8f68740 1 -- 172.21.15.27:0/3128843675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41b0005020 con 0x564cc2acf570 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f419effd640 1 -- 172.21.15.27:0/3128843675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f41b8030050 con 0x564cc2acf570 2024-04-25T01:27:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41c8f68740 10 client.16047 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f41c8f68740 1 -- 172.21.15.27:0/3128843675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564cc273d3f0 con 0x564cc2acf570 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.268+0000 7f419effd640 1 -- 172.21.15.27:0/3128843675 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b0005020 con 0x564cc2acf570 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.269+0000 7f419effd640 1 -- 172.21.15.27:0/3128843675 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f41b8024c70 con 0x564cc2acf570 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.269+0000 7f41c8f68740 10 client.16047 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.269+0000 7f41c8f68740 10 client.16047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.269+0000 7f41c8f68740 20 client.16047 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.269+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564cc2adf660 0x564cc2ae1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.269+0000 7f41c8f68740 4 client.16047 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.269+0000 7f41c8f68740 1 -- 172.21.15.27:0/3128843675 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cc2a36bb0 con 0x564cc2adf660 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.269+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564cc2adf660 0x564cc2ae1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.270+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564cc2adf660 0x564cc2ae1b00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8797631ba97c0f13 server_cookie=7058105309d0ec79 in_seq=0 out_seq=0 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.270+0000 7f419effd640 10 client.16047 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.270+0000 7f419effd640 1 -- 172.21.15.27:0/3128843675 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564cc2a36bb0 con 0x564cc2adf660 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.270+0000 7f419effd640 10 client.16047 handle_command_reply: tid=0 2024-04-25T01:27:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.270+0000 7f41c8f68740 1 client.16047 shutdown 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 -- 172.21.15.27:0/3128843675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 msgr2=0x564cc2acc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2acc720 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f41b800af50 tx=0x7f41b8004b50 comp rx=0 tx=0).stop 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 -- 172.21.15.27:0/3128843675 shutdown_connections 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564cc2adf660 0x564cc2ae1b00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2acf570 0x564cc2acc720 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4ea0 0x564cc2acc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f419effd640 0 client.16047 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3128843675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2aceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 -- 172.21.15.27:0/3128843675 >> 172.21.15.27:0/3128843675 conn(0x564cc2ac08a0 msgr2=0x564cc2ac66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 -- 172.21.15.27:0/3128843675 shutdown_connections 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 1 -- 172.21.15.27:0/3128843675 wait complete. 2024-04-25T01:27:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.271+0000 7f41c8f68740 20 client.16047 trim_cache size 0 max 16384 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41c8f68740 1 Processor -- start 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41c8f68740 1 -- start start 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 0x564cc2adb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2abfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2acf570 0x564cc2acea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41c8f68740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01b1660 con 0x564cc2ac4ea0 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41c8f68740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01aef30 con 0x564cc2acf570 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41c8f68740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01acd80 con 0x564cc2ac4580 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2abfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2abfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36388/0 (socket says 172.21.15.27:36388) 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.284+0000 7f41beffd640 1 -- 172.21.15.27:0/3987940645 learned_addr learned my addr 172.21.15.27:0/3987940645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41beffd640 1 -- 172.21.15.27:0/3987940645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 msgr2=0x564cc2adb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41beffd640 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 0x564cc2adb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41beffd640 1 -- 172.21.15.27:0/3987940645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2acf570 msgr2=0x564cc2acea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41bffff640 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2acf570 0x564cc2acea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41beffd640 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2acf570 0x564cc2acea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41beffd640 1 -- 172.21.15.27:0/3987940645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cc28077d0 con 0x564cc2ac4ea0 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41bffff640 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2acf570 0x564cc2acea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41beffd640 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2abfd80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f41b0042000 tx=0x7f41b0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99cf5de392fef793 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41a57fa640 1 -- 172.21.15.27:0/3987940645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0040070 con 0x564cc2ac4ea0 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41a57fa640 1 -- 172.21.15.27:0/3987940645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b0048070 con 0x564cc2ac4ea0 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.285+0000 7f41a57fa640 1 -- 172.21.15.27:0/3987940645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0035040 con 0x564cc2ac4ea0 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 -- 172.21.15.27:0/3987940645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 msgr2=0x564cc2abfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2abfd80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f41b0042000 tx=0x7f41b0005850 comp rx=0 tx=0).stop 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 -- 172.21.15.27:0/3987940645 shutdown_connections 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2acf570 0x564cc2acea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2abfd80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3987940645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2ac4580 0x564cc2adb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 -- 172.21.15.27:0/3987940645 >> 172.21.15.27:0/3987940645 conn(0x564cc2ac08a0 msgr2=0x564cc2adaaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 -- 172.21.15.27:0/3987940645 shutdown_connections 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 -- 172.21.15.27:0/3987940645 wait complete. 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 -- start start 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 Processor -- start 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.286+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2ad59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2ad5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2acf570 0x564cc2b3ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01ad3f0 con 0x564cc2ac4ea0 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01ae860 con 0x564cc2ac4580 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01adac0 con 0x564cc2acf570 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 10 client.0 _ll_register_callbacks cb 0x7f41b800a5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2ad5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2ad5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36396/0 (socket says 172.21.15.27:36396) 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 -- 172.21.15.27:0/3635856585 learned_addr learned my addr 172.21.15.27:0/3635856585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2ad59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41bffff640 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2acf570 0x564cc2b3ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 -- 172.21.15.27:0/3635856585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2acf570 msgr2=0x564cc2b3ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2acf570 0x564cc2b3ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 -- 172.21.15.27:0/3635856585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 msgr2=0x564cc2ad59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2ad59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 -- 172.21.15.27:0/3635856585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cc2b3f2c0 con 0x564cc2ac4ea0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41bffff640 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2acf570 0x564cc2b3ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41beffd640 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2ad5f10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f41b0006010 tx=0x7f41b00058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6819d646d01fb6e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f419effd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f419effd640 1 -- 172.21.15.27:0/3635856585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0051070 con 0x564cc2ac4ea0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f419effd640 1 -- 172.21.15.27:0/3635856585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f41b0003d50 con 0x564cc2ac4ea0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 1 -- 172.21.15.27:0/3635856585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41b0048070 con 0x564cc2ac4ea0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f419effd640 1 -- 172.21.15.27:0/3635856585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f41b005d050 con 0x564cc2ac4ea0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 10 client.16053 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.287+0000 7f41c8f68740 1 -- 172.21.15.27:0/3635856585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564cc2b465d0 con 0x564cc2ac4ea0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f419effd640 1 -- 172.21.15.27:0/3635856585 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b0048070 con 0x564cc2ac4ea0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f419effd640 1 -- 172.21.15.27:0/3635856585 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f41b004db40 con 0x564cc2ac4ea0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f41c8f68740 10 client.16053 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f41c8f68740 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f41c8f68740 20 client.16053 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564cc2b46900 0x564cc2b48d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f41c8f68740 4 client.16053 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f41c8f68740 1 -- 172.21.15.27:0/3635856585 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564cc2a30f00 con 0x564cc2b46900 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.288+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564cc2b46900 0x564cc2b48d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41bf7fe640 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564cc2b46900 0x564cc2b48d20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d1836bded04c512 server_cookie=37a10c54fdddcbee in_seq=0 out_seq=0 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f419effd640 10 client.16053 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f419effd640 1 -- 172.21.15.27:0/3635856585 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564cc2a30f00 con 0x564cc2b46900 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f419effd640 10 client.16053 handle_command_reply: tid=0 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 client.16053 shutdown 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 -- 172.21.15.27:0/3635856585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 msgr2=0x564cc2ad5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2ad5f10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f41b0006010 tx=0x7f41b00058d0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 -- 172.21.15.27:0/3635856585 shutdown_connections 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564cc2b46900 0x564cc2b48d20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564cc2acf570 0x564cc2b3ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564cc2ac4ea0 0x564cc2ad5f10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f419effd640 0 client.16053 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 --2- 172.21.15.27:0/3635856585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564cc2ac4580 0x564cc2ad59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 -- 172.21.15.27:0/3635856585 >> 172.21.15.27:0/3635856585 conn(0x564cc2ac08a0 msgr2=0x564cc2acd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 -- 172.21.15.27:0/3635856585 shutdown_connections 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 1 -- 172.21.15.27:0/3635856585 wait complete. 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.289+0000 7f41c8f68740 20 client.16053 trim_cache size 0 max 16384 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:27:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 -- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41a80a2d00 msgr2=0x7f41a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41a80a2d00 0x7f41a80a51c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009fd0 tx=0x7f41b8009540 comp rx=0 tx=0).stop 2024-04-25T01:27:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 -- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 msgr2=0x7f41c01981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 0x7f41c01981f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f41b400b450 tx=0x7f41b400b910 comp rx=0 tx=0).stop 2024-04-25T01:27:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 -- 172.21.15.27:0/1216972813 shutdown_connections 2024-04-25T01:27:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41c0107720 0x7f41c019cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41c01020d0 0x7f41c0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41a80a2d00 0x7f41a80a51c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 --2- 172.21.15.27:0/1216972813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c0101710 0x7f41c01981f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 -- 172.21.15.27:0/1216972813 >> 172.21.15.27:0/1216972813 conn(0x7f41c0078b90 msgr2=0x7f41c00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 -- 172.21.15.27:0/1216972813 shutdown_connections 2024-04-25T01:27:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:09.290+0000 7f41c6261640 1 -- 172.21.15.27:0/1216972813 wait complete. 2024-04-25T01:27:09.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:09 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/388296181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:10.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:09 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/388296181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:10.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:09 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/388296181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:10.280 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:27:10.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:10.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.422+0000 7f0e5d730640 1 -- 172.21.15.27:0/3200762272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 msgr2=0x7f0e5810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.422+0000 7f0e5d730640 1 --2- 172.21.15.27:0/3200762272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 0x7f0e5810b7c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0e40009900 tx=0x7f0e4002a510 comp rx=0 tx=0).stop 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.422+0000 7f0e5d730640 1 -- 172.21.15.27:0/3200762272 shutdown_connections 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.422+0000 7f0e5d730640 1 --2- 172.21.15.27:0/3200762272 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e5810bd00 0x7f0e5810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.422+0000 7f0e5d730640 1 --2- 172.21.15.27:0/3200762272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 0x7f0e5810b7c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.422+0000 7f0e5d730640 1 --2- 172.21.15.27:0/3200762272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e581029c0 0x7f0e58102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.423+0000 7f0e5d730640 1 -- 172.21.15.27:0/3200762272 >> 172.21.15.27:0/3200762272 conn(0x7f0e58077740 msgr2=0x7f0e58077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.423+0000 7f0e5d730640 1 -- 172.21.15.27:0/3200762272 shutdown_connections 2024-04-25T01:27:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.423+0000 7f0e5d730640 1 -- 172.21.15.27:0/3200762272 wait complete. 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.423+0000 7f0e5d730640 1 Processor -- start 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.423+0000 7f0e5d730640 1 -- start start 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e5d730640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e581029c0 0x7f0e58198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e5d730640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 0x7f0e581985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e5d730640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e5810bd00 0x7f0e5819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e5d730640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58111b00 con 0x7f0e581032e0 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e5d730640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58111980 con 0x7f0e581029c0 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e5d730640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58111c80 con 0x7f0e5810bd00 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 0x7f0e581985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 0x7f0e581985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36416/0 (socket says 172.21.15.27:36416) 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 -- 172.21.15.27:0/2900679910 learned_addr learned my addr 172.21.15.27:0/2900679910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e577fe640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e5810bd00 0x7f0e5819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e581029c0 0x7f0e58198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 -- 172.21.15.27:0/2900679910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e5810bd00 msgr2=0x7f0e5819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e5810bd00 0x7f0e5819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 -- 172.21.15.27:0/2900679910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e581029c0 msgr2=0x7f0e58198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e581029c0 0x7f0e58198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 -- 172.21.15.27:0/2900679910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5819d070 con 0x7f0e581032e0 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e4e7fc640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 0x7f0e581985c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0e40006010 tx=0x7f0e400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e54ff9640 1 -- 172.21.15.27:0/2900679910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4003f030 con 0x7f0e581032e0 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e54ff9640 1 -- 172.21.15.27:0/2900679910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e40003ce0 con 0x7f0e581032e0 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e5d730640 1 -- 172.21.15.27:0/2900679910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5819d300 con 0x7f0e581032e0 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.424+0000 7f0e54ff9640 1 -- 172.21.15.27:0/2900679910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40035070 con 0x7f0e581032e0 2024-04-25T01:27:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.425+0000 7f0e5d730640 1 -- 172.21.15.27:0/2900679910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e581a4bf0 con 0x7f0e581032e0 2024-04-25T01:27:10.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.427+0000 7f0e54ff9640 1 -- 172.21.15.27:0/2900679910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0e40042030 con 0x7f0e581032e0 2024-04-25T01:27:10.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.427+0000 7f0e6043f740 1 Processor -- start 2024-04-25T01:27:10.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.427+0000 7f0e6043f740 1 -- start start 2024-04-25T01:27:10.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358bbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bc290 0x55d2358bc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358c7010 0x55d2358c9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e6043f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d2358202c0 con 0x55d2358bc290 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e6043f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d2357c8910 con 0x55d2358bd980 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e6043f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d23581c280 con 0x55d2358c7010 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bc290 0x55d2358bc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bc290 0x55d2358bc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36432/0 (socket says 172.21.15.27:36432) 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 -- 172.21.15.27:0/1175708640 learned_addr learned my addr 172.21.15.27:0/1175708640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e577fe640 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358bbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e54ff9640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0e440a2d00 0x7f0e440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 -- 172.21.15.27:0/1175708640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358c7010 msgr2=0x55d2358c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358c7010 0x55d2358c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 -- 172.21.15.27:0/1175708640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 msgr2=0x55d2358bbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358bbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 -- 172.21.15.27:0/1175708640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2357d9fb0 con 0x55d2358bc290 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e577fe640 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358bbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e577fe640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0e440a2d00 0x7f0e440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e54ff9640 1 -- 172.21.15.27:0/2900679910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0e400efb20 con 0x7f0e581032e0 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.428+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bc290 0x55d2358bc710 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800bcf0 tx=0x7f0e4800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfe4312bdd2de5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.429+0000 7f0e4d7fa640 1 -- 172.21.15.27:0/1175708640 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48015070 con 0x55d2358bc290 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.429+0000 7f0e4d7fa640 1 -- 172.21.15.27:0/1175708640 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e48011920 con 0x55d2358bc290 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.429+0000 7f0e4d7fa640 1 -- 172.21.15.27:0/1175708640 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48010930 con 0x55d2358bc290 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.429+0000 7f0e577fe640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0e440a2d00 0x7f0e440a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0e50008e00 tx=0x7f0e50004a40 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.429+0000 7f0e6043f740 1 -- 172.21.15.27:0/1175708640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bc290 msgr2=0x55d2358bc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.429+0000 7f0e6043f740 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bc290 0x55d2358bc710 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800bcf0 tx=0x7f0e4800f730 comp rx=0 tx=0).stop 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 -- 172.21.15.27:0/1175708640 shutdown_connections 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358c7010 0x55d2358c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bc290 0x55d2358bc710 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 --2- 172.21.15.27:0/1175708640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358bbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 -- 172.21.15.27:0/1175708640 >> 172.21.15.27:0/1175708640 conn(0x55d235884950 msgr2=0x55d2358b87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 -- 172.21.15.27:0/1175708640 shutdown_connections 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 -- 172.21.15.27:0/1175708640 wait complete. 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 -- start start 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 Processor -- start 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 0x55d2358c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bd980 0x55d2358c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358c7010 0x55d2358c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d2358c9ea0 con 0x55d2358bd980 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d2357dbe90 con 0x55d2358c7010 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d2358ca020 con 0x55d2358bc290 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e6043f740 10 client.0 _ll_register_callbacks cb 0x55d235817970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e4e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bd980 0x55d2358c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e4e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bd980 0x55d2358c44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36440/0 (socket says 172.21.15.27:36440) 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.430+0000 7f0e4e7fc640 1 -- 172.21.15.27:0/908440968 learned_addr learned my addr 172.21.15.27:0/908440968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e4e7fc640 1 -- 172.21.15.27:0/908440968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 msgr2=0x55d2358c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e4e7fc640 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 0x55d2358c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e4e7fc640 1 -- 172.21.15.27:0/908440968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358c7010 msgr2=0x55d2358c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e4e7fc640 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358c7010 0x55d2358c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e4e7fc640 1 -- 172.21.15.27:0/908440968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2355fc7e0 con 0x55d2358bd980 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e4e7fc640 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bd980 0x55d2358c44d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0e400bb9b0 tx=0x7f0e400b7f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744b56952b17059a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e3e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/908440968 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40049070 con 0x55d2358bd980 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/908440968 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0e400b8320 con 0x55d2358bd980 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e6043f740 1 -- 172.21.15.27:0/908440968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e48011920 con 0x55d2358bd980 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/908440968 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0e400bb7f0 con 0x55d2358bd980 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e6043f740 10 client.16065 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e6043f740 1 -- 172.21.15.27:0/908440968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d23565cf70 con 0x55d2358bd980 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.431+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/908440968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e48011920 con 0x55d2358bd980 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/908440968 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f0e4004c060 con 0x55d2358bd980 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e6043f740 10 client.16065 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e6043f740 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e6043f740 20 client.16065 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e6043f740 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d2358d70d0 0x55d2358d9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e6043f740 4 client.16065 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e6043f740 1 -- 172.21.15.27:0/908440968 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2357f45a0 con 0x55d2358d70d0 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d2358d70d0 0x55d2358d9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d2358d70d0 0x55d2358d9560 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9fa841f5408c981 server_cookie=574fd98299640761 in_seq=0 out_seq=0 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.432+0000 7f0e3e7fc640 10 client.16065 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.433+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/908440968 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d2357f45a0 con 0x55d2358d70d0 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.433+0000 7f0e3e7fc640 10 client.16065 handle_command_reply: tid=0 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.433+0000 7f0e6043f740 1 client.16065 shutdown 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 -- 172.21.15.27:0/908440968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bd980 msgr2=0x55d2358c44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bd980 0x55d2358c44d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0e400bb9b0 tx=0x7f0e400b7f90 comp rx=0 tx=0).stop 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 -- 172.21.15.27:0/908440968 shutdown_connections 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d2358d70d0 0x55d2358d9560 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358c7010 0x55d2358c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358bd980 0x55d2358c44d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e3e7fc640 0 client.16065 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 --2- 172.21.15.27:0/908440968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 0x55d2358c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 -- 172.21.15.27:0/908440968 >> 172.21.15.27:0/908440968 conn(0x55d235884950 msgr2=0x55d2358be660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 -- 172.21.15.27:0/908440968 shutdown_connections 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 1 -- 172.21.15.27:0/908440968 wait complete. 2024-04-25T01:27:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.434+0000 7f0e6043f740 20 client.16065 trim_cache size 0 max 16384 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e6043f740 1 Processor -- start 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e6043f740 1 -- start start 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bc290 0x55d2358b7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bd980 0x55d2358d2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e6043f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b1430 con 0x55d2358c7010 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e6043f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581aed00 con 0x55d2358bc290 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e6043f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581acb50 con 0x55d2358bd980 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358be4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36448/0 (socket says 172.21.15.27:36448) 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.452+0000 7f0e577fe640 1 -- 172.21.15.27:0/3789781758 learned_addr learned my addr 172.21.15.27:0/3789781758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e4e7fc640 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bd980 0x55d2358d2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bc290 0x55d2358b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e577fe640 1 -- 172.21.15.27:0/3789781758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bd980 msgr2=0x55d2358d2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e577fe640 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bd980 0x55d2358d2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e577fe640 1 -- 172.21.15.27:0/3789781758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bc290 msgr2=0x55d2358b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e577fe640 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bc290 0x55d2358b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e577fe640 1 -- 172.21.15.27:0/3789781758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2354f4170 con 0x55d2358c7010 2024-04-25T01:27:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bc290 0x55d2358b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e577fe640 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358be4e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0e50012550 tx=0x7f0e50004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda929c6234a1abd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e4cff9640 1 -- 172.21.15.27:0/3789781758 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5001d870 con 0x55d2358c7010 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e4cff9640 1 -- 172.21.15.27:0/3789781758 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e5001b450 con 0x55d2358c7010 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.453+0000 7f0e4cff9640 1 -- 172.21.15.27:0/3789781758 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e500266a0 con 0x55d2358c7010 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- 172.21.15.27:0/3789781758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 msgr2=0x55d2358be4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358be4e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0e50012550 tx=0x7f0e50004ab0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- 172.21.15.27:0/3789781758 shutdown_connections 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358be4e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bd980 0x55d2358d2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 --2- 172.21.15.27:0/3789781758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bc290 0x55d2358b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- 172.21.15.27:0/3789781758 >> 172.21.15.27:0/3789781758 conn(0x55d235884950 msgr2=0x55d2358bddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- 172.21.15.27:0/3789781758 shutdown_connections 2024-04-25T01:27:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- 172.21.15.27:0/3789781758 wait complete. 2024-04-25T01:27:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- start start 2024-04-25T01:27:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 Processor -- start 2024-04-25T01:27:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 0x55d2358cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581ad1c0 con 0x55d2358c7010 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581ae630 con 0x55d2358bd980 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.454+0000 7f0e6043f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581ad890 con 0x55d2358bc290 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e6043f740 10 client.0 _ll_register_callbacks cb 0x7f0e400044e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36452/0 (socket says 172.21.15.27:36452) 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 -- 172.21.15.27:0/2127707262 learned_addr learned my addr 172.21.15.27:0/2127707262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 0x55d2358cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e4e7fc640 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 -- 172.21.15.27:0/2127707262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 msgr2=0x55d2358cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 0x55d2358cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 -- 172.21.15.27:0/2127707262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 msgr2=0x55d2358cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358cc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 -- 172.21.15.27:0/2127707262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d23593d7b0 con 0x55d2358c7010 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e577fe640 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358ccd30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0e50012de0 tx=0x7f0e5001b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270dd1ec5c6b1d93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e3e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/2127707262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5001d670 con 0x55d2358c7010 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/2127707262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0e5001d810 con 0x55d2358c7010 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e6043f740 1 -- 172.21.15.27:0/2127707262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e5001b450 con 0x55d2358c7010 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/2127707262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0e5003c050 con 0x55d2358c7010 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e6043f740 10 client.16071 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.455+0000 7f0e6043f740 1 -- 172.21.15.27:0/2127707262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d23593ea00 con 0x55d2358c7010 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/2127707262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e5001b450 con 0x55d2358c7010 2024-04-25T01:27:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/2127707262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f0e50024020 con 0x55d2358c7010 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e6043f740 10 client.16071 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e6043f740 10 client.16071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e6043f740 20 client.16071 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e6043f740 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d23593f030 0x55d235941450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e6043f740 4 client.16071 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e6043f740 1 -- 172.21.15.27:0/2127707262 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d23582d800 con 0x55d23593f030 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.456+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d23593f030 0x55d235941450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.457+0000 7f0e56ffd640 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d23593f030 0x55d235941450 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b96527625a73a2d server_cookie=d885147daff7d502 in_seq=0 out_seq=0 2024-04-25T01:27:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.457+0000 7f0e3e7fc640 10 client.16071 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:10.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e3e7fc640 1 -- 172.21.15.27:0/2127707262 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d23582d800 con 0x55d23593f030 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e3e7fc640 10 client.16071 handle_command_reply: tid=0 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 client.16071 shutdown 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 -- 172.21.15.27:0/2127707262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 msgr2=0x55d2358ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358ccd30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0e50012de0 tx=0x7f0e5001b830 comp rx=0 tx=0).stop 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 -- 172.21.15.27:0/2127707262 shutdown_connections 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d23593f030 0x55d235941450 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e3e7fc640 0 client.16071 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d2358c7010 0x55d2358ccd30 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d2358bd980 0x55d2358cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 --2- 172.21.15.27:0/2127707262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d2358bc290 0x55d2358cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 -- 172.21.15.27:0/2127707262 >> 172.21.15.27:0/2127707262 conn(0x55d235884950 msgr2=0x55d2358c47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 -- 172.21.15.27:0/2127707262 shutdown_connections 2024-04-25T01:27:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 1 -- 172.21.15.27:0/2127707262 wait complete. 2024-04-25T01:27:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.458+0000 7f0e6043f740 20 client.16071 trim_cache size 0 max 16384 2024-04-25T01:27:10.445 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:27:10.445 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:27:10.445 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "b3fbeceb-69b5-43a1-8e76-8e68ecc37247", 2024-04-25T01:27:10.445 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:27:10.445 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:27:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 -- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0e440a2d00 msgr2=0x7f0e440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0e440a2d00 0x7f0e440a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0e50008e00 tx=0x7f0e50004a40 comp rx=0 tx=0).stop 2024-04-25T01:27:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 -- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 msgr2=0x7f0e581985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 0x7f0e581985c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0e40006010 tx=0x7f0e400042f0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 -- 172.21.15.27:0/2900679910 shutdown_connections 2024-04-25T01:27:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e5810bd00 0x7f0e5819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0e440a2d00 0x7f0e440a51c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e581032e0 0x7f0e581985c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 --2- 172.21.15.27:0/2900679910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e581029c0 0x7f0e58198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 -- 172.21.15.27:0/2900679910 >> 172.21.15.27:0/2900679910 conn(0x7f0e58077740 msgr2=0x7f0e580fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 -- 172.21.15.27:0/2900679910 shutdown_connections 2024-04-25T01:27:10.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.459+0000 7f0e5d730640 1 -- 172.21.15.27:0/2900679910 wait complete. 2024-04-25T01:27:10.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:27:10.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:27:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/2843255216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 msgr2=0x7f1c9810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/2843255216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 0x7f1c9810e5f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009cd0 tx=0x7f1c9002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/2843255216 shutdown_connections 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/2843255216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 0x7f1c9810e5f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/2843255216 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c98069dc0 0x7f1c9806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/2843255216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c98107730 0x7f1c98107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/2843255216 >> 172.21.15.27:0/2843255216 conn(0x7f1c980776a0 msgr2=0x7f1c98077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/2843255216 shutdown_connections 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.574+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/2843255216 wait complete. 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c9d8cd640 1 Processor -- start 2024-04-25T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c9d8cd640 1 -- start start 2024-04-25T01:27:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c9d8cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c98069dc0 0x7f1c98100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c9d8cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 0x7f1c98100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c9d8cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c98107730 0x7f1c981010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c9d8cd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1c98111b00 con 0x7f1c9806a780 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c9d8cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1c98111980 con 0x7f1c98107730 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c9d8cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c98111c80 con 0x7f1c98069dc0 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c967fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 0x7f1c98100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c967fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 0x7f1c98100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36466/0 (socket says 172.21.15.27:36466) 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.575+0000 7f1c967fc640 1 -- 172.21.15.27:0/551324023 learned_addr learned my addr 172.21.15.27:0/551324023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c977fe640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c98107730 0x7f1c981010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c98069dc0 0x7f1c98100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c967fc640 1 -- 172.21.15.27:0/551324023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c98069dc0 msgr2=0x7f1c98100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c967fc640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c98069dc0 0x7f1c98100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c967fc640 1 -- 172.21.15.27:0/551324023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c98107730 msgr2=0x7f1c981010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c967fc640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c98107730 0x7f1c981010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c967fc640 1 -- 172.21.15.27:0/551324023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c981a6460 con 0x7f1c9806a780 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c977fe640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c98107730 0x7f1c981010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c98069dc0 0x7f1c98100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c967fc640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 0x7f1c98100bb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00d950 tx=0x7f1c8c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c9c8cb640 1 -- 172.21.15.27:0/551324023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c017070 con 0x7f1c9806a780 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/551324023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c981a6750 con 0x7f1c9806a780 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c9c8cb640 1 -- 172.21.15.27:0/551324023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c009c30 con 0x7f1c9806a780 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c9c8cb640 1 -- 172.21.15.27:0/551324023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c00f6b0 con 0x7f1c9806a780 2024-04-25T01:27:10.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.576+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/551324023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c981a6ca0 con 0x7f1c9806a780 2024-04-25T01:27:10.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1c9c8cb640 1 -- 172.21.15.27:0/551324023 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1c8c0109d0 con 0x7f1c9806a780 2024-04-25T01:27:10.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1ca05d9740 1 Processor -- start 2024-04-25T01:27:10.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1ca05d9740 1 -- start start 2024-04-25T01:27:10.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926630 0x55c9cc926a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926f50 0x55c9cc9310e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc933a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc88b6b0 con 0x55c9cc926f50 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc833880 con 0x55c9cc926630 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.579+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc887670 con 0x55c9cc931620 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926f50 0x55c9cc9310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926f50 0x55c9cc9310e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36472/0 (socket says 172.21.15.27:36472) 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c9c8cb640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1c800a2d30 0x7f1c800a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 -- 172.21.15.27:0/2685367259 learned_addr learned my addr 172.21.15.27:0/2685367259 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c967fc640 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc933a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c977fe640 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926630 0x55c9cc926a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c977fe640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1c800a2d30 0x7f1c800a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 -- 172.21.15.27:0/2685367259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 msgr2=0x55c9cc933a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc933a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 -- 172.21.15.27:0/2685367259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926630 msgr2=0x55c9cc926a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c9c8cb640 1 -- 172.21.15.27:0/551324023 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f1c8c0cf4b0 con 0x7f1c9806a780 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926630 0x55c9cc926a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 -- 172.21.15.27:0/2685367259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9cc6697d0 con 0x55c9cc926f50 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926f50 0x55c9cc9310e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f1c88009bd0 tx=0x7f1c8800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0692374cbc2c1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c977fe640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1c800a2d30 0x7f1c800a51f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1c90004af0 tx=0x7f1c900095c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c7dffb640 1 -- 172.21.15.27:0/2685367259 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c88016070 con 0x55c9cc926f50 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c7dffb640 1 -- 172.21.15.27:0/2685367259 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c88005500 con 0x55c9cc926f50 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.580+0000 7f1c7dffb640 1 -- 172.21.15.27:0/2685367259 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c880116d0 con 0x55c9cc926f50 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 -- 172.21.15.27:0/2685367259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926f50 msgr2=0x55c9cc9310e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926f50 0x55c9cc9310e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f1c88009bd0 tx=0x7f1c8800e6c0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 -- 172.21.15.27:0/2685367259 shutdown_connections 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc933a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926f50 0x55c9cc9310e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/2685367259 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926630 0x55c9cc926a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 -- 172.21.15.27:0/2685367259 >> 172.21.15.27:0/2685367259 conn(0x55c9cc922950 msgr2=0x55c9cc922d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 -- 172.21.15.27:0/2685367259 shutdown_connections 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 -- 172.21.15.27:0/2685367259 wait complete. 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.581+0000 7f1ca05d9740 1 -- start start 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1ca05d9740 1 Processor -- start 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc92e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc92ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc92f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc934410 con 0x55c9cc926630 2024-04-25T01:27:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc846f50 con 0x55c9cc926f50 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc934590 con 0x55c9cc931620 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1ca05d9740 10 client.0 _ll_register_callbacks cb 0x55c9cc882000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc92e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc92e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36486/0 (socket says 172.21.15.27:36486) 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 -- 172.21.15.27:0/843177289 learned_addr learned my addr 172.21.15.27:0/843177289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c967fc640 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc92ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c977fe640 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc92f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 -- 172.21.15.27:0/843177289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 msgr2=0x55c9cc92f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc92f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 -- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 msgr2=0x55c9cc92ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc92ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 -- 172.21.15.27:0/843177289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9cc59f3f0 con 0x55c9cc926630 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c967fc640 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc92ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc92e590 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f1c880048f0 tx=0x7f1c88004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748aca4335a0c1cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.582+0000 7f1c76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1c76ffd640 1 -- 172.21.15.27:0/843177289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c88016040 con 0x55c9cc926630 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1c76ffd640 1 -- 172.21.15.27:0/843177289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f1c880050f0 con 0x55c9cc926630 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 1 -- 172.21.15.27:0/843177289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c88005500 con 0x55c9cc926630 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1c76ffd640 1 -- 172.21.15.27:0/843177289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1c880097a0 con 0x55c9cc926630 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 10 client.16083 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 1 -- 172.21.15.27:0/843177289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9cc55f0f0 con 0x55c9cc926630 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1c76ffd640 1 -- 172.21.15.27:0/843177289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c88005500 con 0x55c9cc926630 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1c76ffd640 1 -- 172.21.15.27:0/843177289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f1c8800f040 con 0x55c9cc926630 2024-04-25T01:27:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 10 client.16083 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 20 client.16083 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c9cc941830 0x55c9cc943c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 4 client.16083 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.583+0000 7f1ca05d9740 1 -- 172.21.15.27:0/843177289 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9cc898bf0 con 0x55c9cc941830 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.584+0000 7f1c967fc640 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c9cc941830 0x55c9cc943c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.584+0000 7f1c967fc640 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c9cc941830 0x55c9cc943c20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4da5ffed63f3d966 server_cookie=dfd5e90caa1a7ad9 in_seq=0 out_seq=0 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.584+0000 7f1c76ffd640 10 client.16083 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1c76ffd640 1 -- 172.21.15.27:0/843177289 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9cc898bf0 con 0x55c9cc941830 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1c76ffd640 10 client.16083 handle_command_reply: tid=0 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 client.16083 shutdown 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 -- 172.21.15.27:0/843177289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 msgr2=0x55c9cc92e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc92e590 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f1c880048f0 tx=0x7f1c88004920 comp rx=0 tx=0).stop 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 -- 172.21.15.27:0/843177289 shutdown_connections 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c9cc941830 0x55c9cc943c20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc92f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc92ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1c76ffd640 0 client.16083 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/843177289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc92e590 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 -- 172.21.15.27:0/843177289 >> 172.21.15.27:0/843177289 conn(0x55c9cc922950 msgr2=0x55c9cc928750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 -- 172.21.15.27:0/843177289 shutdown_connections 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 1 -- 172.21.15.27:0/843177289 wait complete. 2024-04-25T01:27:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.585+0000 7f1ca05d9740 20 client.16083 trim_cache size 0 max 16384 2024-04-25T01:27:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1ca05d9740 1 Processor -- start 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1ca05d9740 1 -- start start 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc93d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc949430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc949970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1c981b3680 con 0x55c9cc926630 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1c981b0f50 con 0x55c9cc926f50 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c981aeda0 con 0x55c9cc931620 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1c96ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc93d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1c96ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc93d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36500/0 (socket says 172.21.15.27:36500) 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1c96ffd640 1 -- 172.21.15.27:0/3936643310 learned_addr learned my addr 172.21.15.27:0/3936643310 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1c967fc640 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc949430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.603+0000 7f1c977fe640 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc949970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c96ffd640 1 -- 172.21.15.27:0/3936643310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 msgr2=0x55c9cc949970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc949970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c96ffd640 1 -- 172.21.15.27:0/3936643310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 msgr2=0x55c9cc949430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc949430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c96ffd640 1 -- 172.21.15.27:0/3936643310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9cc926d80 con 0x55c9cc926630 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc93d2d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f1c88009bd0 tx=0x7f1c88005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd63c9becf52d8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c7d7fa640 1 -- 172.21.15.27:0/3936643310 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c88016040 con 0x55c9cc926630 2024-04-25T01:27:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c7d7fa640 1 -- 172.21.15.27:0/3936643310 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c88023710 con 0x55c9cc926630 2024-04-25T01:27:10.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.604+0000 7f1c7d7fa640 1 -- 172.21.15.27:0/3936643310 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c880050f0 con 0x55c9cc926630 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 -- 172.21.15.27:0/3936643310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 msgr2=0x55c9cc93d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc93d2d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f1c88009bd0 tx=0x7f1c88005760 comp rx=0 tx=0).stop 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 -- 172.21.15.27:0/3936643310 shutdown_connections 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc931620 0x55c9cc949970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc949430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/3936643310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc926630 0x55c9cc93d2d0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 -- 172.21.15.27:0/3936643310 >> 172.21.15.27:0/3936643310 conn(0x55c9cc922950 msgr2=0x55c9cc922d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 -- 172.21.15.27:0/3936643310 shutdown_connections 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 -- 172.21.15.27:0/3936643310 wait complete. 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.605+0000 7f1ca05d9740 1 -- start start 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1ca05d9740 1 Processor -- start 2024-04-25T01:27:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc939a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc931620 0x55c9cc934d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1ca05d9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc935280 0x55c9cc935700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1c981af410 con 0x55c9cc931620 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1c981b0880 con 0x55c9cc926f50 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1ca05d9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c981afae0 con 0x55c9cc935280 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1ca05d9740 10 client.0 _ll_register_callbacks cb 0x7f1c88009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc931620 0x55c9cc934d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc931620 0x55c9cc934d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36514/0 (socket says 172.21.15.27:36514) 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 -- 172.21.15.27:0/1282156797 learned_addr learned my addr 172.21.15.27:0/1282156797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc939a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c977fe640 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc935280 0x55c9cc935700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 -- 172.21.15.27:0/1282156797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc935280 msgr2=0x55c9cc935700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc935280 0x55c9cc935700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 -- 172.21.15.27:0/1282156797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 msgr2=0x55c9cc939a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc939a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 -- 172.21.15.27:0/1282156797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9cc9a6ca0 con 0x55c9cc931620 2024-04-25T01:27:10.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c967fc640 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc931620 0x55c9cc934d40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c09b340 tx=0x7f1c8c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e626b46255876b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.606+0000 7f1c76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:10.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1c76ffd640 1 -- 172.21.15.27:0/1282156797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c020040 con 0x55c9cc931620 2024-04-25T01:27:10.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1c76ffd640 1 -- 172.21.15.27:0/1282156797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f1c8c0ccaf0 con 0x55c9cc931620 2024-04-25T01:27:10.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 1 -- 172.21.15.27:0/1282156797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c88023710 con 0x55c9cc931620 2024-04-25T01:27:10.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1c76ffd640 1 -- 172.21.15.27:0/1282156797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1c8c0a42b0 con 0x55c9cc931620 2024-04-25T01:27:10.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 10 client.16089 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:10.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 1 -- 172.21.15.27:0/1282156797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9cc9a7fc0 con 0x55c9cc931620 2024-04-25T01:27:10.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1c76ffd640 1 -- 172.21.15.27:0/1282156797 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c88023710 con 0x55c9cc931620 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1c76ffd640 1 -- 172.21.15.27:0/1282156797 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f1c8c09aa90 con 0x55c9cc931620 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 10 client.16089 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 10 client.16089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 20 client.16089 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c9cc9a8770 0x55c9cc9aab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 4 client.16089 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.610+0000 7f1ca05d9740 1 -- 172.21.15.27:0/1282156797 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c9cc892f40 con 0x55c9cc9a8770 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.611+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c9cc9a8770 0x55c9cc9aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.611+0000 7f1c76ffd640 10 client.16089 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.611+0000 7f1c96ffd640 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c9cc9a8770 0x55c9cc9aab90 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bee3ee93d11735d server_cookie=39ea49252128140c in_seq=0 out_seq=0 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.611+0000 7f1c76ffd640 1 -- 172.21.15.27:0/1282156797 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c9cc892f40 con 0x55c9cc9a8770 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.611+0000 7f1c76ffd640 10 client.16089 handle_command_reply: tid=0 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.611+0000 7f1ca05d9740 1 client.16089 shutdown 2024-04-25T01:27:10.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 -- 172.21.15.27:0/1282156797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc931620 msgr2=0x55c9cc934d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc931620 0x55c9cc934d40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c09b340 tx=0x7f1c8c010d70 comp rx=0 tx=0).stop 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 -- 172.21.15.27:0/1282156797 shutdown_connections 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c9cc9a8770 0x55c9cc9aab90 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9cc935280 0x55c9cc935700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9cc931620 0x55c9cc934d40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c76ffd640 0 client.16089 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 --2- 172.21.15.27:0/1282156797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c9cc926f50 0x55c9cc939a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 -- 172.21.15.27:0/1282156797 >> 172.21.15.27:0/1282156797 conn(0x55c9cc922950 msgr2=0x55c9cc921260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 -- 172.21.15.27:0/1282156797 shutdown_connections 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 1 -- 172.21.15.27:0/1282156797 wait complete. 2024-04-25T01:27:10.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1ca05d9740 20 client.16089 trim_cache size 0 max 16384 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout: "b3fbeceb-69b5-43a1-8e76-8e68ecc37247": { 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "b3fbeceb-69b5-43a1-8e76-8e68ecc37247", 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:27:10.595 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:27:10.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1c800a2d30 msgr2=0x7f1c800a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1c800a2d30 0x7f1c800a51f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1c90004af0 tx=0x7f1c900095c0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 msgr2=0x7f1c98100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 0x7f1c98100bb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00d950 tx=0x7f1c8c00de10 comp rx=0 tx=0).stop 2024-04-25T01:27:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/551324023 shutdown_connections 2024-04-25T01:27:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1c98107730 0x7f1c981010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1c800a2d30 0x7f1c800a51f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c9806a780 0x7f1c98100bb0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 --2- 172.21.15.27:0/551324023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c98069dc0 0x7f1c98100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.612+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/551324023 >> 172.21.15.27:0/551324023 conn(0x7f1c980776a0 msgr2=0x7f1c980fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.613+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/551324023 shutdown_connections 2024-04-25T01:27:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:10.613+0000 7f1c9d8cd640 1 -- 172.21.15.27:0/551324023 wait complete. 2024-04-25T01:27:10.602 INFO:tasks.cephfs.filesystem:scrub status for tag:b3fbeceb-69b5-43a1-8e76-8e68ecc37247 - {'path': '/', 'tag': 'b3fbeceb-69b5-43a1-8e76-8e68ecc37247', 'options': 'recursive,force'} 2024-04-25T01:27:10.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:10 smithi027 ceph-mon[29197]: pgmap v55: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:27:10.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:10 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:27:10.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:10 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:10.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:10 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:27:11.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:10 smithi153 ceph-mon[32621]: pgmap v55: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:27:11.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:10 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:27:11.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:10 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:11.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:10 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:27:11.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:10 smithi139 ceph-mon[32760]: pgmap v55: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:27:11.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:10 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:27:11.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:10 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:11.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:10 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:27:11.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:11 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:27:11.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:11 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:11.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:11 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:12.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:11 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:27:12.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:11 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:12.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:11 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:12.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:11 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:27:12.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:11 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:12.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:11 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:12.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:12 smithi027 ceph-mon[29197]: pgmap v56: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:27:13.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:12 smithi153 ceph-mon[32621]: pgmap v56: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:27:13.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:12 smithi139 ceph-mon[32760]: pgmap v56: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:27:14.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:14 smithi027 ceph-mon[29197]: pgmap v57: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 22 op/s 2024-04-25T01:27:15.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:14 smithi153 ceph-mon[32621]: pgmap v57: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 22 op/s 2024-04-25T01:27:15.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:14 smithi139 ceph-mon[32760]: pgmap v57: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 22 op/s 2024-04-25T01:27:16.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:16 smithi027 ceph-mon[29197]: pgmap v58: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 22 op/s 2024-04-25T01:27:17.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:16 smithi153 ceph-mon[32621]: pgmap v58: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 22 op/s 2024-04-25T01:27:17.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:16 smithi139 ceph-mon[32760]: pgmap v58: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 22 op/s 2024-04-25T01:27:17.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:17 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:27:18.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:17 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:27:18.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:17 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:27:18.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:18 smithi027 ceph-mon[29197]: pgmap v59: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 26 op/s 2024-04-25T01:27:18.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:18 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:27:19.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:18 smithi153 ceph-mon[32621]: pgmap v59: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 26 op/s 2024-04-25T01:27:19.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:18 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:27:19.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:18 smithi139 ceph-mon[32760]: pgmap v59: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 26 op/s 2024-04-25T01:27:19.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:18 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:27:20.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:20 smithi027 ceph-mon[29197]: pgmap v60: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:27:21.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:20 smithi153 ceph-mon[32621]: pgmap v60: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:27:21.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:20 smithi139 ceph-mon[32760]: pgmap v60: 129 pgs: 129 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:27:22.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:22 smithi027 ceph-mon[29197]: pgmap v61: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:23.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:22 smithi153 ceph-mon[32621]: pgmap v61: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:23.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:22 smithi139 ceph-mon[32760]: pgmap v61: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:24.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:24 smithi027 ceph-mon[29197]: pgmap v62: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:25.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:24 smithi153 ceph-mon[32621]: pgmap v62: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:25.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:24 smithi139 ceph-mon[32760]: pgmap v62: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:26.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:26 smithi027 ceph-mon[29197]: pgmap v63: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:27.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:26 smithi153 ceph-mon[32621]: pgmap v63: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:27.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:26 smithi139 ceph-mon[32760]: pgmap v63: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:28.946 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:28 smithi027 ceph-mon[29197]: pgmap v64: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 14 op/s 2024-04-25T01:27:29.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:28 smithi153 ceph-mon[32621]: pgmap v64: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 14 op/s 2024-04-25T01:27:29.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:28 smithi139 ceph-mon[32760]: pgmap v64: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 14 op/s 2024-04-25T01:27:30.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:30 smithi027 ceph-mon[29197]: pgmap v65: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:31.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:30 smithi153 ceph-mon[32621]: pgmap v65: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:31.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:30 smithi139 ceph-mon[32760]: pgmap v65: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:27:31.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:31 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:32.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:31 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:32.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:31 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:32.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:32 smithi027 ceph-mon[29197]: pgmap v66: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:27:32.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:32 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:32.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:32 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:32.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:32 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:27:33.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:32 smithi153 ceph-mon[32621]: pgmap v66: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:27:33.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:32 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:33.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:32 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:33.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:32 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:27:33.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:32 smithi139 ceph-mon[32760]: pgmap v66: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:27:33.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:32 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:33.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:32 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:33.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:32 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:27:34.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:34 smithi027 ceph-mon[29197]: pgmap v67: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 9 op/s 2024-04-25T01:27:35.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:34 smithi153 ceph-mon[32621]: pgmap v67: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 9 op/s 2024-04-25T01:27:35.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:34 smithi139 ceph-mon[32760]: pgmap v67: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 9 op/s 2024-04-25T01:27:36.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:36 smithi027 ceph-mon[29197]: pgmap v68: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 9 op/s 2024-04-25T01:27:37.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:36 smithi153 ceph-mon[32621]: pgmap v68: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 9 op/s 2024-04-25T01:27:37.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:36 smithi139 ceph-mon[32760]: pgmap v68: 129 pgs: 129 active+clean; 1.6 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 9 op/s 2024-04-25T01:27:37.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:37 smithi027 ceph-mon[29197]: pgmap v69: 129 pgs: 129 active+clean; 1.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 KiB/s wr, 14 op/s 2024-04-25T01:27:38.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:37 smithi153 ceph-mon[32621]: pgmap v69: 129 pgs: 129 active+clean; 1.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 KiB/s wr, 14 op/s 2024-04-25T01:27:38.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:37 smithi139 ceph-mon[32760]: pgmap v69: 129 pgs: 129 active+clean; 1.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 KiB/s wr, 14 op/s 2024-04-25T01:27:40.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:40 smithi027 ceph-mon[29197]: pgmap v70: 129 pgs: 129 active+clean; 1.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:40.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:40 smithi153 ceph-mon[32621]: pgmap v70: 129 pgs: 129 active+clean; 1.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:40.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:40 smithi139 ceph-mon[32760]: pgmap v70: 129 pgs: 129 active+clean; 1.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:40.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:27:40.603 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:27:40.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 -- 172.21.15.27:0/654233788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17381002e0 msgr2=0x7f17381006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 --2- 172.21.15.27:0/654233788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17381002e0 0x7f17381006c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1728009900 tx=0x7f172802a410 comp rx=0 tx=0).stop 2024-04-25T01:27:40.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 -- 172.21.15.27:0/654233788 shutdown_connections 2024-04-25T01:27:40.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 --2- 172.21.15.27:0/654233788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1738106aa0 0x7f173810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 --2- 172.21.15.27:0/654233788 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1738100c90 0x7f17381063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 --2- 172.21.15.27:0/654233788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17381002e0 0x7f17381006c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 -- 172.21.15.27:0/654233788 >> 172.21.15.27:0/654233788 conn(0x7f17380758d0 msgr2=0x7f1738075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 -- 172.21.15.27:0/654233788 shutdown_connections 2024-04-25T01:27:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.741+0000 7f173c909640 1 -- 172.21.15.27:0/654233788 wait complete. 2024-04-25T01:27:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f173c909640 1 Processor -- start 2024-04-25T01:27:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f173c909640 1 -- start start 2024-04-25T01:27:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f173c909640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17381002e0 0x7f173819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f173c909640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1738106aa0 0x7f173819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f173c909640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f173819eae0 0x7f173819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f173c909640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1738111b90 con 0x7f1738106aa0 2024-04-25T01:27:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f173c909640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1738111a10 con 0x7f173819eae0 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f173c909640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1738111d10 con 0x7f17381002e0 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.742+0000 7f1735d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1738106aa0 0x7f173819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1735d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1738106aa0 0x7f173819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54322/0 (socket says 172.21.15.27:54322) 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1735d9b640 1 -- 172.21.15.27:0/2994860192 learned_addr learned my addr 172.21.15.27:0/2994860192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f173659c640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17381002e0 0x7f173819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1736d9d640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f173819eae0 0x7f173819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1735d9b640 1 -- 172.21.15.27:0/2994860192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17381002e0 msgr2=0x7f173819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1735d9b640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17381002e0 0x7f173819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1735d9b640 1 -- 172.21.15.27:0/2994860192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f173819eae0 msgr2=0x7f173819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1735d9b640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f173819eae0 0x7f173819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1735d9b640 1 -- 172.21.15.27:0/2994860192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173819f5d0 con 0x7f1738106aa0 2024-04-25T01:27:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1736d9d640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f173819eae0 0x7f173819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f1735d9b640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1738106aa0 0x7f173819a760 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c970 tx=0x7f172c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f17277fe640 1 -- 172.21.15.27:0/2994860192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c014070 con 0x7f1738106aa0 2024-04-25T01:27:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f173c909640 1 -- 172.21.15.27:0/2994860192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173819f8c0 con 0x7f1738106aa0 2024-04-25T01:27:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f17277fe640 1 -- 172.21.15.27:0/2994860192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f172c004440 con 0x7f1738106aa0 2024-04-25T01:27:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f17277fe640 1 -- 172.21.15.27:0/2994860192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c00f450 con 0x7f1738106aa0 2024-04-25T01:27:40.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.743+0000 7f173c909640 1 -- 172.21.15.27:0/2994860192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17381a7100 con 0x7f1738106aa0 2024-04-25T01:27:40.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.746+0000 7f17277fe640 1 -- 172.21.15.27:0/2994860192 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f172c017070 con 0x7f1738106aa0 2024-04-25T01:27:40.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.746+0000 7f173f60c740 1 Processor -- start 2024-04-25T01:27:40.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.746+0000 7f173f60c740 1 -- start start 2024-04-25T01:27:40.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173f60c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 0x5638a6050760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173f60c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a60638f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173f60c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 0x5638a6066220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173f60c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5638a5fb56b0 con 0x5638a6050d30 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173f60c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5638a5f5d880 con 0x5638a6050380 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173f60c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5638a5fb1670 con 0x5638a6063e30 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f17277fe640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f17200a2d00 0x7f17200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a60638f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a60638f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54336/0 (socket says 172.21.15.27:54336) 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173659c640 1 -- 172.21.15.27:0/3762962649 learned_addr learned my addr 172.21.15.27:0/3762962649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f1735d9b640 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 0x5638a6066220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f1736d9d640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f17200a2d00 0x7f17200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f1736d9d640 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 0x5638a6050760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f17277fe640 1 -- 172.21.15.27:0/2994860192 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f172c0cf680 con 0x7f1738106aa0 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173659c640 1 -- 172.21.15.27:0/3762962649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 msgr2=0x5638a6066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173659c640 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 0x5638a6066220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173659c640 1 -- 172.21.15.27:0/3762962649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 msgr2=0x5638a6050760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173659c640 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 0x5638a6050760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f173659c640 1 -- 172.21.15.27:0/3762962649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638a5d937d0 con 0x5638a6050d30 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f1735d9b640 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 0x5638a6066220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.747+0000 7f1736d9d640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f17200a2d00 0x7f17200a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1730009fd0 tx=0x7f1730009490 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.748+0000 7f173659c640 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a60638f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1728009900 tx=0x7f1728031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535a4acc4c61a0b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.748+0000 7f1724ff9640 1 -- 172.21.15.27:0/3762962649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728040070 con 0x5638a6050d30 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.748+0000 7f1724ff9640 1 -- 172.21.15.27:0/3762962649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1728005020 con 0x5638a6050d30 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.748+0000 7f1724ff9640 1 -- 172.21.15.27:0/3762962649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728035040 con 0x5638a6050d30 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.748+0000 7f173f60c740 1 -- 172.21.15.27:0/3762962649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 msgr2=0x5638a60638f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.748+0000 7f173f60c740 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a60638f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1728009900 tx=0x7f1728031630 comp rx=0 tx=0).stop 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.748+0000 7f173f60c740 1 -- 172.21.15.27:0/3762962649 shutdown_connections 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 0x5638a6066220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a60638f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 --2- 172.21.15.27:0/3762962649 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 0x5638a6050760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 -- 172.21.15.27:0/3762962649 >> 172.21.15.27:0/3762962649 conn(0x5638a604c950 msgr2=0x5638a604cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 -- 172.21.15.27:0/3762962649 shutdown_connections 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 -- 172.21.15.27:0/3762962649 wait complete. 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 -- start start 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 Processor -- start 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050380 0x5638a6060e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050d30 0x5638a6061340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6063e30 0x5638a6061880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5638a6066b90 con 0x5638a6050380 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5638a5f70f50 con 0x5638a6063e30 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5638a6066d10 con 0x5638a6050d30 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173f60c740 10 client.0 _ll_register_callbacks cb 0x5638a5fac000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050380 0x5638a6060e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050380 0x5638a6060e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54352/0 (socket says 172.21.15.27:54352) 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.749+0000 7f173659c640 1 -- 172.21.15.27:0/3829176900 learned_addr learned my addr 172.21.15.27:0/3829176900 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f1735d9b640 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050d30 0x5638a6061340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f1736d9d640 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6063e30 0x5638a6061880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173659c640 1 -- 172.21.15.27:0/3829176900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050d30 msgr2=0x5638a6061340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173659c640 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050d30 0x5638a6061340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173659c640 1 -- 172.21.15.27:0/3829176900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6063e30 msgr2=0x5638a6061880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173659c640 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6063e30 0x5638a6061880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173659c640 1 -- 172.21.15.27:0/3829176900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638a5cc93f0 con 0x5638a6050380 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173659c640 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050380 0x5638a6060e00 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1728000c00 tx=0x7f1728005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad4b1ddd883d799 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f171dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f171dffb640 1 -- 172.21.15.27:0/3829176900 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728040070 con 0x5638a6050380 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f171dffb640 1 -- 172.21.15.27:0/3829176900 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f1728004a90 con 0x5638a6050380 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173f60c740 1 -- 172.21.15.27:0/3829176900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1728005020 con 0x5638a6050380 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f171dffb640 1 -- 172.21.15.27:0/3829176900 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1728056050 con 0x5638a6050380 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173f60c740 10 client.16101 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f173f60c740 1 -- 172.21.15.27:0/3829176900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5638a5c890f0 con 0x5638a6050380 2024-04-25T01:27:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.750+0000 7f171dffb640 1 -- 172.21.15.27:0/3829176900 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1728005020 con 0x5638a6050380 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f171dffb640 1 -- 172.21.15.27:0/3829176900 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f1728044400 con 0x5638a6050380 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f173f60c740 10 client.16101 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f173f60c740 10 client.16101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f173f60c740 20 client.16101 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f173f60c740 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5638a6073f00 0x5638a60762f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f173f60c740 4 client.16101 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f173f60c740 1 -- 172.21.15.27:0/3829176900 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638a5fc2bf0 con 0x5638a6073f00 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f1735d9b640 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5638a6073f00 0x5638a60762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f171dffb640 10 client.16101 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.751+0000 7f1735d9b640 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5638a6073f00 0x5638a60762f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a248e319354a23e1 server_cookie=e70631a1f333fdf0 in_seq=0 out_seq=0 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f171dffb640 1 -- 172.21.15.27:0/3829176900 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638a5fc2bf0 con 0x5638a6073f00 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f171dffb640 10 client.16101 handle_command_reply: tid=0 2024-04-25T01:27:40.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 client.16101 shutdown 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 -- 172.21.15.27:0/3829176900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050380 msgr2=0x5638a6060e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050380 0x5638a6060e00 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1728000c00 tx=0x7f1728005400 comp rx=0 tx=0).stop 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 -- 172.21.15.27:0/3829176900 shutdown_connections 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5638a6073f00 0x5638a60762f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6063e30 0x5638a6061880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050d30 0x5638a6061340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 --2- 172.21.15.27:0/3829176900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050380 0x5638a6060e00 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 -- 172.21.15.27:0/3829176900 >> 172.21.15.27:0/3829176900 conn(0x5638a604c950 msgr2=0x5638a60527c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f171dffb640 0 client.16101 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 -- 172.21.15.27:0/3829176900 shutdown_connections 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.752+0000 7f173f60c740 1 -- 172.21.15.27:0/3829176900 wait complete. 2024-04-25T01:27:40.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.753+0000 7f173f60c740 20 client.16101 trim_cache size 0 max 16384 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f173f60c740 1 Processor -- start 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f173f60c740 1 -- start start 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f173f60c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 0x5638a606fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f173f60c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a6052810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f173f60c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 0x5638a604b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f173f60c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17381af280 con 0x5638a6050d30 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f173f60c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f17381acb50 con 0x5638a6050380 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f173f60c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17381aa9a0 con 0x5638a6063e30 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f1735d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a6052810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f1735d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a6052810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54354/0 (socket says 172.21.15.27:54354) 2024-04-25T01:27:40.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.770+0000 7f1735d9b640 1 -- 172.21.15.27:0/2001881461 learned_addr learned my addr 172.21.15.27:0/2001881461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f173659c640 1 --2- 172.21.15.27:0/2001881461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 0x5638a606fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f1735d9b640 1 -- 172.21.15.27:0/2001881461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 msgr2=0x5638a604b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f1735d9b640 1 --2- 172.21.15.27:0/2001881461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 0x5638a604b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f1735d9b640 1 -- 172.21.15.27:0/2001881461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 msgr2=0x5638a606fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f1735d9b640 1 --2- 172.21.15.27:0/2001881461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 0x5638a606fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f1735d9b640 1 -- 172.21.15.27:0/2001881461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638a6066220 con 0x5638a6050d30 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f1735d9b640 1 --2- 172.21.15.27:0/2001881461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a6052810 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f172c098180 tx=0x7f172c06e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8ec3210f1f4d9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f171ffff640 1 -- 172.21.15.27:0/2001881461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c0a4420 con 0x5638a6050d30 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f171ffff640 1 -- 172.21.15.27:0/2001881461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f172c099850 con 0x5638a6050d30 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f171ffff640 1 -- 172.21.15.27:0/2001881461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c003730 con 0x5638a6050d30 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f173f60c740 1 -- 172.21.15.27:0/2001881461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 msgr2=0x5638a6052810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.771+0000 7f173f60c740 1 --2- 172.21.15.27:0/2001881461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a6052810 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f172c098180 tx=0x7f172c06e820 comp rx=0 tx=0).stop 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 -- 172.21.15.27:0/2001881461 shutdown_connections 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 --2- 172.21.15.27:0/2001881461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6063e30 0x5638a604b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 --2- 172.21.15.27:0/2001881461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6050d30 0x5638a6052810 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 --2- 172.21.15.27:0/2001881461 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050380 0x5638a606fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 -- 172.21.15.27:0/2001881461 >> 172.21.15.27:0/2001881461 conn(0x5638a604c950 msgr2=0x5638a604b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 -- 172.21.15.27:0/2001881461 shutdown_connections 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 -- 172.21.15.27:0/2001881461 wait complete. 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 -- start start 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 Processor -- start 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050380 0x5638a6067150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050d30 0x5638a6067690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6063e30 0x5638a60d81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17381ab010 con 0x5638a6063e30 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f17381ac480 con 0x5638a6050d30 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17381ab6e0 con 0x5638a6050380 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173f60c740 10 client.0 _ll_register_callbacks cb 0x7f1728009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:40.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f1736d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6063e30 0x5638a60d81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f1736d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6063e30 0x5638a60d81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54370/0 (socket says 172.21.15.27:54370) 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f1736d9d640 1 -- 172.21.15.27:0/2246120997 learned_addr learned my addr 172.21.15.27:0/2246120997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.772+0000 7f173659c640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050380 0x5638a6067150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f1736d9d640 1 -- 172.21.15.27:0/2246120997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050380 msgr2=0x5638a6067150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f1736d9d640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050380 0x5638a6067150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f1735d9b640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050d30 0x5638a6067690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f1736d9d640 1 -- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050d30 msgr2=0x5638a6067690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f1736d9d640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050d30 0x5638a6067690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f1736d9d640 1 -- 172.21.15.27:0/2246120997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638a60d8750 con 0x5638a6063e30 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f173659c640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050380 0x5638a6067150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f1735d9b640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050d30 0x5638a6067690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f1736d9d640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6063e30 0x5638a60d81e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1730009b20 tx=0x7f1730009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e131918775f3845 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f171dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f171dffb640 1 -- 172.21.15.27:0/2246120997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730024070 con 0x5638a6063e30 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f171dffb640 1 -- 172.21.15.27:0/2246120997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f17300209e0 con 0x5638a6063e30 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f173f60c740 1 -- 172.21.15.27:0/2246120997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f172c099850 con 0x5638a6063e30 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f171dffb640 1 -- 172.21.15.27:0/2246120997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1730020be0 con 0x5638a6063e30 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f173f60c740 10 client.16107 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f173f60c740 1 -- 172.21.15.27:0/2246120997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5638a60d9ac0 con 0x5638a6063e30 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f171dffb640 1 -- 172.21.15.27:0/2246120997 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f172c099850 con 0x5638a6063e30 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f171dffb640 1 -- 172.21.15.27:0/2246120997 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f173001d040 con 0x5638a6063e30 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f173f60c740 10 client.16107 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.773+0000 7f173f60c740 10 client.16107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f173f60c740 20 client.16107 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f173f60c740 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5638a60da2a0 0x5638a60dc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f173f60c740 4 client.16107 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f173f60c740 1 -- 172.21.15.27:0/2246120997 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638a5fbcf40 con 0x5638a60da2a0 2024-04-25T01:27:40.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f173659c640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5638a60da2a0 0x5638a60dc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f173659c640 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5638a60da2a0 0x5638a60dc6c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcf7980b5262a4d9 server_cookie=f8c8797e002269bf in_seq=0 out_seq=0 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f171dffb640 10 client.16107 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f171dffb640 1 -- 172.21.15.27:0/2246120997 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5638a5fbcf40 con 0x5638a60da2a0 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f171dffb640 10 client.16107 handle_command_reply: tid=0 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.774+0000 7f173f60c740 1 client.16107 shutdown 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 -- 172.21.15.27:0/2246120997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6063e30 msgr2=0x5638a60d81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6063e30 0x5638a60d81e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1730009b20 tx=0x7f1730009cc0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 -- 172.21.15.27:0/2246120997 shutdown_connections 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5638a60da2a0 0x5638a60dc6c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5638a6063e30 0x5638a60d81e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5638a6050d30 0x5638a6067690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f171dffb640 0 client.16107 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 --2- 172.21.15.27:0/2246120997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5638a6050380 0x5638a6067150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 -- 172.21.15.27:0/2246120997 >> 172.21.15.27:0/2246120997 conn(0x5638a604c950 msgr2=0x5638a6061480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 -- 172.21.15.27:0/2246120997 shutdown_connections 2024-04-25T01:27:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 1 -- 172.21.15.27:0/2246120997 wait complete. 2024-04-25T01:27:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173f60c740 20 client.16107 trim_cache size 0 max 16384 2024-04-25T01:27:40.760 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:27:40.760 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:27:40.760 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:27:40.760 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:27:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 -- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f17200a2d00 msgr2=0x7f17200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f17200a2d00 0x7f17200a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1730009fd0 tx=0x7f1730009490 comp rx=0 tx=0).stop 2024-04-25T01:27:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 -- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1738106aa0 msgr2=0x7f173819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1738106aa0 0x7f173819a760 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c970 tx=0x7f172c00ce30 comp rx=0 tx=0).stop 2024-04-25T01:27:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 -- 172.21.15.27:0/2994860192 shutdown_connections 2024-04-25T01:27:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f173819eae0 0x7f173819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f17200a2d00 0x7f17200a51c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1738106aa0 0x7f173819a760 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 --2- 172.21.15.27:0/2994860192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17381002e0 0x7f173819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 -- 172.21.15.27:0/2994860192 >> 172.21.15.27:0/2994860192 conn(0x7f17380758d0 msgr2=0x7f17380ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:40.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 -- 172.21.15.27:0/2994860192 shutdown_connections 2024-04-25T01:27:40.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.775+0000 7f173c909640 1 -- 172.21.15.27:0/2994860192 wait complete. 2024-04-25T01:27:40.765 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:27:40.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:27:40.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 -- 172.21.15.27:0/1092868080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0101710 msgr2=0x7f7bd0101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 --2- 172.21.15.27:0/1092868080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0101710 0x7f7bd0101b90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009930 tx=0x7f7bcc02a420 comp rx=0 tx=0).stop 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 -- 172.21.15.27:0/1092868080 shutdown_connections 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 --2- 172.21.15.27:0/1092868080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd01020d0 0x7f7bd010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 --2- 172.21.15.27:0/1092868080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0101710 0x7f7bd0101b90 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 --2- 172.21.15.27:0/1092868080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bd0107720 0x7f7bd0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 -- 172.21.15.27:0/1092868080 >> 172.21.15.27:0/1092868080 conn(0x7f7bd0078b00 msgr2=0x7f7bd0078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 -- 172.21.15.27:0/1092868080 shutdown_connections 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 -- 172.21.15.27:0/1092868080 wait complete. 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 Processor -- start 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.889+0000 7f7bd7893640 1 -- start start 2024-04-25T01:27:40.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd7893640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bd0101710 0x7f7bd0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd7893640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd01020d0 0x7f7bd0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd7893640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd0107720 0x7f7bd019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd7893640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd0111b30 con 0x7f7bd01020d0 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd7893640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd01119b0 con 0x7f7bd0101710 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd7893640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd0111cb0 con 0x7f7bd0107720 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd01020d0 0x7f7bd0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd01020d0 0x7f7bd0198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54386/0 (socket says 172.21.15.27:54386) 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 -- 172.21.15.27:0/320005402 learned_addr learned my addr 172.21.15.27:0/320005402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd562f640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bd0101710 0x7f7bd0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd5e30640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd0107720 0x7f7bd019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 -- 172.21.15.27:0/320005402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd0107720 msgr2=0x7f7bd019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd0107720 0x7f7bd019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 -- 172.21.15.27:0/320005402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bd0101710 msgr2=0x7f7bd0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:40.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bd0101710 0x7f7bd0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 -- 172.21.15.27:0/320005402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd019cfb0 con 0x7f7bd01020d0 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd5e30640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd0107720 0x7f7bd019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd562f640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bd0101710 0x7f7bd0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd4e2e640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd01020d0 0x7f7bd0198550 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009230 tx=0x7f7bcc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bc27fc640 1 -- 172.21.15.27:0/320005402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc03f030 con 0x7f7bd01020d0 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bc27fc640 1 -- 172.21.15.27:0/320005402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bcc003ad0 con 0x7f7bd01020d0 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd019d240 con 0x7f7bd01020d0 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.890+0000 7f7bc27fc640 1 -- 172.21.15.27:0/320005402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc035070 con 0x7f7bd01020d0 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.891+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd01a4b30 con 0x7f7bd01020d0 2024-04-25T01:27:40.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.891+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd00689b0 con 0x7f7bd01020d0 2024-04-25T01:27:40.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.892+0000 7f7bc27fc640 1 -- 172.21.15.27:0/320005402 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7bcc042030 con 0x7f7bd01020d0 2024-04-25T01:27:40.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.893+0000 7f7bc27fc640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7bbc0a2d00 0x7f7bbc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:40.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.893+0000 7f7bd562f640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7bbc0a2d00 0x7f7bbc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:40.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.894+0000 7f7bc27fc640 1 -- 172.21.15.27:0/320005402 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7bcc0efd90 con 0x7f7bd01020d0 2024-04-25T01:27:40.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.894+0000 7f7bd562f640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7bbc0a2d00 0x7f7bbc0a51c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4005fd0 tx=0x7f7bc4005850 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:40.898+0000 7f7bc27fc640 1 -- 172.21.15.27:0/320005402 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7bcc0bc620 con 0x7f7bd01020d0 2024-04-25T01:27:41.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.123+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bd0109cb0 con 0x7f7bd01020d0 2024-04-25T01:27:41.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.124+0000 7f7bc27fc640 1 -- 172.21.15.27:0/320005402 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11883 (secure 0 0 0) 0x7f7bcc0bbf20 con 0x7f7bd01020d0 2024-04-25T01:27:41.103 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:27:41.103 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:26:47.433054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24517,24521,24505]},"id":1}]} 2024-04-25T01:27:41.103 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-04-25T01:27:41.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7bbc0a2d00 msgr2=0x7f7bbc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7bbc0a2d00 0x7f7bbc0a51c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4005fd0 tx=0x7f7bc4005850 comp rx=0 tx=0).stop 2024-04-25T01:27:41.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd01020d0 msgr2=0x7f7bd0198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd01020d0 0x7f7bd0198550 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009230 tx=0x7f7bcc004270 comp rx=0 tx=0).stop 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 shutdown_connections 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bd0107720 0x7f7bd019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7bbc0a2d00 0x7f7bbc0a51c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd01020d0 0x7f7bd0198550 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 --2- 172.21.15.27:0/320005402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bd0101710 0x7f7bd0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 >> 172.21.15.27:0/320005402 conn(0x7f7bd0078b00 msgr2=0x7f7bd0104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 shutdown_connections 2024-04-25T01:27:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.126+0000 7f7bd7893640 1 -- 172.21.15.27:0/320005402 wait complete. 2024-04-25T01:27:41.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:27:41.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 -- 172.21.15.27:0/796815494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48069dc0 msgr2=0x7f3f4806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/796815494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48069dc0 0x7f3f4806a240 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009930 tx=0x7f3f3802a420 comp rx=0 tx=0).stop 2024-04-25T01:27:41.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 -- 172.21.15.27:0/796815494 shutdown_connections 2024-04-25T01:27:41.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/796815494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f4806a780 0x7f3f4810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/796815494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48069dc0 0x7f3f4806a240 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/796815494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48107730 0x7f3f48107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 -- 172.21.15.27:0/796815494 >> 172.21.15.27:0/796815494 conn(0x7f3f480776a0 msgr2=0x7f3f48077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 -- 172.21.15.27:0/796815494 shutdown_connections 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.241+0000 7f3f4f48c640 1 -- 172.21.15.27:0/796815494 wait complete. 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4f48c640 1 Processor -- start 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4f48c640 1 -- start start 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4f48c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48069dc0 0x7f3f4819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4f48c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f4806a780 0x7f3f4819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4f48c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f48107730 0x7f3f4819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4f48c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48111bc0 con 0x7f3f4806a780 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4f48c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48111a40 con 0x7f3f48107730 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4f48c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48111d40 con 0x7f3f48069dc0 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f4806a780 0x7f3f4819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f4806a780 0x7f3f4819a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54414/0 (socket says 172.21.15.27:54414) 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 -- 172.21.15.27:0/635307172 learned_addr learned my addr 172.21.15.27:0/635307172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4da29640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f48107730 0x7f3f4819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4d228640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48069dc0 0x7f3f4819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 -- 172.21.15.27:0/635307172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48069dc0 msgr2=0x7f3f4819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48069dc0 0x7f3f4819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 -- 172.21.15.27:0/635307172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f48107730 msgr2=0x7f3f4819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f48107730 0x7f3f4819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 -- 172.21.15.27:0/635307172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4819f350 con 0x7f3f4806a780 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4d228640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48069dc0 0x7f3f4819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.242+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f4806a780 0x7f3f4819a8a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009a50 tx=0x7f3f38004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.243+0000 7f3f3e7fc640 1 -- 172.21.15.27:0/635307172 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3803f030 con 0x7f3f4806a780 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.243+0000 7f3f4f48c640 1 -- 172.21.15.27:0/635307172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4819f5e0 con 0x7f3f4806a780 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.243+0000 7f3f3e7fc640 1 -- 172.21.15.27:0/635307172 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f38003ce0 con 0x7f3f4806a780 2024-04-25T01:27:41.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.243+0000 7f3f3e7fc640 1 -- 172.21.15.27:0/635307172 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38035070 con 0x7f3f4806a780 2024-04-25T01:27:41.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.243+0000 7f3f4f48c640 1 -- 172.21.15.27:0/635307172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f481a6ed0 con 0x7f3f4806a780 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f3e7fc640 1 -- 172.21.15.27:0/635307172 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3f38042030 con 0x7f3f4806a780 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f52197740 1 Processor -- start 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f52197740 1 -- start start 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f52197740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812f580 0x560f1812f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f52197740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812fea0 0x560f1813a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f52197740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f1813c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f52197740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560f18094670 con 0x560f1813a570 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f52197740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560f1803c970 con 0x560f1812fea0 2024-04-25T01:27:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.245+0000 7f3f52197740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560f18090630 con 0x560f1812f580 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f1813c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f1813c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54424/0 (socket says 172.21.15.27:54424) 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 -- 172.21.15.27:0/3301865120 learned_addr learned my addr 172.21.15.27:0/3301865120 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4da29640 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812f580 0x560f1812f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4d228640 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812fea0 0x560f1813a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f3e7fc640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f340a2d00 0x7f3f340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 -- 172.21.15.27:0/3301865120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812f580 msgr2=0x560f1812f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812f580 0x560f1812f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 -- 172.21.15.27:0/3301865120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812fea0 msgr2=0x560f1813a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812fea0 0x560f1813a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 -- 172.21.15.27:0/3301865120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f18060400 con 0x560f1813a570 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4d228640 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812fea0 0x560f1813a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4da29640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f340a2d00 0x7f3f340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f3e7fc640 1 -- 172.21.15.27:0/635307172 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3f380efd60 con 0x7f3f4806a780 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f1813c960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3f380ed090 tx=0x7f3f380b8190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2870fef58322003f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f337fe640 1 -- 172.21.15.27:0/3301865120 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38049070 con 0x560f1813a570 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.246+0000 7f3f4da29640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f340a2d00 0x7f3f340a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000a4c0 tx=0x7f3f40004b50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f337fe640 1 -- 172.21.15.27:0/3301865120 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f380bc5f0 con 0x560f1813a570 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f337fe640 1 -- 172.21.15.27:0/3301865120 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f380bbcc0 con 0x560f1813a570 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 -- 172.21.15.27:0/3301865120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 msgr2=0x560f1813c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f1813c960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3f380ed090 tx=0x7f3f380b8190 comp rx=0 tx=0).stop 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 -- 172.21.15.27:0/3301865120 shutdown_connections 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f1813c960 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812fea0 0x560f1813a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 --2- 172.21.15.27:0/3301865120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812f580 0x560f1812f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 -- 172.21.15.27:0/3301865120 >> 172.21.15.27:0/3301865120 conn(0x560f1812b8a0 msgr2=0x560f1812bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 -- 172.21.15.27:0/3301865120 shutdown_connections 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.247+0000 7f3f52197740 1 -- 172.21.15.27:0/3301865120 wait complete. 2024-04-25T01:27:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 1 -- start start 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 1 Processor -- start 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18139f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f18137200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181377a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560f1813d120 con 0x560f1813a570 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560f18050050 con 0x560f1812f580 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560f1813d2a0 con 0x560f1812fea0 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f52197740 10 client.0 _ll_register_callbacks cb 0x560f17da7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4da29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4da29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181377a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54428/0 (socket says 172.21.15.27:54428) 2024-04-25T01:27:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4da29640 1 -- 172.21.15.27:0/475161530 learned_addr learned my addr 172.21.15.27:0/475161530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4d228640 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18139f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f18137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4da29640 1 -- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 msgr2=0x560f18137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4da29640 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f18137200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4da29640 1 -- 172.21.15.27:0/475161530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 msgr2=0x560f18139f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4da29640 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18139f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4da29640 1 -- 172.21.15.27:0/475161530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f18021dc0 con 0x560f1813a570 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.248+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f18137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f4da29640 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181377a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3f40016df0 tx=0x7f3f4000d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0227186ea0f55d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f30ff9640 1 -- 172.21.15.27:0/475161530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40027060 con 0x560f1813a570 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f30ff9640 1 -- 172.21.15.27:0/475161530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3f40022610 con 0x560f1813a570 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f52197740 1 -- 172.21.15.27:0/475161530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f380bc5f0 con 0x560f1813a570 2024-04-25T01:27:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f30ff9640 1 -- 172.21.15.27:0/475161530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3f4002e050 con 0x560f1813a570 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f52197740 10 client.16125 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f52197740 1 -- 172.21.15.27:0/475161530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560f17da83f0 con 0x560f1813a570 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f30ff9640 1 -- 172.21.15.27:0/475161530 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f380bc5f0 con 0x560f1813a570 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f30ff9640 1 -- 172.21.15.27:0/475161530 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f3f40021840 con 0x560f1813a570 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f52197740 10 client.16125 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f52197740 10 client.16125 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f52197740 20 client.16125 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f52197740 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560f1814a3a0 0x560f1814c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.249+0000 7f3f52197740 4 client.16125 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.250+0000 7f3f52197740 1 -- 172.21.15.27:0/475161530 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f180a1bb0 con 0x560f1814a3a0 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.250+0000 7f3f4d228640 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560f1814a3a0 0x560f1814c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.250+0000 7f3f4d228640 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560f1814a3a0 0x560f1814c7d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c69b84227d9535af server_cookie=e8484b39a367eb85 in_seq=0 out_seq=0 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.250+0000 7f3f30ff9640 10 client.16125 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f30ff9640 1 -- 172.21.15.27:0/475161530 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f180a1bb0 con 0x560f1814a3a0 2024-04-25T01:27:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f30ff9640 10 client.16125 handle_command_reply: tid=0 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 client.16125 shutdown 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 -- 172.21.15.27:0/475161530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 msgr2=0x560f181377a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181377a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3f40016df0 tx=0x7f3f4000d040 comp rx=0 tx=0).stop 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 -- 172.21.15.27:0/475161530 shutdown_connections 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560f1814a3a0 0x560f1814c7d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181377a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f18137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f30ff9640 0 client.16125 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 --2- 172.21.15.27:0/475161530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18139f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 -- 172.21.15.27:0/475161530 >> 172.21.15.27:0/475161530 conn(0x560f1812b8a0 msgr2=0x560f1812ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 -- 172.21.15.27:0/475161530 shutdown_connections 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 1 -- 172.21.15.27:0/475161530 wait complete. 2024-04-25T01:27:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.251+0000 7f3f52197740 20 client.16125 trim_cache size 0 max 16384 2024-04-25T01:27:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.264+0000 7f3f52197740 1 Processor -- start 2024-04-25T01:27:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.264+0000 7f3f52197740 1 -- start start 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.264+0000 7f3f52197740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f1812b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.264+0000 7f3f52197740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1812fea0 0x560f18139a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f52197740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1813a570 0x560f181424f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f52197740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481b3770 con 0x560f1812fea0 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f52197740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481b1040 con 0x560f1812f580 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f52197740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481aee90 con 0x560f1813a570 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1812fea0 0x560f18139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1812fea0 0x560f18139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54430/0 (socket says 172.21.15.27:54430) 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 -- 172.21.15.27:0/461128370 learned_addr learned my addr 172.21.15.27:0/461128370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4da29640 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1813a570 0x560f181424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4d228640 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f1812b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 -- 172.21.15.27:0/461128370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1813a570 msgr2=0x560f181424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1813a570 0x560f181424f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 -- 172.21.15.27:0/461128370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 msgr2=0x560f1812b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f1812b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 -- 172.21.15.27:0/461128370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f1813ccd0 con 0x560f1812fea0 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4d228640 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f1812b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1812fea0 0x560f18139a40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3f380ba1a0 tx=0x7f3f3803b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56086bae2d8acaca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f32ffd640 1 -- 172.21.15.27:0/461128370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38049040 con 0x560f1812fea0 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f32ffd640 1 -- 172.21.15.27:0/461128370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f380bc5f0 con 0x560f1812fea0 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.265+0000 7f3f32ffd640 1 -- 172.21.15.27:0/461128370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f380bb850 con 0x560f1812fea0 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 -- 172.21.15.27:0/461128370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1812fea0 msgr2=0x560f18139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1812fea0 0x560f18139a40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3f380ba1a0 tx=0x7f3f3803b800 comp rx=0 tx=0).stop 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 -- 172.21.15.27:0/461128370 shutdown_connections 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1813a570 0x560f181424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1812fea0 0x560f18139a40 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 --2- 172.21.15.27:0/461128370 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f1812b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 -- 172.21.15.27:0/461128370 >> 172.21.15.27:0/461128370 conn(0x560f1812b8a0 msgr2=0x560f18144fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 -- 172.21.15.27:0/461128370 shutdown_connections 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 -- 172.21.15.27:0/461128370 wait complete. 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 -- start start 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 Processor -- start 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18142190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.266+0000 7f3f52197740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f181426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f52197740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181407a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f52197740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481af500 con 0x560f1813a570 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f52197740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481b0970 con 0x560f1812f580 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f52197740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481afbd0 con 0x560f1812fea0 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f52197740 10 client.0 _ll_register_callbacks cb 0x7f3f40019040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181407a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54440/0 (socket says 172.21.15.27:54440) 2024-04-25T01:27:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 -- 172.21.15.27:0/1828158256 learned_addr learned my addr 172.21.15.27:0/1828158256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4ca27640 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f181426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 -- 172.21.15.27:0/1828158256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 msgr2=0x560f181426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f181426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4d228640 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 -- 172.21.15.27:0/1828158256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 msgr2=0x560f18142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18142190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 -- 172.21.15.27:0/1828158256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f17e727d0 con 0x560f1813a570 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4d228640 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18142190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f4da29640 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181407a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3f40000f80 tx=0x7f3f40021f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f5860a329347e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f30ff9640 1 -- 172.21.15.27:0/1828158256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40037020 con 0x560f1813a570 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f30ff9640 1 -- 172.21.15.27:0/1828158256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3f400217c0 con 0x560f1813a570 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f52197740 1 -- 172.21.15.27:0/1828158256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f380bc5f0 con 0x560f1813a570 2024-04-25T01:27:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f30ff9640 1 -- 172.21.15.27:0/1828158256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3f40027020 con 0x560f1813a570 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f52197740 10 client.16131 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.267+0000 7f3f52197740 1 -- 172.21.15.27:0/1828158256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560f181b1210 con 0x560f1813a570 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f30ff9640 1 -- 172.21.15.27:0/1828158256 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f380bc5f0 con 0x560f1813a570 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f30ff9640 1 -- 172.21.15.27:0/1828158256 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f3f40022410 con 0x560f1813a570 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f52197740 10 client.16131 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f52197740 10 client.16131 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f52197740 20 client.16131 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f52197740 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560f181b1980 0x560f181b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f52197740 4 client.16131 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f52197740 1 -- 172.21.15.27:0/1828158256 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f1809bf00 con 0x560f181b1980 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f4d228640 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560f181b1980 0x560f181b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f4d228640 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560f181b1980 0x560f181b3da0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9d09b03f5a2a6f2 server_cookie=1162a5dd81649e03 in_seq=0 out_seq=0 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.268+0000 7f3f30ff9640 10 client.16131 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:27:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f30ff9640 1 -- 172.21.15.27:0/1828158256 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f1809bf00 con 0x560f181b1980 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f30ff9640 10 client.16131 handle_command_reply: tid=0 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 client.16131 shutdown 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 -- 172.21.15.27:0/1828158256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 msgr2=0x560f181407a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181407a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3f40000f80 tx=0x7f3f40021f70 comp rx=0 tx=0).stop 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 -- 172.21.15.27:0/1828158256 shutdown_connections 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560f181b1980 0x560f181b3da0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f1813a570 0x560f181407a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560f1812fea0 0x560f181426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 --2- 172.21.15.27:0/1828158256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560f1812f580 0x560f18142190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f30ff9640 0 client.16131 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 -- 172.21.15.27:0/1828158256 >> 172.21.15.27:0/1828158256 conn(0x560f1812b8a0 msgr2=0x560f1812b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 -- 172.21.15.27:0/1828158256 shutdown_connections 2024-04-25T01:27:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 1 -- 172.21.15.27:0/1828158256 wait complete. 2024-04-25T01:27:41.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f52197740 20 client.16131 trim_cache size 0 max 16384 2024-04-25T01:27:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 -- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f340a2d00 msgr2=0x7f3f340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f340a2d00 0x7f3f340a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000a4c0 tx=0x7f3f40004b50 comp rx=0 tx=0).stop 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 -- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f4806a780 msgr2=0x7f3f4819a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f4806a780 0x7f3f4819a8a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009a50 tx=0x7f3f38004270 comp rx=0 tx=0).stop 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 -- 172.21.15.27:0/635307172 shutdown_connections 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f48107730 0x7f3f4819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f340a2d00 0x7f3f340a51c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f4806a780 0x7f3f4819a8a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 --2- 172.21.15.27:0/635307172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48069dc0 0x7f3f4819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.269+0000 7f3f4f48c640 1 -- 172.21.15.27:0/635307172 >> 172.21.15.27:0/635307172 conn(0x7f3f480776a0 msgr2=0x7f3f4810cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.270+0000 7f3f4f48c640 1 -- 172.21.15.27:0/635307172 shutdown_connections 2024-04-25T01:27:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.270+0000 7f3f4f48c640 1 -- 172.21.15.27:0/635307172 wait complete. 2024-04-25T01:27:41.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:27:41.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.384+0000 7f0d25b09640 1 -- 172.21.15.27:0/2557420824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d20101710 msgr2=0x7f0d20101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.384+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2557420824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d20101710 0x7f0d20101b90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009930 tx=0x7f0d1402a420 comp rx=0 tx=0).stop 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.384+0000 7f0d25b09640 1 -- 172.21.15.27:0/2557420824 shutdown_connections 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.384+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2557420824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d201020d0 0x7f0d2010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.384+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2557420824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d20101710 0x7f0d20101b90 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.384+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2557420824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d20107720 0x7f0d20107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.384+0000 7f0d25b09640 1 -- 172.21.15.27:0/2557420824 >> 172.21.15.27:0/2557420824 conn(0x7f0d200fd180 msgr2=0x7f0d200ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.384+0000 7f0d25b09640 1 -- 172.21.15.27:0/2557420824 shutdown_connections 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 -- 172.21.15.27:0/2557420824 wait complete. 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 Processor -- start 2024-04-25T01:27:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 -- start start 2024-04-25T01:27:41.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d20101710 0x7f0d201980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d201020d0 0x7f0d20198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d20107720 0x7f0d2019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20111b80 con 0x7f0d201020d0 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20111a00 con 0x7f0d20107720 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.385+0000 7f0d25b09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20111d00 con 0x7f0d20101710 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d201020d0 0x7f0d20198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d201020d0 0x7f0d20198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54450/0 (socket says 172.21.15.27:54450) 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 -- 172.21.15.27:0/2947909932 learned_addr learned my addr 172.21.15.27:0/2947909932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d20107720 0x7f0d2019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d20101710 0x7f0d201980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 -- 172.21.15.27:0/2947909932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d20101710 msgr2=0x7f0d201980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d20101710 0x7f0d201980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 -- 172.21.15.27:0/2947909932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d20107720 msgr2=0x7f0d2019c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d20107720 0x7f0d2019c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 -- 172.21.15.27:0/2947909932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2019d0b0 con 0x7f0d201020d0 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d20107720 0x7f0d2019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d20101710 0x7f0d201980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1effd640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d201020d0 0x7f0d20198600 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0d1402a940 tx=0x7f0d14003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1cff9640 1 -- 172.21.15.27:0/2947909932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1403f070 con 0x7f0d201020d0 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1cff9640 1 -- 172.21.15.27:0/2947909932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d14031c90 con 0x7f0d201020d0 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d1cff9640 1 -- 172.21.15.27:0/2947909932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14035030 con 0x7f0d201020d0 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d25b09640 1 -- 172.21.15.27:0/2947909932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2019d340 con 0x7f0d201020d0 2024-04-25T01:27:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.386+0000 7f0d25b09640 1 -- 172.21.15.27:0/2947909932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d201a4c30 con 0x7f0d201020d0 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.388+0000 7f0d1cff9640 1 -- 172.21.15.27:0/2947909932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0d14042070 con 0x7f0d201020d0 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d28810740 1 Processor -- start 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d28810740 1 -- start start 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d1cff9640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0d080a2d00 0x7f0d080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d28810740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd56960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d28810740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd61030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d28810740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd63960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d28810740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56417ccbb670 con 0x56417cd61570 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d28810740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56417cc63970 con 0x56417cd56ea0 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d28810740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56417ccb7630 con 0x56417cd56580 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0d080a2d00 0x7f0d080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d1cff9640 1 -- 172.21.15.27:0/2947909932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0d140effd0 con 0x7f0d201020d0 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd63960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd63960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54464/0 (socket says 172.21.15.27:54464) 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d1ffff640 1 -- 172.21.15.27:0/1628278136 learned_addr learned my addr 172.21.15.27:0/1628278136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd56960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.389+0000 7f0d1effd640 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd61030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d1ffff640 1 -- 172.21.15.27:0/1628278136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 msgr2=0x56417cd56960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd56960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d1ffff640 1 -- 172.21.15.27:0/1628278136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 msgr2=0x56417cd61030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd61030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d1ffff640 1 -- 172.21.15.27:0/1628278136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417cc87400 con 0x56417cd61570 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0d080a2d00 0x7f0d080a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0d18005c40 tx=0x7f0d18009e40 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd63960 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0d18010bd0 tx=0x7f0d1801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16671faebf060e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d05ffb640 1 -- 172.21.15.27:0/1628278136 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1800cb40 con 0x56417cd61570 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d05ffb640 1 -- 172.21.15.27:0/1628278136 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d1800cce0 con 0x56417cd61570 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.390+0000 7f0d05ffb640 1 -- 172.21.15.27:0/1628278136 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d180058f0 con 0x56417cd61570 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 -- 172.21.15.27:0/1628278136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 msgr2=0x56417cd63960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd63960 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0d18010bd0 tx=0x7f0d1801c000 comp rx=0 tx=0).stop 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 -- 172.21.15.27:0/1628278136 shutdown_connections 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd63960 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd61030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 --2- 172.21.15.27:0/1628278136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd56960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 -- 172.21.15.27:0/1628278136 >> 172.21.15.27:0/1628278136 conn(0x56417cd528a0 msgr2=0x56417cd52c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 -- 172.21.15.27:0/1628278136 shutdown_connections 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 -- 172.21.15.27:0/1628278136 wait complete. 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 -- start start 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 Processor -- start 2024-04-25T01:27:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56580 0x56417cd60ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.391+0000 7f0d28810740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd5e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d28810740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd61570 0x56417cd5e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d28810740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56417cd64300 con 0x56417cd56580 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d28810740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56417cc77050 con 0x56417cd56ea0 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d28810740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56417cd64480 con 0x56417cd61570 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56580 0x56417cd60ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d28810740 10 client.0 _ll_register_callbacks cb 0x56417c9cebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56580 0x56417cd60ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54474/0 (socket says 172.21.15.27:54474) 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 -- 172.21.15.27:0/518248588 learned_addr learned my addr 172.21.15.27:0/518248588 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd61570 0x56417cd5e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 -- 172.21.15.27:0/518248588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd61570 msgr2=0x56417cd5e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd61570 0x56417cd5e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 -- 172.21.15.27:0/518248588 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 msgr2=0x56417cd5e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd5e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 -- 172.21.15.27:0/518248588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417cc48dc0 con 0x56417cd56580 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56580 0x56417cd60ea0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009fd0 tx=0x7f0d100039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21df096dec4c6831 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0cfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0cfeffd640 1 -- 172.21.15.27:0/518248588 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10013930 con 0x56417cd56580 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0cfeffd640 1 -- 172.21.15.27:0/518248588 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0d10013ad0 con 0x56417cd56580 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d28810740 1 -- 172.21.15.27:0/518248588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d1800cce0 con 0x56417cd56580 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0cfeffd640 1 -- 172.21.15.27:0/518248588 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0d10013cd0 con 0x56417cd56580 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d28810740 10 client.16143 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.392+0000 7f0d28810740 1 -- 172.21.15.27:0/518248588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56417c9cf3f0 con 0x56417cd56580 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0cfeffd640 1 -- 172.21.15.27:0/518248588 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d1800cce0 con 0x56417cd56580 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0cfeffd640 1 -- 172.21.15.27:0/518248588 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f0d1001a070 con 0x56417cd56580 2024-04-25T01:27:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0d28810740 10 client.16143 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0d28810740 10 client.16143 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0d28810740 20 client.16143 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0d28810740 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56417cd716c0 0x56417cd73b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0d28810740 4 client.16143 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0d28810740 1 -- 172.21.15.27:0/518248588 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56417ccc8bb0 con 0x56417cd716c0 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.393+0000 7f0d1effd640 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56417cd716c0 0x56417cd73b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.394+0000 7f0d1effd640 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56417cd716c0 0x56417cd73b10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c0f989fb79c0a9fc server_cookie=945919c8f9f5b2d0 in_seq=0 out_seq=0 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.394+0000 7f0cfeffd640 10 client.16143 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.394+0000 7f0cfeffd640 1 -- 172.21.15.27:0/518248588 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56417ccc8bb0 con 0x56417cd716c0 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.394+0000 7f0cfeffd640 10 client.16143 handle_command_reply: tid=0 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.394+0000 7f0d28810740 1 client.16143 shutdown 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 -- 172.21.15.27:0/518248588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56580 msgr2=0x56417cd60ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56580 0x56417cd60ea0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009fd0 tx=0x7f0d100039c0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 -- 172.21.15.27:0/518248588 shutdown_connections 2024-04-25T01:27:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56417cd716c0 0x56417cd73b10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd61570 0x56417cd5e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd5e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0cfeffd640 0 client.16143 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:27:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 --2- 172.21.15.27:0/518248588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56580 0x56417cd60ea0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 -- 172.21.15.27:0/518248588 >> 172.21.15.27:0/518248588 conn(0x56417cd528a0 msgr2=0x56417cd51b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 -- 172.21.15.27:0/518248588 shutdown_connections 2024-04-25T01:27:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 1 -- 172.21.15.27:0/518248588 wait complete. 2024-04-25T01:27:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.395+0000 7f0d28810740 20 client.16143 trim_cache size 0 max 16384 2024-04-25T01:27:41.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d28810740 1 Processor -- start 2024-04-25T01:27:41.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d28810740 1 -- start start 2024-04-25T01:27:41.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d28810740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd59860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d28810740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d28810740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd793b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d28810740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b14a0 con 0x56417cd61570 2024-04-25T01:27:41.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d28810740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201aed70 con 0x56417cd56ea0 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d28810740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201acbc0 con 0x56417cd56580 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd793b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd793b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54478/0 (socket says 172.21.15.27:54478) 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.408+0000 7f0d1ffff640 1 -- 172.21.15.27:0/3199733126 learned_addr learned my addr 172.21.15.27:0/3199733126 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/3199733126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd59860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d1ffff640 1 -- 172.21.15.27:0/3199733126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 msgr2=0x56417cd59860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/3199733126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd59860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d1ffff640 1 -- 172.21.15.27:0/3199733126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 msgr2=0x56417cd609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/3199733126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d1ffff640 1 -- 172.21.15.27:0/3199733126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417ca997d0 con 0x56417cd61570 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/3199733126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd793b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0d180054e0 tx=0x7f0d18005470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613772c0f9c1591b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d057fa640 1 -- 172.21.15.27:0/3199733126 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18005620 con 0x56417cd61570 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d057fa640 1 -- 172.21.15.27:0/3199733126 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d180057c0 con 0x56417cd61570 2024-04-25T01:27:41.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.409+0000 7f0d057fa640 1 -- 172.21.15.27:0/3199733126 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d180316e0 con 0x56417cd61570 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 -- 172.21.15.27:0/3199733126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 msgr2=0x56417cd793b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 --2- 172.21.15.27:0/3199733126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd793b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0d180054e0 tx=0x7f0d18005470 comp rx=0 tx=0).stop 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 -- 172.21.15.27:0/3199733126 shutdown_connections 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 --2- 172.21.15.27:0/3199733126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd61570 0x56417cd793b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 --2- 172.21.15.27:0/3199733126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd56ea0 0x56417cd609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 --2- 172.21.15.27:0/3199733126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd59860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 -- 172.21.15.27:0/3199733126 >> 172.21.15.27:0/3199733126 conn(0x56417cd528a0 msgr2=0x56417cd57de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 -- 172.21.15.27:0/3199733126 shutdown_connections 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 -- 172.21.15.27:0/3199733126 wait complete. 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 -- start start 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 Processor -- start 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd65480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.410+0000 7f0d28810740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56ea0 0x56417cd659c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d28810740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd61570 0x56417cdd6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d28810740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201ad230 con 0x56417cd56ea0 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d28810740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201ae6a0 con 0x56417cd61570 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d28810740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201ad900 con 0x56417cd56580 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d28810740 10 client.0 _ll_register_callbacks cb 0x7f0d100041e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56ea0 0x56417cd659c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56ea0 0x56417cd659c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54484/0 (socket says 172.21.15.27:54484) 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 -- 172.21.15.27:0/639289368 learned_addr learned my addr 172.21.15.27:0/639289368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd61570 0x56417cdd6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd65480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 -- 172.21.15.27:0/639289368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 msgr2=0x56417cd65480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd65480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 -- 172.21.15.27:0/639289368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd61570 msgr2=0x56417cdd6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd61570 0x56417cdd6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 -- 172.21.15.27:0/639289368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417cd66250 con 0x56417cd56ea0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1ffff640 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd61570 0x56417cdd6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d1effd640 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56ea0 0x56417cd659c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0d140ed990 tx=0x7f0d14005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b8bbb24d76184d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0cfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0cfeffd640 1 -- 172.21.15.27:0/639289368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1409e070 con 0x56417cd56ea0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0cfeffd640 1 -- 172.21.15.27:0/639289368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0d1403b460 con 0x56417cd56ea0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d28810740 1 -- 172.21.15.27:0/639289368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d180057c0 con 0x56417cd56ea0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0cfeffd640 1 -- 172.21.15.27:0/639289368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0d14049040 con 0x56417cd56ea0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d28810740 10 client.16149 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.411+0000 7f0d28810740 1 -- 172.21.15.27:0/639289368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56417cdd8290 con 0x56417cd56ea0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0cfeffd640 1 -- 172.21.15.27:0/639289368 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d180057c0 con 0x56417cd56ea0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0cfeffd640 1 -- 172.21.15.27:0/639289368 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f0d1403bce0 con 0x56417cd56ea0 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0d28810740 10 client.16149 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0d28810740 10 client.16149 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:27:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0d28810740 20 client.16149 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0d28810740 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56417cdd8a40 0x56417cddae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0d28810740 4 client.16149 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0d28810740 1 -- 172.21.15.27:0/639289368 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56417ccc2f00 con 0x56417cdd8a40 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.412+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56417cdd8a40 0x56417cddae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d1f7fe640 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56417cdd8a40 0x56417cddae30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d32ec8fffc6713a server_cookie=2e2377677c5b2d0b in_seq=0 out_seq=0 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0cfeffd640 10 client.16149 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0cfeffd640 1 -- 172.21.15.27:0/639289368 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56417ccc2f00 con 0x56417cdd8a40 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0cfeffd640 10 client.16149 handle_command_reply: tid=0 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 client.16149 shutdown 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 -- 172.21.15.27:0/639289368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56ea0 msgr2=0x56417cd659c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56ea0 0x56417cd659c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0d140ed990 tx=0x7f0d14005dc0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 -- 172.21.15.27:0/639289368 shutdown_connections 2024-04-25T01:27:41.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56417cdd8a40 0x56417cddae30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56417cd61570 0x56417cdd6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56417cd56ea0 0x56417cd659c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0cfeffd640 0 client.16149 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 --2- 172.21.15.27:0/639289368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56417cd56580 0x56417cd65480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 -- 172.21.15.27:0/639289368 >> 172.21.15.27:0/639289368 conn(0x56417cd528a0 msgr2=0x56417cd51b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 -- 172.21.15.27:0/639289368 shutdown_connections 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 1 -- 172.21.15.27:0/639289368 wait complete. 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.413+0000 7f0d28810740 20 client.16149 trim_cache size 0 max 16384 2024-04-25T01:27:41.398 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:27:41.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 -- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0d080a2d00 msgr2=0x7f0d080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0d080a2d00 0x7f0d080a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0d18005c40 tx=0x7f0d18009e40 comp rx=0 tx=0).stop 2024-04-25T01:27:41.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 -- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d201020d0 msgr2=0x7f0d20198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d201020d0 0x7f0d20198600 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0d1402a940 tx=0x7f0d14003b90 comp rx=0 tx=0).stop 2024-04-25T01:27:41.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 -- 172.21.15.27:0/2947909932 shutdown_connections 2024-04-25T01:27:41.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d20107720 0x7f0d2019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0d080a2d00 0x7f0d080a51c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d201020d0 0x7f0d20198600 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 --2- 172.21.15.27:0/2947909932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d20101710 0x7f0d201980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 -- 172.21.15.27:0/2947909932 >> 172.21.15.27:0/2947909932 conn(0x7f0d200fd180 msgr2=0x7f0d200fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 -- 172.21.15.27:0/2947909932 shutdown_connections 2024-04-25T01:27:41.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.414+0000 7f0d25b09640 1 -- 172.21.15.27:0/2947909932 wait complete. 2024-04-25T01:27:41.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 -- 172.21.15.27:0/1724372260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 msgr2=0x7fa5b410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1724372260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 0x7fa5b410e5f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009cd0 tx=0x7fa5ac02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 -- 172.21.15.27:0/1724372260 shutdown_connections 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1724372260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 0x7fa5b410e5f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1724372260 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5b4069dc0 0x7fa5b406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1724372260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5b4107730 0x7fa5b4107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 -- 172.21.15.27:0/1724372260 >> 172.21.15.27:0/1724372260 conn(0x7fa5b40776a0 msgr2=0x7fa5b4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 -- 172.21.15.27:0/1724372260 shutdown_connections 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.532+0000 7fa5b9695640 1 -- 172.21.15.27:0/1724372260 wait complete. 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b9695640 1 Processor -- start 2024-04-25T01:27:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b9695640 1 -- start start 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b9695640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5b4069dc0 0x7fa5b4100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b9695640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 0x7fa5b4100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b9695640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5b4107730 0x7fa5b41010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b9695640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b4111b40 con 0x7fa5b406a780 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b9695640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41119c0 con 0x7fa5b4069dc0 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b9695640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b4111cc0 con 0x7fa5b4107730 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 0x7fa5b4100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 0x7fa5b4100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54502/0 (socket says 172.21.15.27:54502) 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5abfff640 1 -- 172.21.15.27:0/1647974172 learned_addr learned my addr 172.21.15.27:0/1647974172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5b4069dc0 0x7fa5b4100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5b4107730 0x7fa5b41010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5abfff640 1 -- 172.21.15.27:0/1647974172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5b4107730 msgr2=0x7fa5b41010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5abfff640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5b4107730 0x7fa5b41010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5abfff640 1 -- 172.21.15.27:0/1647974172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5b4069dc0 msgr2=0x7fa5b4100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5abfff640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5b4069dc0 0x7fa5b4100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5abfff640 1 -- 172.21.15.27:0/1647974172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b41a64c0 con 0x7fa5b406a780 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5b4107730 0x7fa5b41010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.533+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5b4069dc0 0x7fa5b4100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.534+0000 7fa5abfff640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 0x7fa5b4100bb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00e9f0 tx=0x7fa59c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.534+0000 7fa5b0ff9640 1 -- 172.21.15.27:0/1647974172 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c00cd80 con 0x7fa5b406a780 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.534+0000 7fa5b0ff9640 1 -- 172.21.15.27:0/1647974172 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59c010910 con 0x7fa5b406a780 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.534+0000 7fa5b9695640 1 -- 172.21.15.27:0/1647974172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b41a67b0 con 0x7fa5b406a780 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.534+0000 7fa5b0ff9640 1 -- 172.21.15.27:0/1647974172 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c018990 con 0x7fa5b406a780 2024-04-25T01:27:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.534+0000 7fa5b9695640 1 -- 172.21.15.27:0/1647974172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b41a6d00 con 0x7fa5b406a780 2024-04-25T01:27:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.536+0000 7fa5b0ff9640 1 -- 172.21.15.27:0/1647974172 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa59c010ac0 con 0x7fa5b406a780 2024-04-25T01:27:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.536+0000 7fa5bc39c740 1 Processor -- start 2024-04-25T01:27:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.536+0000 7fa5bc39c740 1 -- start start 2024-04-25T01:27:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22ea9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9ea0 0x558d22eb4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22eb6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558d22e0e670 con 0x558d22ea9580 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558d22db6970 con 0x558d22eb4570 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558d22e0a630 con 0x558d22ea9ea0 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22ea9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b0ff9640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa5a00a2d00 0x7fa5a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22ea9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54506/0 (socket says 172.21.15.27:54506) 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 -- 172.21.15.27:0/1332485748 learned_addr learned my addr 172.21.15.27:0/1332485748 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5abfff640 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22eb6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9ea0 0x558d22eb4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa5a00a2d00 0x7fa5a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b0ff9640 1 -- 172.21.15.27:0/1647974172 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa59c0cfc50 con 0x7fa5b406a780 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 -- 172.21.15.27:0/1332485748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9ea0 msgr2=0x558d22eb4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9ea0 0x558d22eb4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 -- 172.21.15.27:0/1332485748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 msgr2=0x558d22eb6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22eb6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b37fe640 1 -- 172.21.15.27:0/1332485748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d22dda400 con 0x558d22ea9580 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.537+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9ea0 0x558d22eb4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.538+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22ea9960 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009a60 tx=0x7fa5ac031dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620ebdc3e35f91d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.538+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa5a00a2d00 0x7fa5a00a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009590 tx=0x7fa5ac036000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.538+0000 7fa5a97fa640 1 -- 172.21.15.27:0/1332485748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac04c070 con 0x558d22ea9580 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.538+0000 7fa5a97fa640 1 -- 172.21.15.27:0/1332485748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5ac036070 con 0x558d22ea9580 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.538+0000 7fa5a97fa640 1 -- 172.21.15.27:0/1332485748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac030ac0 con 0x558d22ea9580 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.538+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1332485748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 msgr2=0x558d22ea9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22ea9960 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009a60 tx=0x7fa5ac031dc0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1332485748 shutdown_connections 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22eb6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9ea0 0x558d22eb4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/1332485748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22ea9960 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1332485748 >> 172.21.15.27:0/1332485748 conn(0x558d22ea58a0 msgr2=0x558d22ea5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1332485748 shutdown_connections 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1332485748 wait complete. 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 -- start start 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 Processor -- start 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22eb3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22ea9ea0 0x558d22eb1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22eb4570 0x558d22eb1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558d22eb7300 con 0x558d22ea9580 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558d22dca050 con 0x558d22ea9ea0 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558d22eb7480 con 0x558d22eb4570 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5b2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22eb3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5bc39c740 10 client.0 _ll_register_callbacks cb 0x558d22b21be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5b2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22eb3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54520/0 (socket says 172.21.15.27:54520) 2024-04-25T01:27:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.539+0000 7fa5b2ffd640 1 -- 172.21.15.27:0/916157781 learned_addr learned my addr 172.21.15.27:0/916157781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5abfff640 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22ea9ea0 0x558d22eb1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5b2ffd640 1 -- 172.21.15.27:0/916157781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22eb4570 msgr2=0x558d22eb1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22eb4570 0x558d22eb1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5b2ffd640 1 -- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22ea9ea0 msgr2=0x558d22eb1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22ea9ea0 0x558d22eb1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5b2ffd640 1 -- 172.21.15.27:0/916157781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d22d9bdc0 con 0x558d22ea9580 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5abfff640 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22ea9ea0 0x558d22eb1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22eb3f40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400bf40 tx=0x7fa5a40128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673ad6b472e21fcf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa59a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa59a7fc640 1 -- 172.21.15.27:0/916157781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4005220 con 0x558d22ea9580 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa59a7fc640 1 -- 172.21.15.27:0/916157781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa5a40053c0 con 0x558d22ea9580 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5bc39c740 1 -- 172.21.15.27:0/916157781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5ac036070 con 0x558d22ea9580 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa59a7fc640 1 -- 172.21.15.27:0/916157781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa5a4025050 con 0x558d22ea9580 2024-04-25T01:27:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5bc39c740 10 client.16161 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa5bc39c740 1 -- 172.21.15.27:0/916157781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558d22b223f0 con 0x558d22ea9580 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.540+0000 7fa59a7fc640 1 -- 172.21.15.27:0/916157781 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5ac036070 con 0x558d22ea9580 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa59a7fc640 1 -- 172.21.15.27:0/916157781 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fa5a4019030 con 0x558d22ea9580 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa5bc39c740 10 client.16161 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa5bc39c740 10 client.16161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa5bc39c740 20 client.16161 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558d22ec4670 0x558d22ec6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa5bc39c740 4 client.16161 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa5bc39c740 1 -- 172.21.15.27:0/916157781 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d22e1bbb0 con 0x558d22ec4670 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa5abfff640 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558d22ec4670 0x558d22ec6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa5abfff640 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558d22ec4670 0x558d22ec6b10 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af7f6e0e7029fce6 server_cookie=7c8f3428c917d6dd in_seq=0 out_seq=0 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.541+0000 7fa59a7fc640 10 client.16161 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa59a7fc640 1 -- 172.21.15.27:0/916157781 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d22e1bbb0 con 0x558d22ec4670 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa59a7fc640 10 client.16161 handle_command_reply: tid=0 2024-04-25T01:27:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 client.16161 shutdown 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 -- 172.21.15.27:0/916157781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 msgr2=0x558d22eb3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22eb3f40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400bf40 tx=0x7fa5a40128e0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 -- 172.21.15.27:0/916157781 shutdown_connections 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558d22ec4670 0x558d22ec6b10 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22eb4570 0x558d22eb1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22ea9ea0 0x558d22eb1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/916157781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9580 0x558d22eb3f40 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa59a7fc640 0 client.16161 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 -- 172.21.15.27:0/916157781 >> 172.21.15.27:0/916157781 conn(0x558d22ea58a0 msgr2=0x558d22eab6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 -- 172.21.15.27:0/916157781 shutdown_connections 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 1 -- 172.21.15.27:0/916157781 wait complete. 2024-04-25T01:27:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.542+0000 7fa5bc39c740 20 client.16161 trim_cache size 0 max 16384 2024-04-25T01:27:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5bc39c740 1 Processor -- start 2024-04-25T01:27:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5bc39c740 1 -- start start 2024-04-25T01:27:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 0x558d22ec01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22ecc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ecc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41b36b0 con 0x558d22ea9ea0 2024-04-25T01:27:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41b0f80 con 0x558d22eb4570 2024-04-25T01:27:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41aedd0 con 0x558d22ea9580 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22ecc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22ecc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54528/0 (socket says 172.21.15.27:54528) 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5abfff640 1 -- 172.21.15.27:0/1584710797 learned_addr learned my addr 172.21.15.27:0/1584710797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5abfff640 1 -- 172.21.15.27:0/1584710797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 msgr2=0x558d22ec01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5abfff640 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 0x558d22ec01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5abfff640 1 -- 172.21.15.27:0/1584710797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 msgr2=0x558d22ecc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ecc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5abfff640 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ecc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5abfff640 1 -- 172.21.15.27:0/1584710797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d22bec7d0 con 0x558d22ea9ea0 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.556+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ecc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5abfff640 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22ecc3e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0cd610 tx=0x7fa59c06ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60260575bfca5f55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5a8ff9640 1 -- 172.21.15.27:0/1584710797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c0a0190 con 0x558d22ea9ea0 2024-04-25T01:27:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5a8ff9640 1 -- 172.21.15.27:0/1584710797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59c0a4a50 con 0x558d22ea9ea0 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5a8ff9640 1 -- 172.21.15.27:0/1584710797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c018d90 con 0x558d22ea9ea0 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1584710797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 msgr2=0x558d22ecc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22ecc3e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0cd610 tx=0x7fa59c06ef10 comp rx=0 tx=0).stop 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1584710797 shutdown_connections 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ecc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22ecc3e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/1584710797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 0x558d22ec01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1584710797 >> 172.21.15.27:0/1584710797 conn(0x558d22ea58a0 msgr2=0x558d22eafb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1584710797 shutdown_connections 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.557+0000 7fa5bc39c740 1 -- 172.21.15.27:0/1584710797 wait complete. 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 1 -- start start 2024-04-25T01:27:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 1 Processor -- start 2024-04-25T01:27:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 0x558d22eba370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22eba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ebadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41af440 con 0x558d22ea9ea0 2024-04-25T01:27:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41b08b0 con 0x558d22eb4570 2024-04-25T01:27:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b41afb10 con 0x558d22ea9580 2024-04-25T01:27:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5bc39c740 10 client.0 _ll_register_callbacks cb 0x7fa5a400b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22eba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22eba8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54530/0 (socket says 172.21.15.27:54530) 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5abfff640 1 -- 172.21.15.27:0/246281132 learned_addr learned my addr 172.21.15.27:0/246281132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 0x558d22eba370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ebadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5abfff640 1 -- 172.21.15.27:0/246281132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 msgr2=0x558d22eba370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5abfff640 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 0x558d22eba370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5abfff640 1 -- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 msgr2=0x558d22ebadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5abfff640 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ebadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5abfff640 1 -- 172.21.15.27:0/246281132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d22f29eb0 con 0x558d22ea9ea0 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.558+0000 7fa5b37fe640 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ebadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5abfff640 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22eba8b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0a4520 tx=0x7fa59c005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1883c3d08386a85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa59a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa59a7fc640 1 -- 172.21.15.27:0/246281132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c018b30 con 0x558d22ea9ea0 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa59a7fc640 1 -- 172.21.15.27:0/246281132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa59c018cd0 con 0x558d22ea9ea0 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 1 -- 172.21.15.27:0/246281132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa59c0a4a50 con 0x558d22ea9ea0 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa59a7fc640 1 -- 172.21.15.27:0/246281132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa59c09b230 con 0x558d22ea9ea0 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 10 client.16167 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 1 -- 172.21.15.27:0/246281132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558d22f2b120 con 0x558d22ea9ea0 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa59a7fc640 1 -- 172.21.15.27:0/246281132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa59c0a4a50 con 0x558d22ea9ea0 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa59a7fc640 1 -- 172.21.15.27:0/246281132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fa59c09aca0 con 0x558d22ea9ea0 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 10 client.16167 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 10 client.16167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 20 client.16167 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558d22f2b790 0x558d22f2dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 4 client.16167 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.559+0000 7fa5bc39c740 1 -- 172.21.15.27:0/246281132 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558d22e15f00 con 0x558d22f2b790 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558d22f2b790 0x558d22f2dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5b2ffd640 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558d22f2b790 0x558d22f2dbb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6c66b7f5e601a07 server_cookie=45a716eea309fa7c in_seq=0 out_seq=0 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa59a7fc640 10 client.16167 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa59a7fc640 1 -- 172.21.15.27:0/246281132 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558d22e15f00 con 0x558d22f2b790 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa59a7fc640 10 client.16167 handle_command_reply: tid=0 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 client.16167 shutdown 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 -- 172.21.15.27:0/246281132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 msgr2=0x558d22eba8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22eba8b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0a4520 tx=0x7fa59c005cc0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 -- 172.21.15.27:0/246281132 shutdown_connections 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558d22f2b790 0x558d22f2dbb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558d22eb4570 0x558d22ebadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d22ea9ea0 0x558d22eba8b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 --2- 172.21.15.27:0/246281132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d22ea9580 0x558d22eba370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 -- 172.21.15.27:0/246281132 >> 172.21.15.27:0/246281132 conn(0x558d22ea58a0 msgr2=0x558d22ebf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa59a7fc640 0 client.16167 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 -- 172.21.15.27:0/246281132 shutdown_connections 2024-04-25T01:27:41.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 1 -- 172.21.15.27:0/246281132 wait complete. 2024-04-25T01:27:41.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.560+0000 7fa5bc39c740 20 client.16167 trim_cache size 0 max 16384 2024-04-25T01:27:41.546 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 -- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa5a00a2d00 msgr2=0x7fa5a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa5a00a2d00 0x7fa5a00a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009590 tx=0x7fa5ac036000 comp rx=0 tx=0).stop 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 -- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 msgr2=0x7fa5b4100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 0x7fa5b4100bb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00e9f0 tx=0x7fa59c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 -- 172.21.15.27:0/1647974172 shutdown_connections 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5b4107730 0x7fa5b41010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa5a00a2d00 0x7fa5a00a51c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b406a780 0x7fa5b4100bb0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 --2- 172.21.15.27:0/1647974172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5b4069dc0 0x7fa5b4100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 -- 172.21.15.27:0/1647974172 >> 172.21.15.27:0/1647974172 conn(0x7fa5b40776a0 msgr2=0x7fa5b40fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 -- 172.21.15.27:0/1647974172 shutdown_connections 2024-04-25T01:27:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:41.561+0000 7fa5b9695640 1 -- 172.21.15.27:0/1647974172 wait complete. 2024-04-25T01:27:42.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:42 smithi027 ceph-mon[29197]: pgmap v71: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:42.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:42 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/320005402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:42.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:42 smithi153 ceph-mon[32621]: pgmap v71: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:42.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:42 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/320005402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:42.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:42 smithi139 ceph-mon[32760]: pgmap v71: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:42.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:42 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/320005402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:42.550 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:27:42.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:42.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 -- 172.21.15.27:0/3191792507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610106810 msgr2=0x7ff610106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 --2- 172.21.15.27:0/3191792507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610106810 0x7ff610106bf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff608009930 tx=0x7ff60802a420 comp rx=0 tx=0).stop 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 -- 172.21.15.27:0/3191792507 shutdown_connections 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 --2- 172.21.15.27:0/3191792507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6101012e0 0x7ff61010d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 --2- 172.21.15.27:0/3191792507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff610100920 0x7ff610100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 --2- 172.21.15.27:0/3191792507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610106810 0x7ff610106bf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 -- 172.21.15.27:0/3191792507 >> 172.21.15.27:0/3191792507 conn(0x7ff610077620 msgr2=0x7ff610077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 -- 172.21.15.27:0/3191792507 shutdown_connections 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.691+0000 7ff61864d640 1 -- 172.21.15.27:0/3191792507 wait complete. 2024-04-25T01:27:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff61864d640 1 Processor -- start 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff61864d640 1 -- start start 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff61864d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610100920 0x7ff610102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff61864d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6101012e0 0x7ff610102850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff61864d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff610106810 0x7ff610102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff61864d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff610110ae0 con 0x7ff610100920 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff61864d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff610110960 con 0x7ff6101012e0 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff61864d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff610110c60 con 0x7ff610106810 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff6163e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610100920 0x7ff610102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff6163e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610100920 0x7ff610102310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54540/0 (socket says 172.21.15.27:54540) 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.692+0000 7ff6163e9640 1 -- 172.21.15.27:0/3913613454 learned_addr learned my addr 172.21.15.27:0/3913613454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff615be8640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6101012e0 0x7ff610102850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff616bea640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff610106810 0x7ff610102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff6163e9640 1 -- 172.21.15.27:0/3913613454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff610106810 msgr2=0x7ff610102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff6163e9640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff610106810 0x7ff610102d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff6163e9640 1 -- 172.21.15.27:0/3913613454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6101012e0 msgr2=0x7ff610102850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff6163e9640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6101012e0 0x7ff610102850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff6163e9640 1 -- 172.21.15.27:0/3913613454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6101a6520 con 0x7ff610100920 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff615be8640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6101012e0 0x7ff610102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff6163e9640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610100920 0x7ff610102310 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff608009230 tx=0x7ff608004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff60f7fe640 1 -- 172.21.15.27:0/3913613454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60803f030 con 0x7ff610100920 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff60f7fe640 1 -- 172.21.15.27:0/3913613454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff608003ad0 con 0x7ff610100920 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff61864d640 1 -- 172.21.15.27:0/3913613454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6101a67b0 con 0x7ff610100920 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff60f7fe640 1 -- 172.21.15.27:0/3913613454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608035070 con 0x7ff610100920 2024-04-25T01:27:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.693+0000 7ff61864d640 1 -- 172.21.15.27:0/3913613454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6101a6ca0 con 0x7ff610100920 2024-04-25T01:27:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.696+0000 7ff60f7fe640 1 -- 172.21.15.27:0/3913613454 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff608042030 con 0x7ff610100920 2024-04-25T01:27:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.696+0000 7ff61b439740 1 Processor -- start 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff61b439740 1 -- start start 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff61b439740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfe3980 0x55bd5bfe1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff61b439740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff61b439740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfed010 0x55bd5bfef400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff61b439740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bd5bf462c0 con 0x55bd5bfe3980 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff61b439740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bd5beee910 con 0x55bd5bfed010 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff61b439740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bd5bf42280 con 0x55bd5bfe2290 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff60f7fe640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff5f80a2d30 0x7ff5f80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff616bea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfe3980 0x55bd5bfe1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff615be8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfed010 0x55bd5bfef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff616bea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfe3980 0x55bd5bfe1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54556/0 (socket says 172.21.15.27:54556) 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff615be8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfed010 0x55bd5bfef400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:60110/0 (socket says 172.21.15.27:60110) 2024-04-25T01:27:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff616bea640 1 -- 172.21.15.27:0/2321388305 learned_addr learned my addr 172.21.15.27:0/2321388305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff6163e9640 1 --2- 172.21.15.27:0/2321388305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff616bea640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff5f80a2d30 0x7ff5f80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff60f7fe640 1 -- 172.21.15.27:0/3913613454 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff6080efd40 con 0x7ff610100920 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.697+0000 7ff615be8640 1 -- 172.21.15.27:0/2321388305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 msgr2=0x55bd5bfe2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff615be8640 1 --2- 172.21.15.27:0/2321388305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff615be8640 1 -- 172.21.15.27:0/2321388305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfe3980 msgr2=0x55bd5bfe1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff615be8640 1 --2- 172.21.15.27:0/2321388305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfe3980 0x55bd5bfe1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff615be8640 1 -- 172.21.15.27:0/2321388305 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd5befffb0 con 0x55bd5bfed010 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff615be8640 1 --2- 172.21.15.27:0/2321388305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfed010 0x55bd5bfef400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff6040058c0 tx=0x7ff60400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0c3b1905c7159d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff616bea640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff5f80a2d30 0x7ff5f80a51f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff6000137c0 tx=0x7ff600004ab0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff60cbfe640 1 -- 172.21.15.27:0/2321388305 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604005220 con 0x55bd5bfed010 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff60cbfe640 1 -- 172.21.15.27:0/2321388305 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff604004c80 con 0x55bd5bfed010 2024-04-25T01:27:42.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.698+0000 7ff60cbfe640 1 -- 172.21.15.27:0/2321388305 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6040107a0 con 0x55bd5bfed010 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 -- 172.21.15.27:0/2321388305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfed010 msgr2=0x55bd5bfef400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 --2- 172.21.15.27:0/2321388305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfed010 0x55bd5bfef400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff6040058c0 tx=0x7ff60400e720 comp rx=0 tx=0).stop 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 -- 172.21.15.27:0/2321388305 shutdown_connections 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 --2- 172.21.15.27:0/2321388305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfed010 0x55bd5bfef400 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 --2- 172.21.15.27:0/2321388305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 --2- 172.21.15.27:0/2321388305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfe3980 0x55bd5bfe1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 -- 172.21.15.27:0/2321388305 >> 172.21.15.27:0/2321388305 conn(0x55bd5bfaa950 msgr2=0x55bd5bfde7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 -- 172.21.15.27:0/2321388305 shutdown_connections 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 -- 172.21.15.27:0/2321388305 wait complete. 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 -- start start 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.699+0000 7ff61b439740 1 Processor -- start 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff61b439740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff61b439740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bfea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff61b439740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bfea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff61b439740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bd5bfefea0 con 0x55bd5bfed010 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff61b439740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bd5bf01e90 con 0x55bd5bfe2290 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff61b439740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bd5bff0020 con 0x55bd5bfe3980 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff61b439740 10 client.0 _ll_register_callbacks cb 0x55bd5bf3d970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bfea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bfea910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54570/0 (socket says 172.21.15.27:54570) 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 -- 172.21.15.27:0/403409574 learned_addr learned my addr 172.21.15.27:0/403409574 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff6163e9640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff615be8640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bfea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 -- 172.21.15.27:0/403409574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 msgr2=0x55bd5bfea3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bfea3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 -- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 msgr2=0x55bd5bfe9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 -- 172.21.15.27:0/403409574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd5bd227e0 con 0x55bd5bfed010 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff615be8640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bfea3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff6163e9640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff616bea640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bfea910 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff600013300 tx=0x7ff600004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721781ce0239acd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff5f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/403409574 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600028060 con 0x55bd5bfed010 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff61b439740 1 -- 172.21.15.27:0/403409574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff604004c80 con 0x55bd5bfed010 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.700+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/403409574 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff600004030 con 0x55bd5bfed010 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/403409574 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff60002f050 con 0x55bd5bfed010 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff61b439740 10 client.16179 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff61b439740 1 -- 172.21.15.27:0/403409574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bd5bd82f70 con 0x55bd5bfed010 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/403409574 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff604004c80 con 0x55bd5bfed010 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/403409574 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7ff60000f650 con 0x55bd5bfed010 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff61b439740 10 client.16179 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff61b439740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff61b439740 20 client.16179 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff61b439740 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bd5bffd0b0 0x55bd5bfff530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff61b439740 4 client.16179 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff61b439740 1 -- 172.21.15.27:0/403409574 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd5bf1a5a0 con 0x55bd5bffd0b0 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.701+0000 7ff6163e9640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bd5bffd0b0 0x55bd5bfff530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.702+0000 7ff5f5ffb640 10 client.16179 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.702+0000 7ff6163e9640 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bd5bffd0b0 0x55bd5bfff530 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a0d55de3ae34fd server_cookie=d898cbfbcc905982 in_seq=0 out_seq=0 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.702+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/403409574 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd5bf1a5a0 con 0x55bd5bffd0b0 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.702+0000 7ff5f5ffb640 10 client.16179 handle_command_reply: tid=0 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 client.16179 shutdown 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 -- 172.21.15.27:0/403409574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 msgr2=0x55bd5bfea910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bfea910 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff600013300 tx=0x7ff600004a40 comp rx=0 tx=0).stop 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 -- 172.21.15.27:0/403409574 shutdown_connections 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bd5bffd0b0 0x55bd5bfff530 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bfea910 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bfea3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff5f5ffb640 0 client.16179 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:42.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 --2- 172.21.15.27:0/403409574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bfe9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 -- 172.21.15.27:0/403409574 >> 172.21.15.27:0/403409574 conn(0x55bd5bfaa950 msgr2=0x55bd5bfe45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 -- 172.21.15.27:0/403409574 shutdown_connections 2024-04-25T01:27:42.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 1 -- 172.21.15.27:0/403409574 wait complete. 2024-04-25T01:27:42.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.703+0000 7ff61b439740 20 client.16179 trim_cache size 0 max 16384 2024-04-25T01:27:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.721+0000 7ff61b439740 1 Processor -- start 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.721+0000 7ff61b439740 1 -- start start 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.721+0000 7ff61b439740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 0x55bd5bff8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.721+0000 7ff61b439740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe3980 0x55bd5bfebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.721+0000 7ff61b439740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5c004e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.721+0000 7ff61b439740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101b3630 con 0x55bd5bfed010 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.721+0000 7ff61b439740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101b0f00 con 0x55bd5bfe3980 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.721+0000 7ff61b439740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101aed50 con 0x55bd5bfe2290 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5c004e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5c004e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54580/0 (socket says 172.21.15.27:54580) 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 -- 172.21.15.27:0/2970454748 learned_addr learned my addr 172.21.15.27:0/2970454748 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 -- 172.21.15.27:0/2970454748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 msgr2=0x55bd5bff8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 --2- 172.21.15.27:0/2970454748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 0x55bd5bff8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 -- 172.21.15.27:0/2970454748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe3980 msgr2=0x55bd5bfebdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 --2- 172.21.15.27:0/2970454748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe3980 0x55bd5bfebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 -- 172.21.15.27:0/2970454748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd5bc1a170 con 0x55bd5bfed010 2024-04-25T01:27:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff616bea640 1 --2- 172.21.15.27:0/2970454748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5c004e00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff600009570 tx=0x7ff600009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17bd3b0cb21d979 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff5f7fff640 1 -- 172.21.15.27:0/2970454748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600038070 con 0x55bd5bfed010 2024-04-25T01:27:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff5f7fff640 1 -- 172.21.15.27:0/2970454748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff600013330 con 0x55bd5bfed010 2024-04-25T01:27:42.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.722+0000 7ff5f7fff640 1 -- 172.21.15.27:0/2970454748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60001c040 con 0x55bd5bfed010 2024-04-25T01:27:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- 172.21.15.27:0/2970454748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 msgr2=0x55bd5c004e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 --2- 172.21.15.27:0/2970454748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5c004e00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff600009570 tx=0x7ff600009c70 comp rx=0 tx=0).stop 2024-04-25T01:27:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- 172.21.15.27:0/2970454748 shutdown_connections 2024-04-25T01:27:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 --2- 172.21.15.27:0/2970454748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5c004e00 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 --2- 172.21.15.27:0/2970454748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe3980 0x55bd5bfebdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 --2- 172.21.15.27:0/2970454748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe2290 0x55bd5bff8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- 172.21.15.27:0/2970454748 >> 172.21.15.27:0/2970454748 conn(0x55bd5bfaa950 msgr2=0x55bd5bfeb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- 172.21.15.27:0/2970454748 shutdown_connections 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- 172.21.15.27:0/2970454748 wait complete. 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- start start 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 Processor -- start 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bff52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bff5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bff03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101af3c0 con 0x55bd5bfed010 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101b0830 con 0x55bd5bfe2290 2024-04-25T01:27:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101afa90 con 0x55bd5bfe3980 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff61b439740 10 client.0 _ll_register_callbacks cb 0x7ff600013e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff616bea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bff03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff616bea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bff03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54582/0 (socket says 172.21.15.27:54582) 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.723+0000 7ff616bea640 1 -- 172.21.15.27:0/3996558062 learned_addr learned my addr 172.21.15.27:0/3996558062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff6163e9640 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bff52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff615be8640 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bff5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff616bea640 1 -- 172.21.15.27:0/3996558062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 msgr2=0x55bd5bff5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff616bea640 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bff5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff616bea640 1 -- 172.21.15.27:0/3996558062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 msgr2=0x55bd5bff52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff616bea640 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bff52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff616bea640 1 -- 172.21.15.27:0/3996558062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd5bff0be0 con 0x55bd5bfed010 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff615be8640 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bff5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff616bea640 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bff03b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff6000118d0 tx=0x7ff600000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a46798628c6737 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff5f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/3996558062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600041020 con 0x55bd5bfed010 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/3996558062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff60000f600 con 0x55bd5bfed010 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff61b439740 1 -- 172.21.15.27:0/3996558062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff600013330 con 0x55bd5bfed010 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/3996558062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff600049050 con 0x55bd5bfed010 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff61b439740 10 client.16185 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.724+0000 7ff61b439740 1 -- 172.21.15.27:0/3996558062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bd5c0642d0 con 0x55bd5bfed010 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/3996558062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff600013330 con 0x55bd5bfed010 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/3996558062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7ff600032070 con 0x55bd5bfed010 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff61b439740 10 client.16185 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff61b439740 10 client.16185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff61b439740 20 client.16185 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff61b439740 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bd5c064930 0x55bd5c066d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff61b439740 4 client.16185 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff61b439740 1 -- 172.21.15.27:0/3996558062 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bd5bf53800 con 0x55bd5c064930 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff6163e9640 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bd5c064930 0x55bd5c066d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff6163e9640 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bd5c064930 0x55bd5c066d50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9a30530f4ca4e5c server_cookie=3b029ce6650c0df5 in_seq=0 out_seq=0 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.725+0000 7ff5f5ffb640 10 client.16185 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.726+0000 7ff5f5ffb640 1 -- 172.21.15.27:0/3996558062 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bd5bf53800 con 0x55bd5c064930 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.726+0000 7ff5f5ffb640 10 client.16185 handle_command_reply: tid=0 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.726+0000 7ff61b439740 1 client.16185 shutdown 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 -- 172.21.15.27:0/3996558062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 msgr2=0x55bd5bff03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bff03b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff6000118d0 tx=0x7ff600000f40 comp rx=0 tx=0).stop 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 -- 172.21.15.27:0/3996558062 shutdown_connections 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bd5c064930 0x55bd5c066d50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff5f5ffb640 0 client.16185 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bd5bfed010 0x55bd5bff03b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bd5bfe3980 0x55bd5bff5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 --2- 172.21.15.27:0/3996558062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bd5bfe2290 0x55bd5bff52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 -- 172.21.15.27:0/3996558062 >> 172.21.15.27:0/3996558062 conn(0x55bd5bfaa950 msgr2=0x55bd5bfdd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 -- 172.21.15.27:0/3996558062 shutdown_connections 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 1 -- 172.21.15.27:0/3996558062 wait complete. 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61b439740 20 client.16185 trim_cache size 0 max 16384 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "9ff6073b-5bd2-4468-9f51-2112f977ad0d", 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:27:42.711 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:27:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61864d640 1 -- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff5f80a2d30 msgr2=0x7ff5f80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61864d640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff5f80a2d30 0x7ff5f80a51f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff6000137c0 tx=0x7ff600004ab0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61864d640 1 -- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610100920 msgr2=0x7ff610102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61864d640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610100920 0x7ff610102310 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff608009230 tx=0x7ff608004270 comp rx=0 tx=0).stop 2024-04-25T01:27:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61864d640 1 -- 172.21.15.27:0/3913613454 shutdown_connections 2024-04-25T01:27:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61864d640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff610106810 0x7ff610102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.727+0000 7ff61864d640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6101012e0 0x7ff610102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.728+0000 7ff61864d640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff5f80a2d30 0x7ff5f80a51f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.728+0000 7ff61864d640 1 --2- 172.21.15.27:0/3913613454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff610100920 0x7ff610102310 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.728+0000 7ff61864d640 1 -- 172.21.15.27:0/3913613454 >> 172.21.15.27:0/3913613454 conn(0x7ff610077620 msgr2=0x7ff610069f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.728+0000 7ff61864d640 1 -- 172.21.15.27:0/3913613454 shutdown_connections 2024-04-25T01:27:42.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.728+0000 7ff61864d640 1 -- 172.21.15.27:0/3913613454 wait complete. 2024-04-25T01:27:42.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:27:42.717 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:27:42.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 -- 172.21.15.27:0/443109822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c107730 msgr2=0x7f4d7c107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 --2- 172.21.15.27:0/443109822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c107730 0x7f4d7c107b10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009930 tx=0x7f4d6c02a420 comp rx=0 tx=0).stop 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 -- 172.21.15.27:0/443109822 shutdown_connections 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 --2- 172.21.15.27:0/443109822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d7c06a780 0x7f4d7c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 --2- 172.21.15.27:0/443109822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7c069dc0 0x7f4d7c06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 --2- 172.21.15.27:0/443109822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c107730 0x7f4d7c107b10 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 -- 172.21.15.27:0/443109822 >> 172.21.15.27:0/443109822 conn(0x7f4d7c0776a0 msgr2=0x7f4d7c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 -- 172.21.15.27:0/443109822 shutdown_connections 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 -- 172.21.15.27:0/443109822 wait complete. 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 Processor -- start 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.843+0000 7f4d8449e640 1 -- start start 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8449e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c069dc0 0x7f4d7c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8449e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7c06a780 0x7f4d7c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8449e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d7c107730 0x7f4d7c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8449e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c111b40 con 0x7f4d7c069dc0 2024-04-25T01:27:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8449e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1119c0 con 0x7f4d7c107730 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8449e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c111cc0 con 0x7f4d7c06a780 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c069dc0 0x7f4d7c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c069dc0 0x7f4d7c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54600/0 (socket says 172.21.15.27:54600) 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 -- 172.21.15.27:0/2285971661 learned_addr learned my addr 172.21.15.27:0/2285971661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d81a39640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7c06a780 0x7f4d7c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d82a3b640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d7c107730 0x7f4d7c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 -- 172.21.15.27:0/2285971661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7c06a780 msgr2=0x7f4d7c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7c06a780 0x7f4d7c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 -- 172.21.15.27:0/2285971661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d7c107730 msgr2=0x7f4d7c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d7c107730 0x7f4d7c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 -- 172.21.15.27:0/2285971661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c19cf60 con 0x7f4d7c069dc0 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d82a3b640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d7c107730 0x7f4d7c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8223a640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c069dc0 0x7f4d7c197fc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009230 tx=0x7f4d6c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d737fe640 1 -- 172.21.15.27:0/2285971661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c03f030 con 0x7f4d7c069dc0 2024-04-25T01:27:42.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d737fe640 1 -- 172.21.15.27:0/2285971661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d6c003ad0 con 0x7f4d7c069dc0 2024-04-25T01:27:42.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.844+0000 7f4d8449e640 1 -- 172.21.15.27:0/2285971661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c19d1f0 con 0x7f4d7c069dc0 2024-04-25T01:27:42.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.845+0000 7f4d737fe640 1 -- 172.21.15.27:0/2285971661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c035070 con 0x7f4d7c069dc0 2024-04-25T01:27:42.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.845+0000 7f4d8449e640 1 -- 172.21.15.27:0/2285971661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7c1a4ae0 con 0x7f4d7c069dc0 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.847+0000 7f4d737fe640 1 -- 172.21.15.27:0/2285971661 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4d6c042030 con 0x7f4d7c069dc0 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.847+0000 7f4d871ac740 1 Processor -- start 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.847+0000 7f4d871ac740 1 -- start start 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.847+0000 7f4d737fe640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d00 0x7f4d680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1e6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1e6f50 0x558b2a1f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1f1620 0x558b2a1f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d871ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558b2a14b6b0 con 0x558b2a1e6630 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d871ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558b2a0f3880 con 0x558b2a1f1620 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d871ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558b2a147670 con 0x558b2a1e6f50 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d82a3b640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d00 0x7f4d680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d737fe640 1 -- 172.21.15.27:0/2285971661 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4d6c0ef960 con 0x7f4d7c069dc0 2024-04-25T01:27:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1e6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1e6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54616/0 (socket says 172.21.15.27:54616) 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 -- 172.21.15.27:0/961680 learned_addr learned my addr 172.21.15.27:0/961680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d81a39640 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1e6f50 0x558b2a1f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 -- 172.21.15.27:0/961680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1e6f50 msgr2=0x558b2a1f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1e6f50 0x558b2a1f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d82a3b640 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1f1620 0x558b2a1f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 -- 172.21.15.27:0/961680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1f1620 msgr2=0x558b2a1f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1f1620 0x558b2a1f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 -- 172.21.15.27:0/961680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b29f297d0 con 0x558b2a1e6630 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d82a3b640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d00 0x7f4d680a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d74005cc0 tx=0x7f4d7400d710 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.848+0000 7f4d8223a640 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1e6a10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0c4230 tx=0x7f4d6c0bb630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8eba550e9b8f6e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d70bfe640 1 -- 172.21.15.27:0/961680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c049070 con 0x558b2a1e6630 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d70bfe640 1 -- 172.21.15.27:0/961680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d6c03b460 con 0x558b2a1e6630 2024-04-25T01:27:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d70bfe640 1 -- 172.21.15.27:0/961680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c0ecfa0 con 0x558b2a1e6630 2024-04-25T01:27:42.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 -- 172.21.15.27:0/961680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 msgr2=0x558b2a1e6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1e6a10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0c4230 tx=0x7f4d6c0bb630 comp rx=0 tx=0).stop 2024-04-25T01:27:42.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 -- 172.21.15.27:0/961680 shutdown_connections 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1f1620 0x558b2a1f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1e6f50 0x558b2a1f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 --2- 172.21.15.27:0/961680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1e6a10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 -- 172.21.15.27:0/961680 >> 172.21.15.27:0/961680 conn(0x558b2a1e2950 msgr2=0x558b2a1e2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 -- 172.21.15.27:0/961680 shutdown_connections 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.849+0000 7f4d871ac740 1 -- 172.21.15.27:0/961680 wait complete. 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 1 -- start start 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 1 Processor -- start 2024-04-25T01:27:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 0x558b2a1eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1ef030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558b2a1f41d0 con 0x558b2a1e6630 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558b2a106f50 con 0x558b2a1e6f50 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558b2a1f4350 con 0x558b2a1f1620 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d871ac740 10 client.0 _ll_register_callbacks cb 0x558b2a142000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d8223a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d8223a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54620/0 (socket says 172.21.15.27:54620) 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d8223a640 1 -- 172.21.15.27:0/1606794326 learned_addr learned my addr 172.21.15.27:0/1606794326 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d82a3b640 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.850+0000 7f4d81a39640 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 0x558b2a1eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d8223a640 1 -- 172.21.15.27:0/1606794326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 msgr2=0x558b2a1ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d8223a640 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d8223a640 1 -- 172.21.15.27:0/1606794326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 msgr2=0x558b2a1eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d8223a640 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 0x558b2a1eeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d8223a640 1 -- 172.21.15.27:0/1606794326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b29e5f3f0 con 0x558b2a1e6630 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d8223a640 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1ee5b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0b7f50 tx=0x7f4d6c003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73de2454aacf7d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d65ffb640 1 -- 172.21.15.27:0/1606794326 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c049040 con 0x558b2a1e6630 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d65ffb640 1 -- 172.21.15.27:0/1606794326 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4d6c0c42d0 con 0x558b2a1e6630 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 1 -- 172.21.15.27:0/1606794326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558b2a158bf0 con 0x558b2a1e6630 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d65ffb640 1 -- 172.21.15.27:0/1606794326 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4d6c0bc1f0 con 0x558b2a1e6630 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 10 client.16197 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 1 -- 172.21.15.27:0/1606794326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558b29e1f0f0 con 0x558b2a1e6630 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d65ffb640 1 -- 172.21.15.27:0/1606794326 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558b2a158bf0 con 0x558b2a1e6630 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d65ffb640 1 -- 172.21.15.27:0/1606794326 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f4d6c04c060 con 0x558b2a1e6630 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 10 client.16197 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 10 client.16197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 20 client.16197 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558b2a2014c0 0x558b2a2038b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 4 client.16197 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.851+0000 7f4d871ac740 1 -- 172.21.15.27:0/1606794326 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b2a152f40 con 0x558b2a2014c0 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.852+0000 7f4d81a39640 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558b2a2014c0 0x558b2a2038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.852+0000 7f4d81a39640 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558b2a2014c0 0x558b2a2038b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36aaf38dc80ccc6c server_cookie=6cb5ad51e436c775 in_seq=0 out_seq=0 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.852+0000 7f4d65ffb640 10 client.16197 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d65ffb640 1 -- 172.21.15.27:0/1606794326 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b2a152f40 con 0x558b2a2014c0 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d65ffb640 10 client.16197 handle_command_reply: tid=0 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 client.16197 shutdown 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 -- 172.21.15.27:0/1606794326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 msgr2=0x558b2a1ee5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1ee5b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0b7f50 tx=0x7f4d6c003f20 comp rx=0 tx=0).stop 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 -- 172.21.15.27:0/1606794326 shutdown_connections 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558b2a2014c0 0x558b2a2038b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 0x558b2a1eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d65ffb640 0 client.16197 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 --2- 172.21.15.27:0/1606794326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1ee5b0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 -- 172.21.15.27:0/1606794326 >> 172.21.15.27:0/1606794326 conn(0x558b2a1e2950 msgr2=0x558b2a1e8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 -- 172.21.15.27:0/1606794326 shutdown_connections 2024-04-25T01:27:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 1 -- 172.21.15.27:0/1606794326 wait complete. 2024-04-25T01:27:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.853+0000 7f4d871ac740 20 client.16197 trim_cache size 0 max 16384 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d871ac740 1 Processor -- start 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d871ac740 1 -- start start 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 0x558b2a1ed4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d871ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1b14e0 con 0x558b2a1e6630 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d871ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1aedb0 con 0x558b2a1e6f50 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d871ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1acc00 con 0x558b2a1f1620 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d8223a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d8223a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1f0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54622/0 (socket says 172.21.15.27:54622) 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d8223a640 1 -- 172.21.15.27:0/3383386405 learned_addr learned my addr 172.21.15.27:0/3383386405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.871+0000 7f4d81a39640 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 0x558b2a1ed4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d82a3b640 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d8223a640 1 -- 172.21.15.27:0/3383386405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 msgr2=0x558b2a1fcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d8223a640 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1fcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d8223a640 1 -- 172.21.15.27:0/3383386405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 msgr2=0x558b2a1ed4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d8223a640 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 0x558b2a1ed4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d8223a640 1 -- 172.21.15.27:0/3383386405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2a1d4090 con 0x558b2a1e6630 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d82a3b640 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1fcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d8223a640 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1f0ce0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009a50 tx=0x7f4d6c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e74658e1dda0bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d67fff640 1 -- 172.21.15.27:0/3383386405 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c049070 con 0x558b2a1e6630 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d67fff640 1 -- 172.21.15.27:0/3383386405 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d6c0d9070 con 0x558b2a1e6630 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d67fff640 1 -- 172.21.15.27:0/3383386405 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c0c4530 con 0x558b2a1e6630 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d871ac740 1 -- 172.21.15.27:0/3383386405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 msgr2=0x558b2a1f0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d871ac740 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1f0ce0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009a50 tx=0x7f4d6c003c80 comp rx=0 tx=0).stop 2024-04-25T01:27:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d871ac740 1 -- 172.21.15.27:0/3383386405 shutdown_connections 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d871ac740 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1fcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d871ac740 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6f50 0x558b2a1ed4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.872+0000 7f4d871ac740 1 --2- 172.21.15.27:0/3383386405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6630 0x558b2a1f0ce0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 -- 172.21.15.27:0/3383386405 >> 172.21.15.27:0/3383386405 conn(0x558b2a1e2950 msgr2=0x558b2a1e1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 -- 172.21.15.27:0/3383386405 shutdown_connections 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 -- 172.21.15.27:0/3383386405 wait complete. 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 -- start start 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 Processor -- start 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6630 0x558b2a1f9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6f50 0x558b2a1f4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1ad270 con 0x558b2a1e6f50 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1ae6e0 con 0x558b2a1e6630 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c1ad940 con 0x558b2a1f1620 2024-04-25T01:27:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d871ac740 10 client.0 _ll_register_callbacks cb 0x7f4d78009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d81a39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6f50 0x558b2a1f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d81a39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6f50 0x558b2a1f4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54624/0 (socket says 172.21.15.27:54624) 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d81a39640 1 -- 172.21.15.27:0/2944378161 learned_addr learned my addr 172.21.15.27:0/2944378161 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d8223a640 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6630 0x558b2a1f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d81a39640 1 -- 172.21.15.27:0/2944378161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 msgr2=0x558b2a1f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d81a39640 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d81a39640 1 -- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6630 msgr2=0x558b2a1f9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d81a39640 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6630 0x558b2a1f9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d81a39640 1 -- 172.21.15.27:0/2944378161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b2a1f5310 con 0x558b2a1e6f50 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.873+0000 7f4d8223a640 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6630 0x558b2a1f9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d81a39640 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6f50 0x558b2a1f4890 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009d20 tx=0x7f4d78012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee48606bea46e9c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d65ffb640 1 -- 172.21.15.27:0/2944378161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78004460 con 0x558b2a1e6f50 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d65ffb640 1 -- 172.21.15.27:0/2944378161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4d78005030 con 0x558b2a1e6f50 2024-04-25T01:27:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 1 -- 172.21.15.27:0/2944378161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d6c0d9070 con 0x558b2a1e6f50 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d65ffb640 1 -- 172.21.15.27:0/2944378161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4d7802f050 con 0x558b2a1e6f50 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 10 client.16203 fetch_fsmap learned FSMap version 26 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 1 -- 172.21.15.27:0/2944378161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558b2a2674d0 con 0x558b2a1e6f50 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d65ffb640 1 -- 172.21.15.27:0/2944378161 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d6c0d9070 con 0x558b2a1e6f50 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d65ffb640 1 -- 172.21.15.27:0/2944378161 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f4d78021150 con 0x558b2a1e6f50 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 10 client.16203 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 10 client.16203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 20 client.16203 populate_metadata read hostname 'smithi027' 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558b2a267cb0 0x558b2a26a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 4 client.16203 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.874+0000 7f4d871ac740 1 -- 172.21.15.27:0/2944378161 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558b2a0d06f0 con 0x558b2a267cb0 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d8223a640 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558b2a267cb0 0x558b2a26a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d65ffb640 10 client.16203 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d8223a640 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558b2a267cb0 0x558b2a26a0d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6319deae626f1631 server_cookie=2232dad4e8aea54d in_seq=0 out_seq=0 2024-04-25T01:27:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d65ffb640 1 -- 172.21.15.27:0/2944378161 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558b2a0d06f0 con 0x558b2a267cb0 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d65ffb640 10 client.16203 handle_command_reply: tid=0 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 client.16203 shutdown 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 -- 172.21.15.27:0/2944378161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6f50 msgr2=0x558b2a1f4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6f50 0x558b2a1f4890 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009d20 tx=0x7f4d78012820 comp rx=0 tx=0).stop 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 -- 172.21.15.27:0/2944378161 shutdown_connections 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558b2a267cb0 0x558b2a26a0d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b2a1f1620 0x558b2a1f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d65ffb640 0 client.16203 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b2a1e6f50 0x558b2a1f4890 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 --2- 172.21.15.27:0/2944378161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558b2a1e6630 0x558b2a1f9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 -- 172.21.15.27:0/2944378161 >> 172.21.15.27:0/2944378161 conn(0x558b2a1e2950 msgr2=0x558b2a1e7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 -- 172.21.15.27:0/2944378161 shutdown_connections 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 1 -- 172.21.15.27:0/2944378161 wait complete. 2024-04-25T01:27:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.875+0000 7f4d871ac740 20 client.16203 trim_cache size 0 max 16384 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout: "9ff6073b-5bd2-4468-9f51-2112f977ad0d": { 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "9ff6073b-5bd2-4468-9f51-2112f977ad0d", 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:27:42.861 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:27:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 -- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d00 msgr2=0x7f4d680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d00 0x7f4d680a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d74005cc0 tx=0x7f4d7400d710 comp rx=0 tx=0).stop 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 -- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c069dc0 msgr2=0x7f4d7c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c069dc0 0x7f4d7c197fc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009230 tx=0x7f4d6c004270 comp rx=0 tx=0).stop 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 -- 172.21.15.27:0/2285971661 shutdown_connections 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d7c107730 0x7f4d7c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7c06a780 0x7f4d7c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4d680a2d00 0x7f4d680a51c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 --2- 172.21.15.27:0/2285971661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d7c069dc0 0x7f4d7c197fc0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 -- 172.21.15.27:0/2285971661 >> 172.21.15.27:0/2285971661 conn(0x7f4d7c0776a0 msgr2=0x7f4d7c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 -- 172.21.15.27:0/2285971661 shutdown_connections 2024-04-25T01:27:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:27:42.876+0000 7f4d8449e640 1 -- 172.21.15.27:0/2285971661 wait complete. 2024-04-25T01:27:42.865 INFO:tasks.cephfs.filesystem:scrub status for tag:9ff6073b-5bd2-4468-9f51-2112f977ad0d - {'path': '/', 'tag': '9ff6073b-5bd2-4468-9f51-2112f977ad0d', 'options': 'recursive,force'} 2024-04-25T01:27:44.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:44 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:27:44.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:44 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:44 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:27:44.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:44 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:27:44.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:44 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:44 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:44 smithi027 ceph-mon[29197]: pgmap v72: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:44.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:44 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:27:44.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:44 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:44 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:27:44.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:44 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:27:44.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:44 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:44 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:44 smithi153 ceph-mon[32621]: pgmap v72: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:44.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:44 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:27:44.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:44 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:44 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:27:44.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:44 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:27:44.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:44 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:44 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:27:44.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:44 smithi139 ceph-mon[32760]: pgmap v72: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:46.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:46 smithi027 ceph-mon[29197]: pgmap v73: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:46.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:46 smithi153 ceph-mon[32621]: pgmap v73: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:46.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:46 smithi139 ceph-mon[32760]: pgmap v73: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:27:48.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:48 smithi027 ceph-mon[29197]: pgmap v74: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:48.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:48 smithi153 ceph-mon[32621]: pgmap v74: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:48.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:48 smithi139 ceph-mon[32760]: pgmap v74: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:27:49.250 INFO:tasks.workunit.client.0.smithi027.stdout:1/600: open fd = 3 2024-04-25T01:27:49.250 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.250 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.250 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.250 INFO:tasks.workunit.client.0.smithi027.stdout:2/600: open fd = 3 2024-04-25T01:27:49.250 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:3/600: open fd = 3 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:4/600: open fd = 3 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:5/600: open fd = 3 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.251 INFO:tasks.workunit.client.0.smithi027.stdout:6/600: open fd = 3 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:7/600: open fd = 3 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:8/600: open fd = 3 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:9/600: open fd = 3 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.252 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:10/600: open fd = 3 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:11/600: open fd = 3 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:12/600: open fd = 3 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:13/600: open fd = 3 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.253 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:14/600: open fd = 3 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:15/600: open fd = 3 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:16/600: open fd = 3 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:17/600: open fd = 3 2024-04-25T01:27:49.254 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:18/600: open fd = 3 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:19/600: open fd = 3 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:20/600: open fd = 3 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:21/600: open fd = 3 2024-04-25T01:27:49.255 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:22/600: open fd = 3 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:23/600: open fd = 3 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:24/600: open fd = 3 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.256 INFO:tasks.workunit.client.0.smithi027.stdout:25/600: open fd = 3 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:26/600: open fd = 3 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:27/600: open fd = 3 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:28/600: open fd = 3 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.257 INFO:tasks.workunit.client.0.smithi027.stdout:29/600: open fd = 3 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:30/600: open fd = 3 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:31/600: open fd = 3 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:32/600: open fd = 3 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.258 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:33/600: open fd = 3 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:34/600: open fd = 3 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:35/600: open fd = 3 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:36/600: open fd = 3 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.259 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:37/600: open fd = 3 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:38/600: open fd = 3 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:39/600: open fd = 3 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:40/600: open fd = 3 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.260 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:41/600: open fd = 3 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:42/600: open fd = 3 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:43/600: open fd = 3 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:44/600: open fd = 3 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.261 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:45/600: open fd = 3 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:46/600: open fd = 3 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:47/600: open fd = 3 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:48/600: open fd = 3 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.262 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:49/600: open fd = 3 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:50/600: open fd = 3 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:51/600: open fd = 3 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:52/600: open fd = 3 2024-04-25T01:27:49.263 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:53/600: open fd = 3 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:54/600: open fd = 3 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:55/600: open fd = 3 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:56/600: open fd = 3 2024-04-25T01:27:49.264 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:57/600: open fd = 3 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:58/600: open fd = 3 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:59/600: open fd = 3 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.265 INFO:tasks.workunit.client.0.smithi027.stdout:60/600: open fd = 3 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:61/600: open fd = 3 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:62/600: open fd = 3 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:63/600: open fd = 3 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:27:49.266 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:27:50.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:50 smithi027 ceph-mon[29197]: pgmap v75: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-25T01:27:50.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:50 smithi153 ceph-mon[32621]: pgmap v75: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-25T01:27:50.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:50 smithi139 ceph-mon[32760]: pgmap v75: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-25T01:27:52.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:52 smithi027 ceph-mon[29197]: pgmap v76: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:27:52.527 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:52 smithi153 ceph-mon[32621]: pgmap v76: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:27:52.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:52 smithi139 ceph-mon[32760]: pgmap v76: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:27:53.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:53 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:27:53.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:53 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:27:53.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:53 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:27:53.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:53 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:27:53.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:53 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:27:53.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:53 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:27:54.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:54 smithi027 ceph-mon[29197]: pgmap v77: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 9 op/s 2024-04-25T01:27:54.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:54 smithi153 ceph-mon[32621]: pgmap v77: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 9 op/s 2024-04-25T01:27:54.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:54 smithi139 ceph-mon[32760]: pgmap v77: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 9 op/s 2024-04-25T01:27:56.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:56 smithi027 ceph-mon[29197]: pgmap v78: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 9 op/s 2024-04-25T01:27:56.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:56 smithi153 ceph-mon[32621]: pgmap v78: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 9 op/s 2024-04-25T01:27:56.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:56 smithi139 ceph-mon[32760]: pgmap v78: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 9 op/s 2024-04-25T01:27:58.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:27:58 smithi027 ceph-mon[29197]: pgmap v79: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:27:58.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:27:58 smithi153 ceph-mon[32621]: pgmap v79: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:27:58.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:27:58 smithi139 ceph-mon[32760]: pgmap v79: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:28:00.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:00 smithi027 ceph-mon[29197]: pgmap v80: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:28:00.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:00 smithi153 ceph-mon[32621]: pgmap v80: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:28:00.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:00 smithi139 ceph-mon[32760]: pgmap v80: 129 pgs: 129 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 9 op/s 2024-04-25T01:28:02.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:02 smithi027 ceph-mon[29197]: pgmap v81: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 29 op/s 2024-04-25T01:28:02.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:02 smithi153 ceph-mon[32621]: pgmap v81: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 29 op/s 2024-04-25T01:28:02.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:02 smithi139 ceph-mon[32760]: pgmap v81: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 29 op/s 2024-04-25T01:28:04.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:04 smithi027 ceph-mon[29197]: pgmap v82: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:04.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:04 smithi153 ceph-mon[32621]: pgmap v82: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:04.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:04 smithi139 ceph-mon[32760]: pgmap v82: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:06.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:06 smithi027 ceph-mon[29197]: pgmap v83: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:06.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:06 smithi153 ceph-mon[32621]: pgmap v83: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:06.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:06 smithi139 ceph-mon[32760]: pgmap v83: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:08 smithi027 ceph-mon[29197]: pgmap v84: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:28:08.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:08 smithi153 ceph-mon[32621]: pgmap v84: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:28:08.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:08 smithi139 ceph-mon[32760]: pgmap v84: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:28:10.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:10 smithi027 ceph-mon[29197]: pgmap v85: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:10.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:10 smithi153 ceph-mon[32621]: pgmap v85: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:10.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:10 smithi139 ceph-mon[32760]: pgmap v85: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:28:12.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:12 smithi027 ceph-mon[29197]: pgmap v86: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-25T01:28:12.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:12 smithi153 ceph-mon[32621]: pgmap v86: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-25T01:28:12.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:12 smithi139 ceph-mon[32760]: pgmap v86: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-25T01:28:12.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:28:12.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:28:12.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 -- 172.21.15.27:0/264161640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 msgr2=0x7fb71410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:12.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 --2- 172.21.15.27:0/264161640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 0x7fb71410b7d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb700009900 tx=0x7fb70002a510 comp rx=0 tx=0).stop 2024-04-25T01:28:12.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 -- 172.21.15.27:0/264161640 shutdown_connections 2024-04-25T01:28:12.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 --2- 172.21.15.27:0/264161640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71410bd10 0x7fb71410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 --2- 172.21.15.27:0/264161640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 0x7fb71410b7d0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 --2- 172.21.15.27:0/264161640 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7141029d0 0x7fb714102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 -- 172.21.15.27:0/264161640 >> 172.21.15.27:0/264161640 conn(0x7fb7140769c0 msgr2=0x7fb714076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 -- 172.21.15.27:0/264161640 shutdown_connections 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.004+0000 7fb71b031640 1 -- 172.21.15.27:0/264161640 wait complete. 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb71b031640 1 Processor -- start 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb71b031640 1 -- start start 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb71b031640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7141029d0 0x7fb714197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb71b031640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 0x7fb714198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb71b031640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71410bd10 0x7fb71419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb71b031640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb714111b60 con 0x7fb7141032f0 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb71b031640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141119e0 con 0x7fb7141029d0 2024-04-25T01:28:12.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb71b031640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb714111ce0 con 0x7fb71410bd10 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb713fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 0x7fb714198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb713fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 0x7fb714198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34994/0 (socket says 172.21.15.27:34994) 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.005+0000 7fb713fff640 1 -- 172.21.15.27:0/2096941661 learned_addr learned my addr 172.21.15.27:0/2096941661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb7195ce640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71410bd10 0x7fb71419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb718dcd640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7141029d0 0x7fb714197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb713fff640 1 -- 172.21.15.27:0/2096941661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71410bd10 msgr2=0x7fb71419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb713fff640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71410bd10 0x7fb71419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb713fff640 1 -- 172.21.15.27:0/2096941661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7141029d0 msgr2=0x7fb714197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb713fff640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7141029d0 0x7fb714197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb713fff640 1 -- 172.21.15.27:0/2096941661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71419cfb0 con 0x7fb7141032f0 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb713fff640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 0x7fb714198500 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb700004770 tx=0x7fb700003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:12.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb711ffb640 1 -- 172.21.15.27:0/2096941661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70003f070 con 0x7fb7141032f0 2024-04-25T01:28:12.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb711ffb640 1 -- 172.21.15.27:0/2096941661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb700031ca0 con 0x7fb7141032f0 2024-04-25T01:28:12.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb71b031640 1 -- 172.21.15.27:0/2096941661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71419d240 con 0x7fb7141032f0 2024-04-25T01:28:12.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb711ffb640 1 -- 172.21.15.27:0/2096941661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700035030 con 0x7fb7141032f0 2024-04-25T01:28:12.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.006+0000 7fb71b031640 1 -- 172.21.15.27:0/2096941661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7141a4b30 con 0x7fb7141032f0 2024-04-25T01:28:12.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb711ffb640 1 -- 172.21.15.27:0/2096941661 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb700042070 con 0x7fb7141032f0 2024-04-25T01:28:12.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb71de1b740 1 Processor -- start 2024-04-25T01:28:12.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb71de1b740 1 -- start start 2024-04-25T01:28:12.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x557998985740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 0x5579989944f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998994a30 0x557998996e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb71de1b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5579988ea6b0 con 0x557998994a30 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb71de1b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557998892880 con 0x557998985360 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb71de1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5579988e6670 con 0x557998985d10 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.009+0000 7fb711ffb640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb7040a2d00 0x7fb7040a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998994a30 0x557998996e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb7195ce640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb7040a2d00 0x7fb7040a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998994a30 0x557998996e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35002/0 (socket says 172.21.15.27:35002) 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 -- 172.21.15.27:0/1180854074 learned_addr learned my addr 172.21.15.27:0/1180854074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb7195ce640 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x557998985740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb718dcd640 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 0x5579989944f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb711ffb640 1 -- 172.21.15.27:0/2096941661 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb7000f0020 con 0x7fb7141032f0 2024-04-25T01:28:12.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 -- 172.21.15.27:0/1180854074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 msgr2=0x5579989944f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 0x5579989944f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 -- 172.21.15.27:0/1180854074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 msgr2=0x557998985740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x557998985740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 -- 172.21.15.27:0/1180854074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579986c87d0 con 0x557998994a30 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb7195ce640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb7040a2d00 0x7fb7040a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb70c007fd0 tx=0x7fb70c007d60 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb713fff640 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998994a30 0x557998996e20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb7000bc140 tx=0x7fb70008f2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8af41e47164ba5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb6feffd640 1 -- 172.21.15.27:0/1180854074 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700049070 con 0x557998994a30 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb6feffd640 1 -- 172.21.15.27:0/1180854074 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7000bc1b0 con 0x557998994a30 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.010+0000 7fb6feffd640 1 -- 172.21.15.27:0/1180854074 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7000ed660 con 0x557998994a30 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 -- 172.21.15.27:0/1180854074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998994a30 msgr2=0x557998996e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998994a30 0x557998996e20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb7000bc140 tx=0x7fb70008f2e0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 -- 172.21.15.27:0/1180854074 shutdown_connections 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998994a30 0x557998996e20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 0x5579989944f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 --2- 172.21.15.27:0/1180854074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x557998985740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 -- 172.21.15.27:0/1180854074 >> 172.21.15.27:0/1180854074 conn(0x557998981950 msgr2=0x557998981d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 -- 172.21.15.27:0/1180854074 shutdown_connections 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 -- 172.21.15.27:0/1180854074 wait complete. 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.011+0000 7fb71de1b740 1 -- start start 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb71de1b740 1 Processor -- start 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x5579989919e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985d10 0x557998991f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 0x557998992460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb71de1b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5579989975b0 con 0x557998985360 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb71de1b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5579988a5f50 con 0x557998985d10 2024-04-25T01:28:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb71de1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557998997730 con 0x557998994a30 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb71de1b740 10 client.0 _ll_register_callbacks cb 0x5579988e1000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb718dcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x5579989919e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb718dcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x5579989919e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35004/0 (socket says 172.21.15.27:35004) 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb718dcd640 1 -- 172.21.15.27:0/3707310052 learned_addr learned my addr 172.21.15.27:0/3707310052 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb7195ce640 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 0x557998992460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb713fff640 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985d10 0x557998991f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb718dcd640 1 -- 172.21.15.27:0/3707310052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 msgr2=0x557998992460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb718dcd640 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 0x557998992460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb718dcd640 1 -- 172.21.15.27:0/3707310052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985d10 msgr2=0x557998991f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb718dcd640 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985d10 0x557998991f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.012+0000 7fb718dcd640 1 -- 172.21.15.27:0/3707310052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579985fe3f0 con 0x557998985360 2024-04-25T01:28:12.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb718dcd640 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x5579989919e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb70800da10 tx=0x7fb70800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28166ed3116c1fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb6f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3707310052 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70801b440 con 0x557998985360 2024-04-25T01:28:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3707310052 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb70801b5e0 con 0x557998985360 2024-04-25T01:28:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 1 -- 172.21.15.27:0/3707310052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7000bc1b0 con 0x557998985360 2024-04-25T01:28:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3707310052 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb70801b7e0 con 0x557998985360 2024-04-25T01:28:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 10 client.16215 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 1 -- 172.21.15.27:0/3707310052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5579985be0f0 con 0x557998985360 2024-04-25T01:28:13.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3707310052 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7000bc1b0 con 0x557998985360 2024-04-25T01:28:13.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3707310052 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fb708019020 con 0x557998985360 2024-04-25T01:28:13.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 10 client.16215 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:13.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:13.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 20 client.16215 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:13.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5579989a4b90 0x5579989a7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 4 client.16215 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.013+0000 7fb71de1b740 1 -- 172.21.15.27:0/3707310052 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579988f7bf0 con 0x5579989a4b90 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.014+0000 7fb713fff640 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5579989a4b90 0x5579989a7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.014+0000 7fb713fff640 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5579989a4b90 0x5579989a7030 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8aef6d8cae46ea2b server_cookie=98a7fcca1706c0 in_seq=0 out_seq=0 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.014+0000 7fb6f3fff640 10 client.16215 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3707310052 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5579988f7bf0 con 0x5579989a4b90 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb6f3fff640 10 client.16215 handle_command_reply: tid=0 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 client.16215 shutdown 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 -- 172.21.15.27:0/3707310052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 msgr2=0x5579989919e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x5579989919e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb70800da10 tx=0x7fb70800de10 comp rx=0 tx=0).stop 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 -- 172.21.15.27:0/3707310052 shutdown_connections 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5579989a4b90 0x5579989a7030 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 0x557998992460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985d10 0x557998991f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb6f3fff640 0 client.16215 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3707310052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x5579989919e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 -- 172.21.15.27:0/3707310052 >> 172.21.15.27:0/3707310052 conn(0x557998981950 msgr2=0x5579989877a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 -- 172.21.15.27:0/3707310052 shutdown_connections 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.015+0000 7fb71de1b740 1 -- 172.21.15.27:0/3707310052 wait complete. 2024-04-25T01:28:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.016+0000 7fb71de1b740 20 client.16215 trim_cache size 0 max 16384 2024-04-25T01:28:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.033+0000 7fb71de1b740 1 Processor -- start 2024-04-25T01:28:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb71de1b740 1 -- start start 2024-04-25T01:28:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x5579989a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985d10 0x557998994110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 0x557998980970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb71de1b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141af2b0 con 0x557998985d10 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb71de1b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141acb80 con 0x557998985360 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb71de1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141aa9d0 con 0x557998994a30 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985d10 0x557998994110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985d10 0x557998994110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35016/0 (socket says 172.21.15.27:35016) 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 -- 172.21.15.27:0/2686677122 learned_addr learned my addr 172.21.15.27:0/2686677122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb718dcd640 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x5579989a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 -- 172.21.15.27:0/2686677122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 msgr2=0x557998980970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 0x557998980970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 -- 172.21.15.27:0/2686677122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 msgr2=0x5579989a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x5579989a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 -- 172.21.15.27:0/2686677122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557998973090 con 0x557998985d10 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb718dcd640 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x5579989a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb713fff640 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985d10 0x557998994110 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb7000d32e0 tx=0x7fb700009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448a3dad8d86e98e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.034+0000 7fb6fe7fc640 1 -- 172.21.15.27:0/2686677122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700049040 con 0x557998985d10 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb6fe7fc640 1 -- 172.21.15.27:0/2686677122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7000ba250 con 0x557998985d10 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb6fe7fc640 1 -- 172.21.15.27:0/2686677122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7000b8820 con 0x557998985d10 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 -- 172.21.15.27:0/2686677122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985d10 msgr2=0x557998994110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985d10 0x557998994110 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb7000d32e0 tx=0x7fb700009260 comp rx=0 tx=0).stop 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 -- 172.21.15.27:0/2686677122 shutdown_connections 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998994a30 0x557998980970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985d10 0x557998994110 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 --2- 172.21.15.27:0/2686677122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998985360 0x5579989a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 -- 172.21.15.27:0/2686677122 >> 172.21.15.27:0/2686677122 conn(0x557998981950 msgr2=0x557998980260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 -- 172.21.15.27:0/2686677122 shutdown_connections 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 -- 172.21.15.27:0/2686677122 wait complete. 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.035+0000 7fb71de1b740 1 -- start start 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb71de1b740 1 Processor -- start 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x55799899cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 0x55799898e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb71de1b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998994a30 0x55799898e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb71de1b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141ab040 con 0x557998985360 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb71de1b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141ac4b0 con 0x557998994a30 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb71de1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141ab710 con 0x557998985d10 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb71de1b740 10 client.0 _ll_register_callbacks cb 0x7fb708004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb718dcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x55799899cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb718dcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x55799899cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35018/0 (socket says 172.21.15.27:35018) 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb718dcd640 1 -- 172.21.15.27:0/3917783455 learned_addr learned my addr 172.21.15.27:0/3917783455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb713fff640 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 0x55799898e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb7195ce640 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998994a30 0x55799898e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb718dcd640 1 -- 172.21.15.27:0/3917783455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 msgr2=0x55799898e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb718dcd640 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 0x55799898e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb718dcd640 1 -- 172.21.15.27:0/3917783455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998994a30 msgr2=0x55799898e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb718dcd640 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998994a30 0x55799898e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.036+0000 7fb718dcd640 1 -- 172.21.15.27:0/3917783455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55799898eef0 con 0x557998985360 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb718dcd640 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x55799899cc50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb708005710 tx=0x7fb708003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880a5e2c213213ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb6f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3917783455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70801b980 con 0x557998985360 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3917783455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb70801bb20 con 0x557998985360 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 1 -- 172.21.15.27:0/3917783455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7000ba250 con 0x557998985360 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3917783455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb70801bd60 con 0x557998985360 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 10 client.16221 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 1 -- 172.21.15.27:0/3917783455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557998a0b0c0 con 0x557998985360 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3917783455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7000ba250 con 0x557998985360 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3917783455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fb708019040 con 0x557998985360 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 10 client.16221 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 20 client.16221 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557998a0b890 0x557998a0dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 4 client.16221 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.037+0000 7fb71de1b740 1 -- 172.21.15.27:0/3917783455 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5579988f1f40 con 0x557998a0b890 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb713fff640 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557998a0b890 0x557998a0dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb713fff640 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557998a0b890 0x557998a0dcb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7a68d34506a7ed7 server_cookie=8892ba7c8fd9f51f in_seq=0 out_seq=0 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb6f3fff640 10 client.16221 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb6f3fff640 1 -- 172.21.15.27:0/3917783455 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5579988f1f40 con 0x557998a0b890 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb6f3fff640 10 client.16221 handle_command_reply: tid=0 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 client.16221 shutdown 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 -- 172.21.15.27:0/3917783455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 msgr2=0x55799899cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x55799899cc50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb708005710 tx=0x7fb708003b00 comp rx=0 tx=0).stop 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 -- 172.21.15.27:0/3917783455 shutdown_connections 2024-04-25T01:28:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557998a0b890 0x557998a0dcb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557998994a30 0x55799898e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb6f3fff640 0 client.16221 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557998985d10 0x55799898e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 --2- 172.21.15.27:0/3917783455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557998985360 0x55799899cc50 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 -- 172.21.15.27:0/3917783455 >> 172.21.15.27:0/3917783455 conn(0x557998981950 msgr2=0x557998980af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 -- 172.21.15.27:0/3917783455 shutdown_connections 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.038+0000 7fb71de1b740 1 -- 172.21.15.27:0/3917783455 wait complete. 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71de1b740 20 client.16221 trim_cache size 0 max 16384 2024-04-25T01:28:13.024 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:28:13.025 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:28:13.025 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:28:13.025 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:28:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 -- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb7040a2d00 msgr2=0x7fb7040a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb7040a2d00 0x7fb7040a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb70c007fd0 tx=0x7fb70c007d60 comp rx=0 tx=0).stop 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 -- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 msgr2=0x7fb714198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 0x7fb714198500 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb700004770 tx=0x7fb700003c10 comp rx=0 tx=0).stop 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 -- 172.21.15.27:0/2096941661 shutdown_connections 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb71410bd10 0x7fb71419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb7040a2d00 0x7fb7040a51c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7141032f0 0x7fb714198500 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 --2- 172.21.15.27:0/2096941661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7141029d0 0x7fb714197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 -- 172.21.15.27:0/2096941661 >> 172.21.15.27:0/2096941661 conn(0x7fb7140769c0 msgr2=0x7fb7140fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 -- 172.21.15.27:0/2096941661 shutdown_connections 2024-04-25T01:28:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.039+0000 7fb71b031640 1 -- 172.21.15.27:0/2096941661 wait complete. 2024-04-25T01:28:13.029 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:28:13.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:28:13.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 -- 172.21.15.27:0/1871080704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 msgr2=0x7f25ec101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/1871080704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 0x7f25ec101c20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009900 tx=0x7f25d802a480 comp rx=0 tx=0).stop 2024-04-25T01:28:13.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 -- 172.21.15.27:0/1871080704 shutdown_connections 2024-04-25T01:28:13.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/1871080704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25ec102160 0x7f25ec10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/1871080704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 0x7f25ec101c20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/1871080704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25ec107750 0x7f25ec107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 -- 172.21.15.27:0/1871080704 >> 172.21.15.27:0/1871080704 conn(0x7f25ec0788d0 msgr2=0x7f25ec078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 -- 172.21.15.27:0/1871080704 shutdown_connections 2024-04-25T01:28:13.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.153+0000 7f25f2b9e640 1 -- 172.21.15.27:0/1871080704 wait complete. 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f2b9e640 1 Processor -- start 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f2b9e640 1 -- start start 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f2b9e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 0x7f25ec19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f2b9e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25ec102160 0x7f25ec19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f2b9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25ec107750 0x7f25ec19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f2b9e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec111b10 con 0x7f25ec1017c0 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f2b9e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec111990 con 0x7f25ec102160 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f2b9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec111c90 con 0x7f25ec107750 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f093a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 0x7f25ec19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f093a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 0x7f25ec19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35034/0 (socket says 172.21.15.27:35034) 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f093a640 1 -- 172.21.15.27:0/903667539 learned_addr learned my addr 172.21.15.27:0/903667539 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25ebfff640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25ec102160 0x7f25ec19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.154+0000 7f25f113b640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25ec107750 0x7f25ec19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25f093a640 1 -- 172.21.15.27:0/903667539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25ec107750 msgr2=0x7f25ec19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25f093a640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25ec107750 0x7f25ec19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25f093a640 1 -- 172.21.15.27:0/903667539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25ec102160 msgr2=0x7f25ec19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25f093a640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25ec102160 0x7f25ec19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25f093a640 1 -- 172.21.15.27:0/903667539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ec19f2a0 con 0x7f25ec1017c0 2024-04-25T01:28:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25f093a640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 0x7f25ec19a2b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f25e000d950 tx=0x7f25e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25e9ffb640 1 -- 172.21.15.27:0/903667539 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0017030 con 0x7f25ec1017c0 2024-04-25T01:28:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25ec19f590 con 0x7f25ec1017c0 2024-04-25T01:28:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25e9ffb640 1 -- 172.21.15.27:0/903667539 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25e0004020 con 0x7f25ec1017c0 2024-04-25T01:28:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25e9ffb640 1 -- 172.21.15.27:0/903667539 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e00047a0 con 0x7f25ec1017c0 2024-04-25T01:28:13.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.155+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25ec1a6e20 con 0x7f25ec1017c0 2024-04-25T01:28:13.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.156+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25ec0689b0 con 0x7f25ec1017c0 2024-04-25T01:28:13.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.157+0000 7f25e9ffb640 1 -- 172.21.15.27:0/903667539 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f25e00041d0 con 0x7f25ec1017c0 2024-04-25T01:28:13.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.158+0000 7f25e9ffb640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f25dc0a2d00 0x7f25dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.158+0000 7f25ebfff640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f25dc0a2d00 0x7f25dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.158+0000 7f25e9ffb640 1 -- 172.21.15.27:0/903667539 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f25e00cf080 con 0x7f25ec1017c0 2024-04-25T01:28:13.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.158+0000 7f25ebfff640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f25dc0a2d00 0x7f25dc0a51c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009230 tx=0x7f25d80058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.162+0000 7f25e9ffb640 1 -- 172.21.15.27:0/903667539 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f25e009b910 con 0x7f25ec1017c0 2024-04-25T01:28:13.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.387+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25ec1a6ff0 con 0x7f25ec1017c0 2024-04-25T01:28:13.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.388+0000 7f25e9ffb640 1 -- 172.21.15.27:0/903667539 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11883 (secure 0 0 0) 0x7f25e009b210 con 0x7f25ec1017c0 2024-04-25T01:28:13.367 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:28:13.367 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:26:47.433054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24517,24521,24505]},"id":1}]} 2024-04-25T01:28:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-04-25T01:28:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f25dc0a2d00 msgr2=0x7f25dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f25dc0a2d00 0x7f25dc0a51c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009230 tx=0x7f25d80058d0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 msgr2=0x7f25ec19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 0x7f25ec19a2b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f25e000d950 tx=0x7f25e000de10 comp rx=0 tx=0).stop 2024-04-25T01:28:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 shutdown_connections 2024-04-25T01:28:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25ec107750 0x7f25ec19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25ec102160 0x7f25ec19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f25dc0a2d00 0x7f25dc0a51c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 --2- 172.21.15.27:0/903667539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec1017c0 0x7f25ec19a2b0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 >> 172.21.15.27:0/903667539 conn(0x7f25ec0788d0 msgr2=0x7f25ec10cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 shutdown_connections 2024-04-25T01:28:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.390+0000 7f25f2b9e640 1 -- 172.21.15.27:0/903667539 wait complete. 2024-04-25T01:28:13.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:28:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.505+0000 7f41e562a640 1 -- 172.21.15.27:0/2735961928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 msgr2=0x7f41e0107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.505+0000 7f41e562a640 1 --2- 172.21.15.27:0/2735961928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 0x7f41e0107b10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009930 tx=0x7f41d802a420 comp rx=0 tx=0).stop 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.505+0000 7f41e562a640 1 -- 172.21.15.27:0/2735961928 shutdown_connections 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.505+0000 7f41e562a640 1 --2- 172.21.15.27:0/2735961928 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41e006a780 0x7f41e010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.505+0000 7f41e562a640 1 --2- 172.21.15.27:0/2735961928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41e0069dc0 0x7f41e006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.505+0000 7f41e562a640 1 --2- 172.21.15.27:0/2735961928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 0x7f41e0107b10 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.505+0000 7f41e562a640 1 -- 172.21.15.27:0/2735961928 >> 172.21.15.27:0/2735961928 conn(0x7f41e00776a0 msgr2=0x7f41e0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 -- 172.21.15.27:0/2735961928 shutdown_connections 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 -- 172.21.15.27:0/2735961928 wait complete. 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 Processor -- start 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 -- start start 2024-04-25T01:28:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41e0069dc0 0x7f41e0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41e006a780 0x7f41e0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 0x7f41e019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41e0111b50 con 0x7f41e0107730 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41e01119d0 con 0x7f41e006a780 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.506+0000 7f41e562a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41e0111cd0 con 0x7f41e0069dc0 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 0x7f41e019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 0x7f41e019c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35048/0 (socket says 172.21.15.27:35048) 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 -- 172.21.15.27:0/3931069851 learned_addr learned my addr 172.21.15.27:0/3931069851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41e006a780 0x7f41e0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41deffd640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41e0069dc0 0x7f41e0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 -- 172.21.15.27:0/3931069851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41e0069dc0 msgr2=0x7f41e0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41e0069dc0 0x7f41e0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 -- 172.21.15.27:0/3931069851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41e006a780 msgr2=0x7f41e0198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41e006a780 0x7f41e0198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 -- 172.21.15.27:0/3931069851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e019cfe0 con 0x7f41e0107730 2024-04-25T01:28:13.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41e006a780 0x7f41e0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:13.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41df7fe640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 0x7f41e019c8b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f41d000eaa0 tx=0x7f41d000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41c7fff640 1 -- 172.21.15.27:0/3931069851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d000cdb0 con 0x7f41e0107730 2024-04-25T01:28:13.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41e562a640 1 -- 172.21.15.27:0/3931069851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e019d2d0 con 0x7f41e0107730 2024-04-25T01:28:13.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41c7fff640 1 -- 172.21.15.27:0/3931069851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d0005020 con 0x7f41e0107730 2024-04-25T01:28:13.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41c7fff640 1 -- 172.21.15.27:0/3931069851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d00116b0 con 0x7f41e0107730 2024-04-25T01:28:13.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.507+0000 7f41e562a640 1 -- 172.21.15.27:0/3931069851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e01a4b60 con 0x7f41e0107730 2024-04-25T01:28:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.509+0000 7f41c7fff640 1 -- 172.21.15.27:0/3931069851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f41d0005560 con 0x7f41e0107730 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41e8414740 1 Processor -- start 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41e8414740 1 -- start start 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41e8414740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41e8414740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41e8414740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 0x560763db6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41e8414740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560763d0a670 con 0x560763da5460 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41e8414740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560763cb2970 con 0x560763db4980 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41e8414740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560763d06630 con 0x560763da5d80 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35054/0 (socket says 172.21.15.27:35054) 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 -- 172.21.15.27:0/277131842 learned_addr learned my addr 172.21.15.27:0/277131842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41c7fff640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41c80a2d00 0x7f41c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41deffd640 1 --2- 172.21.15.27:0/277131842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 -- 172.21.15.27:0/277131842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 msgr2=0x560763db4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 --2- 172.21.15.27:0/277131842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 -- 172.21.15.27:0/277131842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 msgr2=0x560763db6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 --2- 172.21.15.27:0/277131842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 0x560763db6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 -- 172.21.15.27:0/277131842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560763cd6400 con 0x560763da5460 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41df7fe640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41c80a2d00 0x7f41c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41c7fff640 1 -- 172.21.15.27:0/3931069851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f41d00cfaa0 con 0x7f41e0107730 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.510+0000 7f41de7fc640 1 --2- 172.21.15.27:0/277131842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da5840 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f41d4008cf0 tx=0x7f41d4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab4dd0685650415 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41c57fa640 1 -- 172.21.15.27:0/277131842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4016070 con 0x560763da5460 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41c57fa640 1 -- 172.21.15.27:0/277131842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d4005700 con 0x560763da5460 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41c57fa640 1 -- 172.21.15.27:0/277131842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4012790 con 0x560763da5460 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 -- 172.21.15.27:0/277131842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 msgr2=0x560763da5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 --2- 172.21.15.27:0/277131842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da5840 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f41d4008cf0 tx=0x7f41d4010740 comp rx=0 tx=0).stop 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41df7fe640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41c80a2d00 0x7f41c80a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f41d0098600 tx=0x7f41d00054d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 -- 172.21.15.27:0/277131842 shutdown_connections 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 --2- 172.21.15.27:0/277131842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 0x560763db6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 --2- 172.21.15.27:0/277131842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 --2- 172.21.15.27:0/277131842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da5840 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 -- 172.21.15.27:0/277131842 >> 172.21.15.27:0/277131842 conn(0x560763da18a0 msgr2=0x560763da1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 -- 172.21.15.27:0/277131842 shutdown_connections 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.511+0000 7f41e8414740 1 -- 172.21.15.27:0/277131842 wait complete. 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 1 -- start start 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 1 Processor -- start 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763db4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 0x560763db1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560763db77c0 con 0x560763da5460 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560763cc6050 con 0x560763db4980 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560763db7940 con 0x560763da5d80 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41e8414740 10 client.0 _ll_register_callbacks cb 0x560763a1dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763db4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763db4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35068/0 (socket says 172.21.15.27:35068) 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 -- 172.21.15.27:0/3574878116 learned_addr learned my addr 172.21.15.27:0/3574878116 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 0x560763db1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41deffd640 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 -- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 msgr2=0x560763db1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 -- 172.21.15.27:0/3574878116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 msgr2=0x560763db1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 0x560763db1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41de7fc640 1 -- 172.21.15.27:0/3574878116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560763c97dc0 con 0x560763da5460 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.512+0000 7f41deffd640 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763db4220 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f41d4005e10 tx=0x7f41d40085a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea83125edc60ead server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3574878116 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4016040 con 0x560763da5460 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3574878116 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f41d4012dc0 con 0x560763da5460 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 1 -- 172.21.15.27:0/3574878116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41d4005700 con 0x560763da5460 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3574878116 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f41d4032050 con 0x560763da5460 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 10 client.16239 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 1 -- 172.21.15.27:0/3574878116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560763a1e3f0 con 0x560763da5460 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3574878116 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d4005700 con 0x560763da5460 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3574878116 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f41d401aa00 con 0x560763da5460 2024-04-25T01:28:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 10 client.16239 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 10 client.16239 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 20 client.16239 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560763dc4930 0x560763dc6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 4 client.16239 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.513+0000 7f41e8414740 1 -- 172.21.15.27:0/3574878116 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560763d17bb0 con 0x560763dc4930 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.514+0000 7f41deffd640 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560763dc4930 0x560763dc6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.514+0000 7f41deffd640 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560763dc4930 0x560763dc6d20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a147ec906cddd81 server_cookie=22e4b0169c371118 in_seq=0 out_seq=0 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.514+0000 7f41c2ff5640 10 client.16239 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3574878116 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560763d17bb0 con 0x560763dc4930 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41c2ff5640 10 client.16239 handle_command_reply: tid=0 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 client.16239 shutdown 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 -- 172.21.15.27:0/3574878116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 msgr2=0x560763db4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763db4220 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f41d4005e10 tx=0x7f41d40085a0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 -- 172.21.15.27:0/3574878116 shutdown_connections 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560763dc4930 0x560763dc6d20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763db4980 0x560763db1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763db1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 --2- 172.21.15.27:0/3574878116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763db4220 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41e8414740 1 -- 172.21.15.27:0/3574878116 >> 172.21.15.27:0/3574878116 conn(0x560763da18a0 msgr2=0x560763da7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.515+0000 7f41c2ff5640 0 client.16239 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.516+0000 7f41e8414740 1 -- 172.21.15.27:0/3574878116 shutdown_connections 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.516+0000 7f41e8414740 1 -- 172.21.15.27:0/3574878116 wait complete. 2024-04-25T01:28:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.516+0000 7f41e8414740 20 client.16239 trim_cache size 0 max 16384 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41e8414740 1 Processor -- start 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41e8414740 1 -- start start 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41e8414740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da18a0 0x560763db3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41e8414740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5460 0x560763db0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41e8414740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da5d80 0x560763dcc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41e8414740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41e01af250 con 0x560763da18a0 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41e8414740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41e01acb20 con 0x560763da5d80 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41e8414740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41e01aa970 con 0x560763da5460 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da18a0 0x560763db3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da18a0 0x560763db3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35084/0 (socket says 172.21.15.27:35084) 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.529+0000 7f41de7fc640 1 -- 172.21.15.27:0/818915847 learned_addr learned my addr 172.21.15.27:0/818915847 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41de7fc640 1 -- 172.21.15.27:0/818915847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5460 msgr2=0x560763db0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41de7fc640 1 --2- 172.21.15.27:0/818915847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5460 0x560763db0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41de7fc640 1 -- 172.21.15.27:0/818915847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da5d80 msgr2=0x560763dcc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41de7fc640 1 --2- 172.21.15.27:0/818915847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da5d80 0x560763dcc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41de7fc640 1 -- 172.21.15.27:0/818915847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560763ae87d0 con 0x560763da18a0 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41de7fc640 1 --2- 172.21.15.27:0/818915847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da18a0 0x560763db3d50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f41d4010780 tx=0x7f41d4012780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec53ad032201d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41c4ff9640 1 -- 172.21.15.27:0/818915847 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4016040 con 0x560763da18a0 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41c4ff9640 1 -- 172.21.15.27:0/818915847 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d4026040 con 0x560763da18a0 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.530+0000 7f41c4ff9640 1 -- 172.21.15.27:0/818915847 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d400ee80 con 0x560763da18a0 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- 172.21.15.27:0/818915847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da18a0 msgr2=0x560763db3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 --2- 172.21.15.27:0/818915847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da18a0 0x560763db3d50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f41d4010780 tx=0x7f41d4012780 comp rx=0 tx=0).stop 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- 172.21.15.27:0/818915847 shutdown_connections 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 --2- 172.21.15.27:0/818915847 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da5d80 0x560763dcc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 --2- 172.21.15.27:0/818915847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5460 0x560763db0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 --2- 172.21.15.27:0/818915847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da18a0 0x560763db3d50 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- 172.21.15.27:0/818915847 >> 172.21.15.27:0/818915847 conn(0x560763db1bb0 msgr2=0x560763da75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- 172.21.15.27:0/818915847 shutdown_connections 2024-04-25T01:28:13.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- 172.21.15.27:0/818915847 wait complete. 2024-04-25T01:28:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- start start 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 Processor -- start 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da18a0 0x560763da8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763dbbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41e01aafe0 con 0x560763da5460 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f41e01ac450 con 0x560763da18a0 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41e01ab6b0 con 0x560763da5d80 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41e8414740 10 client.0 _ll_register_callbacks cb 0x7f41d4008bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35094/0 (socket says 172.21.15.27:35094) 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41deffd640 1 -- 172.21.15.27:0/3961203296 learned_addr learned my addr 172.21.15.27:0/3961203296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.531+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da18a0 0x560763da8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763dbbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41deffd640 1 -- 172.21.15.27:0/3961203296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 msgr2=0x560763dbbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41deffd640 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763dbbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41deffd640 1 -- 172.21.15.27:0/3961203296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da18a0 msgr2=0x560763da8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41deffd640 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da18a0 0x560763da8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41deffd640 1 -- 172.21.15.27:0/3961203296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560763da9470 con 0x560763da5460 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da18a0 0x560763da8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41deffd640 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da8ba0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f41d8000c00 tx=0x7f41d8004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bed14e4e44b8af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3961203296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8046020 con 0x560763da5460 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3961203296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f41d8042d60 con 0x560763da5460 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41e8414740 1 -- 172.21.15.27:0/3961203296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41d4026040 con 0x560763da5460 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3961203296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f41d8051050 con 0x560763da5460 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41e8414740 10 client.16245 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41e8414740 1 -- 172.21.15.27:0/3961203296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560763dbd280 con 0x560763da5460 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3961203296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d4026040 con 0x560763da5460 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3961203296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f41d8042700 con 0x560763da5460 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41e8414740 10 client.16245 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.532+0000 7f41e8414740 10 client.16245 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 20 client.16245 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560763db4980 0x560763e31740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 4 client.16245 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:28:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 -- 172.21.15.27:0/3961203296 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560763db7310 con 0x560763db4980 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560763db4980 0x560763e31740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41de7fc640 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560763db4980 0x560763e31740 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16ca085178a87516 server_cookie=4553240d157c6077 in_seq=0 out_seq=0 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41c2ff5640 10 client.16245 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41c2ff5640 1 -- 172.21.15.27:0/3961203296 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560763db7310 con 0x560763db4980 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41c2ff5640 10 client.16245 handle_command_reply: tid=0 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 client.16245 shutdown 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 -- 172.21.15.27:0/3961203296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 msgr2=0x560763da8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da8ba0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f41d8000c00 tx=0x7f41d8004340 comp rx=0 tx=0).stop 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 -- 172.21.15.27:0/3961203296 shutdown_connections 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x560763db4980 0x560763e31740 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560763da5d80 0x560763dbbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41c2ff5640 0 client.16245 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560763da5460 0x560763da8ba0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 --2- 172.21.15.27:0/3961203296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560763da18a0 0x560763da8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 -- 172.21.15.27:0/3961203296 >> 172.21.15.27:0/3961203296 conn(0x560763db1bb0 msgr2=0x560763da02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.533+0000 7f41e8414740 1 -- 172.21.15.27:0/3961203296 shutdown_connections 2024-04-25T01:28:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e8414740 1 -- 172.21.15.27:0/3961203296 wait complete. 2024-04-25T01:28:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e8414740 20 client.16245 trim_cache size 0 max 16384 2024-04-25T01:28:13.522 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 -- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41c80a2d00 msgr2=0x7f41c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41c80a2d00 0x7f41c80a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f41d0098600 tx=0x7f41d00054d0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 -- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 msgr2=0x7f41e019c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 0x7f41e019c8b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f41d000eaa0 tx=0x7f41d000ef60 comp rx=0 tx=0).stop 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 -- 172.21.15.27:0/3931069851 shutdown_connections 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f41c80a2d00 0x7f41c80a51c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e0107730 0x7f41e019c8b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f41e006a780 0x7f41e0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 --2- 172.21.15.27:0/3931069851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41e0069dc0 0x7f41e0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 -- 172.21.15.27:0/3931069851 >> 172.21.15.27:0/3931069851 conn(0x7f41e00776a0 msgr2=0x7f41e00fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 -- 172.21.15.27:0/3931069851 shutdown_connections 2024-04-25T01:28:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.534+0000 7f41e562a640 1 -- 172.21.15.27:0/3931069851 wait complete. 2024-04-25T01:28:13.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 -- 172.21.15.27:0/2645540054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450107730 msgr2=0x7f2450107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 --2- 172.21.15.27:0/2645540054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450107730 0x7f2450107b10 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f244c009930 tx=0x7f244c02a420 comp rx=0 tx=0).stop 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 -- 172.21.15.27:0/2645540054 shutdown_connections 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 --2- 172.21.15.27:0/2645540054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f245006a780 0x7f245010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 --2- 172.21.15.27:0/2645540054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2450069dc0 0x7f245006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 --2- 172.21.15.27:0/2645540054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450107730 0x7f2450107b10 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 -- 172.21.15.27:0/2645540054 >> 172.21.15.27:0/2645540054 conn(0x7f24500776a0 msgr2=0x7f2450077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 -- 172.21.15.27:0/2645540054 shutdown_connections 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.650+0000 7f245759e640 1 -- 172.21.15.27:0/2645540054 wait complete. 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245759e640 1 Processor -- start 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245759e640 1 -- start start 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450069dc0 0x7f2450100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245759e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f245006a780 0x7f2450100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245759e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2450107730 0x7f24501010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245759e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2450111b40 con 0x7f2450069dc0 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245759e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f24501119c0 con 0x7f245006a780 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245759e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2450111cc0 con 0x7f2450107730 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245533a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450069dc0 0x7f2450100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245533a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450069dc0 0x7f2450100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35106/0 (socket says 172.21.15.27:35106) 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245533a640 1 -- 172.21.15.27:0/749264939 learned_addr learned my addr 172.21.15.27:0/749264939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f2454b39640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f245006a780 0x7f2450100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f2455b3b640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2450107730 0x7f24501010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245533a640 1 -- 172.21.15.27:0/749264939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2450107730 msgr2=0x7f24501010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245533a640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2450107730 0x7f24501010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245533a640 1 -- 172.21.15.27:0/749264939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f245006a780 msgr2=0x7f2450100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.651+0000 7f245533a640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f245006a780 0x7f2450100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.652+0000 7f245533a640 1 -- 172.21.15.27:0/749264939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24501a65a0 con 0x7f2450069dc0 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.652+0000 7f245533a640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450069dc0 0x7f2450100670 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f244c009230 tx=0x7f244c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.652+0000 7f24467fc640 1 -- 172.21.15.27:0/749264939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c03f030 con 0x7f2450069dc0 2024-04-25T01:28:13.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.652+0000 7f245759e640 1 -- 172.21.15.27:0/749264939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24501a6830 con 0x7f2450069dc0 2024-04-25T01:28:13.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.652+0000 7f24467fc640 1 -- 172.21.15.27:0/749264939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f244c003ad0 con 0x7f2450069dc0 2024-04-25T01:28:13.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.652+0000 7f24467fc640 1 -- 172.21.15.27:0/749264939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c035070 con 0x7f2450069dc0 2024-04-25T01:28:13.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.652+0000 7f245759e640 1 -- 172.21.15.27:0/749264939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24501a6d70 con 0x7f2450069dc0 2024-04-25T01:28:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.654+0000 7f24467fc640 1 -- 172.21.15.27:0/749264939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f244c042030 con 0x7f2450069dc0 2024-04-25T01:28:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245a389740 1 Processor -- start 2024-04-25T01:28:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245a389740 1 -- start start 2024-04-25T01:28:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f24467fc640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f243c0a2d00 0x7f243c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245a389740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43ba960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245a389740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 0x5596f43c5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245a389740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43c5570 0x5596f43c7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245a389740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5596f431f670 con 0x5596f43ba580 2024-04-25T01:28:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245a389740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5596f42c7970 con 0x5596f43c5570 2024-04-25T01:28:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245a389740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5596f431b630 con 0x5596f43baea0 2024-04-25T01:28:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f2455b3b640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f243c0a2d00 0x7f243c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f24467fc640 1 -- 172.21.15.27:0/749264939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f244c0efb70 con 0x7f2450069dc0 2024-04-25T01:28:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245533a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43ba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245533a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43ba960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35116/0 (socket says 172.21.15.27:35116) 2024-04-25T01:28:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245533a640 1 -- 172.21.15.27:0/1281201592 learned_addr learned my addr 172.21.15.27:0/1281201592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f2455b3b640 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43c5570 0x5596f43c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f2454b39640 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 0x5596f43c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245533a640 1 -- 172.21.15.27:0/1281201592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 msgr2=0x5596f43c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245533a640 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 0x5596f43c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245533a640 1 -- 172.21.15.27:0/1281201592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43c5570 msgr2=0x5596f43c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245533a640 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43c5570 0x5596f43c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.655+0000 7f245533a640 1 -- 172.21.15.27:0/1281201592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596f42eb400 con 0x5596f43ba580 2024-04-25T01:28:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.656+0000 7f2455b3b640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f243c0a2d00 0x7f243c0a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2448005c40 tx=0x7f2448009d70 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.656+0000 7f245533a640 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43ba960 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f244c0c4440 tx=0x7f244c0bb840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2dc75ab242b40e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.656+0000 7f243b7fe640 1 -- 172.21.15.27:0/1281201592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c049070 con 0x5596f43ba580 2024-04-25T01:28:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.656+0000 7f243b7fe640 1 -- 172.21.15.27:0/1281201592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f244c08c320 con 0x5596f43ba580 2024-04-25T01:28:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.656+0000 7f243b7fe640 1 -- 172.21.15.27:0/1281201592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c003c80 con 0x5596f43ba580 2024-04-25T01:28:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.656+0000 7f245a389740 1 -- 172.21.15.27:0/1281201592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 msgr2=0x5596f43ba960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.656+0000 7f245a389740 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43ba960 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f244c0c4440 tx=0x7f244c0bb840 comp rx=0 tx=0).stop 2024-04-25T01:28:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 -- 172.21.15.27:0/1281201592 shutdown_connections 2024-04-25T01:28:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43c5570 0x5596f43c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 0x5596f43c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 --2- 172.21.15.27:0/1281201592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43ba960 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 -- 172.21.15.27:0/1281201592 >> 172.21.15.27:0/1281201592 conn(0x5596f43b68a0 msgr2=0x5596f43b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 -- 172.21.15.27:0/1281201592 shutdown_connections 2024-04-25T01:28:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 -- 172.21.15.27:0/1281201592 wait complete. 2024-04-25T01:28:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 -- start start 2024-04-25T01:28:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 Processor -- start 2024-04-25T01:28:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43c2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5596f43c8120 con 0x5596f43ba580 2024-04-25T01:28:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5596f42db050 con 0x5596f43baea0 2024-04-25T01:28:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5596f43c82a0 con 0x5596f43c5570 2024-04-25T01:28:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245a389740 10 client.0 _ll_register_callbacks cb 0x5596f4032be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245533a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.657+0000 7f245533a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43c4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35128/0 (socket says 172.21.15.27:35128) 2024-04-25T01:28:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245533a640 1 -- 172.21.15.27:0/1926465321 learned_addr learned my addr 172.21.15.27:0/1926465321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f2454b39640 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f2455b3b640 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245533a640 1 -- 172.21.15.27:0/1926465321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 msgr2=0x5596f43c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245533a640 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245533a640 1 -- 172.21.15.27:0/1926465321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 msgr2=0x5596f43c2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245533a640 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245533a640 1 -- 172.21.15.27:0/1926465321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596f42acdc0 con 0x5596f43ba580 2024-04-25T01:28:13.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f2454b39640 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:13.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245533a640 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43c4f60 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f244c000c00 tx=0x7f244c03bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa38f8705bc4e3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f2438ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f2438ff9640 1 -- 172.21.15.27:0/1926465321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c049070 con 0x5596f43ba580 2024-04-25T01:28:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f2438ff9640 1 -- 172.21.15.27:0/1926465321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f244c0ed1b0 con 0x5596f43ba580 2024-04-25T01:28:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245a389740 1 -- 172.21.15.27:0/1926465321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f244c08c320 con 0x5596f43ba580 2024-04-25T01:28:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f2438ff9640 1 -- 172.21.15.27:0/1926465321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f244c0c00b0 con 0x5596f43ba580 2024-04-25T01:28:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245a389740 10 client.16257 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:13.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f245a389740 1 -- 172.21.15.27:0/1926465321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5596f40333f0 con 0x5596f43ba580 2024-04-25T01:28:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.658+0000 7f2438ff9640 1 -- 172.21.15.27:0/1926465321 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f244c08c320 con 0x5596f43ba580 2024-04-25T01:28:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.659+0000 7f2438ff9640 1 -- 172.21.15.27:0/1926465321 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f244c0bbf60 con 0x5596f43ba580 2024-04-25T01:28:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.659+0000 7f245a389740 10 client.16257 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.659+0000 7f245a389740 10 client.16257 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:28:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.659+0000 7f245a389740 20 client.16257 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.659+0000 7f245a389740 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5596f43d5200 0x5596f43d75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.659+0000 7f245a389740 4 client.16257 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.659+0000 7f245a389740 1 -- 172.21.15.27:0/1926465321 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596f432cbb0 con 0x5596f43d5200 2024-04-25T01:28:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.659+0000 7f2454b39640 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5596f43d5200 0x5596f43d75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.660+0000 7f2454b39640 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5596f43d5200 0x5596f43d75f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9456b5db402247a7 server_cookie=5edabc22d981b303 in_seq=0 out_seq=0 2024-04-25T01:28:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.660+0000 7f2438ff9640 10 client.16257 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:28:13.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.660+0000 7f2438ff9640 1 -- 172.21.15.27:0/1926465321 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596f432cbb0 con 0x5596f43d5200 2024-04-25T01:28:13.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.660+0000 7f2438ff9640 10 client.16257 handle_command_reply: tid=0 2024-04-25T01:28:13.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.660+0000 7f245a389740 1 client.16257 shutdown 2024-04-25T01:28:13.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.660+0000 7f245a389740 1 -- 172.21.15.27:0/1926465321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 msgr2=0x5596f43c4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.660+0000 7f245a389740 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43c4f60 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f244c000c00 tx=0x7f244c03bf90 comp rx=0 tx=0).stop 2024-04-25T01:28:13.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 1 -- 172.21.15.27:0/1926465321 shutdown_connections 2024-04-25T01:28:13.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5596f43d5200 0x5596f43d75f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f2438ff9640 0 client.16257 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:28:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 1 --2- 172.21.15.27:0/1926465321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43c4f60 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 1 -- 172.21.15.27:0/1926465321 >> 172.21.15.27:0/1926465321 conn(0x5596f43b68a0 msgr2=0x5596f43bc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 1 -- 172.21.15.27:0/1926465321 shutdown_connections 2024-04-25T01:28:13.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 1 -- 172.21.15.27:0/1926465321 wait complete. 2024-04-25T01:28:13.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.661+0000 7f245a389740 20 client.16257 trim_cache size 0 max 16384 2024-04-25T01:28:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245a389740 1 Processor -- start 2024-04-25T01:28:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245a389740 1 -- start start 2024-04-25T01:28:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245a389740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245a389740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245a389740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43dcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245a389740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f24501b36c0 con 0x5596f43ba580 2024-04-25T01:28:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245a389740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f24501b0f90 con 0x5596f43baea0 2024-04-25T01:28:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245a389740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f24501aede0 con 0x5596f43c5570 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245533a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245533a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43d0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35142/0 (socket says 172.21.15.27:35142) 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.674+0000 7f245533a640 1 -- 172.21.15.27:0/558570313 learned_addr learned my addr 172.21.15.27:0/558570313 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f2454b39640 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f2455b3b640 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43dcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f245533a640 1 -- 172.21.15.27:0/558570313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 msgr2=0x5596f43dcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f245533a640 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43dcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f245533a640 1 -- 172.21.15.27:0/558570313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 msgr2=0x5596f43c4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f245533a640 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f245533a640 1 -- 172.21.15.27:0/558570313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596f43bacd0 con 0x5596f43ba580 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f2455b3b640 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43dcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f245533a640 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43d0f50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f244c009900 tx=0x7f244c0ecfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b82894c3ffe337c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f243affd640 1 -- 172.21.15.27:0/558570313 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c049070 con 0x5596f43ba580 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f243affd640 1 -- 172.21.15.27:0/558570313 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f244c0b84f0 con 0x5596f43ba580 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.675+0000 7f243affd640 1 -- 172.21.15.27:0/558570313 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c003c80 con 0x5596f43ba580 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 -- 172.21.15.27:0/558570313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 msgr2=0x5596f43d0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43d0f50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f244c009900 tx=0x7f244c0ecfe0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 -- 172.21.15.27:0/558570313 shutdown_connections 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43c5570 0x5596f43dcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43baea0 0x5596f43c4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 --2- 172.21.15.27:0/558570313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43ba580 0x5596f43d0f50 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 -- 172.21.15.27:0/558570313 >> 172.21.15.27:0/558570313 conn(0x5596f43b68a0 msgr2=0x5596f43bc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 -- 172.21.15.27:0/558570313 shutdown_connections 2024-04-25T01:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 -- 172.21.15.27:0/558570313 wait complete. 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 -- start start 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.676+0000 7f245a389740 1 Processor -- start 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245a389740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43ba580 0x5596f43cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245a389740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 0x5596f43bea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245a389740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43c5570 0x5596f43bef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245a389740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f24501af450 con 0x5596f43c5570 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245a389740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f24501b08c0 con 0x5596f43ba580 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245a389740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f24501afb20 con 0x5596f43baea0 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245a389740 10 client.0 _ll_register_callbacks cb 0x7f2448014650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43c5570 0x5596f43bef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43c5570 0x5596f43bef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35158/0 (socket says 172.21.15.27:35158) 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 -- 172.21.15.27:0/1646888755 learned_addr learned my addr 172.21.15.27:0/1646888755 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2454b39640 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 0x5596f43bea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245533a640 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43ba580 0x5596f43cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 -- 172.21.15.27:0/1646888755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 msgr2=0x5596f43bea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 0x5596f43bea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 -- 172.21.15.27:0/1646888755 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43ba580 msgr2=0x5596f43cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43ba580 0x5596f43cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 -- 172.21.15.27:0/1646888755 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596f40fd7d0 con 0x5596f43c5570 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245533a640 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43ba580 0x5596f43cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2455b3b640 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43c5570 0x5596f43bef60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2448011cc0 tx=0x7f2448004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8c53c0cf6dc0ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2438ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2438ff9640 1 -- 172.21.15.27:0/1646888755 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448004730 con 0x5596f43c5570 2024-04-25T01:28:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f2438ff9640 1 -- 172.21.15.27:0/1646888755 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f244802b430 con 0x5596f43c5570 2024-04-25T01:28:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.677+0000 7f245a389740 1 -- 172.21.15.27:0/1646888755 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5596f4326f00 con 0x5596f43c5570 2024-04-25T01:28:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f2438ff9640 1 -- 172.21.15.27:0/1646888755 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2448034050 con 0x5596f43c5570 2024-04-25T01:28:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245a389740 10 client.16263 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245a389740 1 -- 172.21.15.27:0/1646888755 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5596f443b130 con 0x5596f43c5570 2024-04-25T01:28:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f2438ff9640 1 -- 172.21.15.27:0/1646888755 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5596f4326f00 con 0x5596f43c5570 2024-04-25T01:28:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f2438ff9640 1 -- 172.21.15.27:0/1646888755 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f244800f050 con 0x5596f43c5570 2024-04-25T01:28:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245a389740 10 client.16263 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245a389740 10 client.16263 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245a389740 20 client.16263 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245a389740 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5596f443b880 0x5596f443dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245a389740 4 client.16263 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245a389740 1 -- 172.21.15.27:0/1646888755 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596f42a4820 con 0x5596f443b880 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.678+0000 7f245533a640 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5596f443b880 0x5596f443dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f2438ff9640 10 client.16263 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245533a640 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5596f443b880 0x5596f443dca0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60ba7f0264e8680a server_cookie=913009728cbab311 in_seq=0 out_seq=0 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f2438ff9640 1 -- 172.21.15.27:0/1646888755 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596f42a4820 con 0x5596f443b880 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f2438ff9640 10 client.16263 handle_command_reply: tid=0 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 client.16263 shutdown 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 -- 172.21.15.27:0/1646888755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43c5570 msgr2=0x5596f43bef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43c5570 0x5596f43bef60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2448011cc0 tx=0x7f2448004ab0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 -- 172.21.15.27:0/1646888755 shutdown_connections 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5596f443b880 0x5596f443dca0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5596f43c5570 0x5596f43bef60 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f2438ff9640 0 client.16263 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:28:13.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596f43baea0 0x5596f43bea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 --2- 172.21.15.27:0/1646888755 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5596f43ba580 0x5596f43cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 -- 172.21.15.27:0/1646888755 >> 172.21.15.27:0/1646888755 conn(0x5596f43b68a0 msgr2=0x5596f43bce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 -- 172.21.15.27:0/1646888755 shutdown_connections 2024-04-25T01:28:13.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 1 -- 172.21.15.27:0/1646888755 wait complete. 2024-04-25T01:28:13.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.679+0000 7f245a389740 20 client.16263 trim_cache size 0 max 16384 2024-04-25T01:28:13.675 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:28:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 -- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f243c0a2d00 msgr2=0x7f243c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f243c0a2d00 0x7f243c0a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2448005c40 tx=0x7f2448009d70 comp rx=0 tx=0).stop 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 -- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450069dc0 msgr2=0x7f2450100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450069dc0 0x7f2450100670 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f244c009230 tx=0x7f244c004270 comp rx=0 tx=0).stop 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 -- 172.21.15.27:0/749264939 shutdown_connections 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2450107730 0x7f24501010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f245006a780 0x7f2450100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f243c0a2d00 0x7f243c0a51c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 --2- 172.21.15.27:0/749264939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2450069dc0 0x7f2450100670 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 -- 172.21.15.27:0/749264939 >> 172.21.15.27:0/749264939 conn(0x7f24500776a0 msgr2=0x7f24500fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 -- 172.21.15.27:0/749264939 shutdown_connections 2024-04-25T01:28:13.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.680+0000 7f245759e640 1 -- 172.21.15.27:0/749264939 wait complete. 2024-04-25T01:28:13.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:28:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.802+0000 7f3ea4f90640 1 -- 172.21.15.27:0/2828513871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100fd0 msgr2=0x7f3ea010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.802+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/2828513871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100fd0 0x7f3ea010d4e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009cd0 tx=0x7f3e9802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 -- 172.21.15.27:0/2828513871 shutdown_connections 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/2828513871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100fd0 0x7f3ea010d4e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/2828513871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ea0100610 0x7f3ea0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/2828513871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ea01066b0 0x7f3ea0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 -- 172.21.15.27:0/2828513871 >> 172.21.15.27:0/2828513871 conn(0x7f3ea0077740 msgr2=0x7f3ea0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 -- 172.21.15.27:0/2828513871 shutdown_connections 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 -- 172.21.15.27:0/2828513871 wait complete. 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 Processor -- start 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.803+0000 7f3ea4f90640 1 -- start start 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3ea4f90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100610 0x7f3ea0197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3ea4f90640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ea0100fd0 0x7f3ea01984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3ea4f90640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ea01066b0 0x7f3ea019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3ea4f90640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea0110a30 con 0x7f3ea0100610 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3ea4f90640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01108b0 con 0x7f3ea01066b0 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3ea4f90640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea0110bb0 con 0x7f3ea0100fd0 2024-04-25T01:28:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100610 0x7f3ea0197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100610 0x7f3ea0197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35168/0 (socket says 172.21.15.27:35168) 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 -- 172.21.15.27:0/588458690 learned_addr learned my addr 172.21.15.27:0/588458690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ea01066b0 0x7f3ea019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9dd9b640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ea0100fd0 0x7f3ea01984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 -- 172.21.15.27:0/588458690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ea0100fd0 msgr2=0x7f3ea01984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ea0100fd0 0x7f3ea01984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 -- 172.21.15.27:0/588458690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ea01066b0 msgr2=0x7f3ea019c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ea01066b0 0x7f3ea019c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 -- 172.21.15.27:0/588458690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea019cf00 con 0x7f3ea0100610 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9dd9b640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ea0100fd0 0x7f3ea01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.804+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100610 0x7f3ea0197f60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800d950 tx=0x7f3e8800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.805+0000 7f3e977fe640 1 -- 172.21.15.27:0/588458690 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88017070 con 0x7f3ea0100610 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.805+0000 7f3e977fe640 1 -- 172.21.15.27:0/588458690 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e88009c30 con 0x7f3ea0100610 2024-04-25T01:28:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.805+0000 7f3ea4f90640 1 -- 172.21.15.27:0/588458690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea019d1f0 con 0x7f3ea0100610 2024-04-25T01:28:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.805+0000 7f3e977fe640 1 -- 172.21.15.27:0/588458690 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8800f600 con 0x7f3ea0100610 2024-04-25T01:28:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.805+0000 7f3ea4f90640 1 -- 172.21.15.27:0/588458690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea01a4a80 con 0x7f3ea0100610 2024-04-25T01:28:13.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.807+0000 7f3e977fe640 1 -- 172.21.15.27:0/588458690 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3e8801a070 con 0x7f3ea0100610 2024-04-25T01:28:13.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3ea7d7b740 1 Processor -- start 2024-04-25T01:28:13.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3ea7d7b740 1 -- start start 2024-04-25T01:28:13.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87ccca960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87ccd5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 0x55b87ccd7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b87cc2f670 con 0x55b87ccca580 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b87cbd7970 con 0x55b87ccd5570 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b87cc2b630 con 0x55b87cccaea0 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3e9ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87ccca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3e9ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87ccca960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35180/0 (socket says 172.21.15.27:35180) 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/900487124 learned_addr learned my addr 172.21.15.27:0/900487124 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3e9dd9b640 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 0x55b87ccd7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3e977fe640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3e8c0a2d00 0x7f3e8c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.808+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87ccd5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/900487124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 msgr2=0x55b87ccd5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87ccd5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/900487124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 msgr2=0x55b87ccd7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 0x55b87ccd7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/900487124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b87cbfb400 con 0x55b87ccca580 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87ccd5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e977fe640 1 -- 172.21.15.27:0/588458690 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3e880cf280 con 0x7f3ea0100610 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3e8c0a2d00 0x7f3e8c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87ccca960 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3e98004ab0 tx=0x7f3e980315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2380fb49441a1db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e94ff9640 1 -- 172.21.15.27:0/900487124 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98041040 con 0x55b87ccca580 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e94ff9640 1 -- 172.21.15.27:0/900487124 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e98043070 con 0x55b87ccca580 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.809+0000 7f3e94ff9640 1 -- 172.21.15.27:0/900487124 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98030c20 con 0x55b87ccca580 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/900487124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 msgr2=0x55b87ccca960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3e8c0a2d00 0x7f3e8c0a51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3e9804c000 tx=0x7f3e9804d000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87ccca960 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3e98004ab0 tx=0x7f3e980315b0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/900487124 shutdown_connections 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 0x55b87ccd7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87ccd5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/900487124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87ccca960 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/900487124 >> 172.21.15.27:0/900487124 conn(0x55b87ccc68a0 msgr2=0x55b87ccc6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/900487124 shutdown_connections 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/900487124 wait complete. 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 -- start start 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.810+0000 7f3ea7d7b740 1 Processor -- start 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87ccca580 0x55b87ccd4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87cccaea0 0x55b87ccd21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87ccd2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b87ccd8360 con 0x55b87ccd5570 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b87cbeb050 con 0x55b87cccaea0 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b87ccd84e0 con 0x55b87ccca580 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3ea7d7b740 10 client.0 _ll_register_callbacks cb 0x55b87c942be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87ccd2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87ccd2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35186/0 (socket says 172.21.15.27:35186) 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/4142210150 learned_addr learned my addr 172.21.15.27:0/4142210150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9dd9b640 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87cccaea0 0x55b87ccd21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87ccca580 0x55b87ccd4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/4142210150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87ccca580 msgr2=0x55b87ccd4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87ccca580 0x55b87ccd4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87cccaea0 msgr2=0x55b87ccd21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87cccaea0 0x55b87ccd21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/4142210150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b87cbbcdc0 con 0x55b87ccd5570 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9dd9b640 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87cccaea0 0x55b87ccd21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87ccd2730 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009430 tx=0x7f3e98030800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61150e07d645907e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e85ffb640 1 -- 172.21.15.27:0/4142210150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98041040 con 0x55b87ccd5570 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3e85ffb640 1 -- 172.21.15.27:0/4142210150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3e9802e070 con 0x55b87ccd5570 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.811+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/4142210150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e98043070 con 0x55b87ccd5570 2024-04-25T01:28:13.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3e85ffb640 1 -- 172.21.15.27:0/4142210150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3e98060050 con 0x55b87ccd5570 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3ea7d7b740 10 client.16275 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/4142210150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b87c9433f0 con 0x55b87ccd5570 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3e85ffb640 1 -- 172.21.15.27:0/4142210150 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e98043070 con 0x55b87ccd5570 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3e85ffb640 1 -- 172.21.15.27:0/4142210150 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f3e98030070 con 0x55b87ccd5570 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3ea7d7b740 10 client.16275 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3ea7d7b740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3ea7d7b740 20 client.16275 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b87cce53f0 0x55b87cce77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3ea7d7b740 4 client.16275 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/4142210150 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b87cc3cbb0 con 0x55b87cce53f0 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.812+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b87cce53f0 0x55b87cce77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.813+0000 7f3e85ffb640 10 client.16275 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.813+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b87cce53f0 0x55b87cce77e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=398a40e1e12b25a6 server_cookie=12601c937e6811a3 in_seq=0 out_seq=0 2024-04-25T01:28:13.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.813+0000 7f3e85ffb640 1 -- 172.21.15.27:0/4142210150 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b87cc3cbb0 con 0x55b87cce53f0 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.813+0000 7f3e85ffb640 10 client.16275 handle_command_reply: tid=0 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.813+0000 7f3ea7d7b740 1 client.16275 shutdown 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/4142210150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 msgr2=0x55b87ccd2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87ccd2730 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009430 tx=0x7f3e98030800 comp rx=0 tx=0).stop 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/4142210150 shutdown_connections 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b87cce53f0 0x55b87cce77e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87ccd2730 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87cccaea0 0x55b87ccd21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/4142210150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87ccca580 0x55b87ccd4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3e85ffb640 0 client.16275 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/4142210150 >> 172.21.15.27:0/4142210150 conn(0x55b87ccc68a0 msgr2=0x55b87ccc5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/4142210150 shutdown_connections 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/4142210150 wait complete. 2024-04-25T01:28:13.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.814+0000 7f3ea7d7b740 20 client.16275 trim_cache size 0 max 16384 2024-04-25T01:28:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3ea7d7b740 1 Processor -- start 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3ea7d7b740 1 -- start start 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87cce1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87ccd4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 0x55b87ccece40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01b1380 con 0x55b87ccca580 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01aec50 con 0x55b87ccd5570 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01acaa0 con 0x55b87cccaea0 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3e9e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87cce1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3e9e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87cce1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35196/0 (socket says 172.21.15.27:35196) 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.827+0000 7f3e9e59c640 1 -- 172.21.15.27:0/3556677629 learned_addr learned my addr 172.21.15.27:0/3556677629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e9e59c640 1 -- 172.21.15.27:0/3556677629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 msgr2=0x55b87ccd4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/3556677629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87ccd4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e9e59c640 1 -- 172.21.15.27:0/3556677629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 msgr2=0x55b87ccece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/3556677629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 0x55b87ccece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e9e59c640 1 -- 172.21.15.27:0/3556677629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b87ca0d7d0 con 0x55b87ccca580 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/3556677629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87cce1250 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3e880cc6f0 tx=0x7f3e88097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952f525956d3fef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e87fff640 1 -- 172.21.15.27:0/3556677629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88021040 con 0x55b87ccca580 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e87fff640 1 -- 172.21.15.27:0/3556677629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e8809b410 con 0x55b87ccca580 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3e87fff640 1 -- 172.21.15.27:0/3556677629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e880cca50 con 0x55b87ccca580 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3556677629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 msgr2=0x55b87cce1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3556677629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87cce1250 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3e880cc6f0 tx=0x7f3e88097040 comp rx=0 tx=0).stop 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3556677629 shutdown_connections 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3556677629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccd5570 0x55b87ccece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3556677629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87ccd4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3556677629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccca580 0x55b87cce1250 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.828+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3556677629 >> 172.21.15.27:0/3556677629 conn(0x55b87ccc68a0 msgr2=0x55b87cccc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3556677629 shutdown_connections 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3556677629 wait complete. 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 -- start start 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 Processor -- start 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccca580 0x55b87ccddb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87cccec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87cccf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01ad110 con 0x55b87ccd5570 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01ae580 con 0x55b87ccca580 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea01ad7e0 con 0x55b87cccaea0 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3ea7d7b740 10 client.0 _ll_register_callbacks cb 0x7f3e90009340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3e9ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87cccf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3e9ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87cccf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35206/0 (socket says 172.21.15.27:35206) 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/3778646255 learned_addr learned my addr 172.21.15.27:0/3778646255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccca580 0x55b87ccddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.829+0000 7f3e9dd9b640 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87cccec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/3778646255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 msgr2=0x55b87cccec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87cccec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/3778646255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccca580 msgr2=0x55b87ccddb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccca580 0x55b87ccddb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e9ed9d640 1 -- 172.21.15.27:0/3778646255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b87cccf9a0 con 0x55b87ccd5570 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e9dd9b640 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87cccec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e9ed9d640 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87cccf1c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3e9802ae00 tx=0x7f3e98030f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e549ca1d0c0954 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e85ffb640 1 -- 172.21.15.27:0/3778646255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9803f020 con 0x55b87ccd5570 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e85ffb640 1 -- 172.21.15.27:0/3778646255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3e9802e040 con 0x55b87ccd5570 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3778646255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e8809b410 con 0x55b87ccd5570 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e85ffb640 1 -- 172.21.15.27:0/3778646255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3e9806b050 con 0x55b87ccd5570 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 10 client.16281 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:13.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3778646255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b87cd4b6c0 con 0x55b87ccd5570 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e85ffb640 1 -- 172.21.15.27:0/3778646255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e8809b410 con 0x55b87ccd5570 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3e85ffb640 1 -- 172.21.15.27:0/3778646255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f3e98051460 con 0x55b87ccd5570 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 10 client.16281 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 10 client.16281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 20 client.16281 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b87cd4bd80 0x55b87cd4e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 4 client.16281 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.830+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3778646255 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b87cc36f00 con 0x55b87cd4bd80 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b87cd4bd80 0x55b87cd4e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3e9e59c640 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b87cd4bd80 0x55b87cd4e170 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93f3e1dd911605d5 server_cookie=1b5a8178b86ab5b0 in_seq=0 out_seq=0 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3e85ffb640 10 client.16281 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3e85ffb640 1 -- 172.21.15.27:0/3778646255 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b87cc36f00 con 0x55b87cd4bd80 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3e85ffb640 10 client.16281 handle_command_reply: tid=0 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 client.16281 shutdown 2024-04-25T01:28:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3778646255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 msgr2=0x55b87cccf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87cccf1c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3e9802ae00 tx=0x7f3e98030f90 comp rx=0 tx=0).stop 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3778646255 shutdown_connections 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b87cd4bd80 0x55b87cd4e170 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87ccd5570 0x55b87cccf1c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3e85ffb640 0 client.16281 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b87cccaea0 0x55b87cccec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 --2- 172.21.15.27:0/3778646255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b87ccca580 0x55b87ccddb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3778646255 >> 172.21.15.27:0/3778646255 conn(0x55b87ccc68a0 msgr2=0x55b87ccd1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3778646255 shutdown_connections 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.831+0000 7f3ea7d7b740 1 -- 172.21.15.27:0/3778646255 wait complete. 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea7d7b740 20 client.16281 trim_cache size 0 max 16384 2024-04-25T01:28:13.816 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:28:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 -- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3e8c0a2d00 msgr2=0x7f3e8c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3e8c0a2d00 0x7f3e8c0a51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3e9804c000 tx=0x7f3e9804d000 comp rx=0 tx=0).stop 2024-04-25T01:28:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 -- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100610 msgr2=0x7f3ea0197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100610 0x7f3ea0197f60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800d950 tx=0x7f3e8800de10 comp rx=0 tx=0).stop 2024-04-25T01:28:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 -- 172.21.15.27:0/588458690 shutdown_connections 2024-04-25T01:28:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ea01066b0 0x7f3ea019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ea0100fd0 0x7f3ea01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3e8c0a2d00 0x7f3e8c0a51c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 --2- 172.21.15.27:0/588458690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea0100610 0x7f3ea0197f60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 -- 172.21.15.27:0/588458690 >> 172.21.15.27:0/588458690 conn(0x7f3ea0077740 msgr2=0x7f3ea00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 -- 172.21.15.27:0/588458690 shutdown_connections 2024-04-25T01:28:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:13.832+0000 7f3ea4f90640 1 -- 172.21.15.27:0/588458690 wait complete. 2024-04-25T01:28:14.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:14 smithi027 ceph-mon[29197]: pgmap v87: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T01:28:14.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:14 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/903667539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:28:14.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:14 smithi153 ceph-mon[32621]: pgmap v87: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T01:28:14.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:14 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/903667539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:28:14.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:14 smithi139 ceph-mon[32760]: pgmap v87: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T01:28:14.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:14 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/903667539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:28:14.821 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:28:14.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:28:14.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:28:14.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 -- 172.21.15.27:0/1034362710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f4100fd0 msgr2=0x7f51f410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 --2- 172.21.15.27:0/1034362710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f4100fd0 0x7f51f410d4e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009cd0 tx=0x7f51ec02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 -- 172.21.15.27:0/1034362710 shutdown_connections 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 --2- 172.21.15.27:0/1034362710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f4100fd0 0x7f51f410d4e0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 --2- 172.21.15.27:0/1034362710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51f4100610 0x7f51f4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 --2- 172.21.15.27:0/1034362710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f41066b0 0x7f51f4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 -- 172.21.15.27:0/1034362710 >> 172.21.15.27:0/1034362710 conn(0x7f51f4077740 msgr2=0x7f51f4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 -- 172.21.15.27:0/1034362710 shutdown_connections 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 -- 172.21.15.27:0/1034362710 wait complete. 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 Processor -- start 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.962+0000 7f51fc076640 1 -- start start 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fc076640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f4100610 0x7f51f4198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fc076640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51f4100fd0 0x7f51f41992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fc076640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f41066b0 0x7f51f41a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fc076640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41109f0 con 0x7f51f41066b0 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fc076640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4110870 con 0x7f51f4100fd0 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fc076640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4110b70 con 0x7f51f4100610 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f41066b0 0x7f51f41a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f41066b0 0x7f51f41a3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35222/0 (socket says 172.21.15.27:35222) 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 -- 172.21.15.27:0/627626405 learned_addr learned my addr 172.21.15.27:0/627626405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51f9e12640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f4100610 0x7f51f4198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51f9611640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51f4100fd0 0x7f51f41992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 -- 172.21.15.27:0/627626405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f4100610 msgr2=0x7f51f4198d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f4100610 0x7f51f4198d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 -- 172.21.15.27:0/627626405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51f4100fd0 msgr2=0x7f51f41992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51f4100fd0 0x7f51f41992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 -- 172.21.15.27:0/627626405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f41a4350 con 0x7f51f41066b0 2024-04-25T01:28:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.963+0000 7f51fa613640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f41066b0 0x7f51f41a3e10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f51ec005cc0 tx=0x7f51ec003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.964+0000 7f51eaffd640 1 -- 172.21.15.27:0/627626405 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec004040 con 0x7f51f41066b0 2024-04-25T01:28:14.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.964+0000 7f51eaffd640 1 -- 172.21.15.27:0/627626405 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51ec0046c0 con 0x7f51f41066b0 2024-04-25T01:28:14.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.964+0000 7f51fc076640 1 -- 172.21.15.27:0/627626405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f41a4520 con 0x7f51f41066b0 2024-04-25T01:28:14.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.964+0000 7f51eaffd640 1 -- 172.21.15.27:0/627626405 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec046480 con 0x7f51f41066b0 2024-04-25T01:28:14.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.964+0000 7f51fc076640 1 -- 172.21.15.27:0/627626405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f41a49e0 con 0x7f51f41066b0 2024-04-25T01:28:14.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.966+0000 7f51eaffd640 1 -- 172.21.15.27:0/627626405 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f51ec0305a0 con 0x7f51f41066b0 2024-04-25T01:28:14.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.966+0000 7f51fed89740 1 Processor -- start 2024-04-25T01:28:14.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.966+0000 7f51fed89740 1 -- start start 2024-04-25T01:28:14.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51fed89740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 0x5648283dacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51fed89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283db290 0x5648283db710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51fed89740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283e6010 0x5648283e8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51fed89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56482833f2c0 con 0x5648283db290 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51fed89740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5648282e7910 con 0x5648283dc980 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51fed89740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56482833b280 con 0x5648283e6010 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51eaffd640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f51e00a2d00 0x7f51e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283db290 0x5648283db710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283db290 0x5648283db710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35232/0 (socket says 172.21.15.27:35232) 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 -- 172.21.15.27:0/2195606660 learned_addr learned my addr 172.21.15.27:0/2195606660 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51fa613640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f51e00a2d00 0x7f51e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9611640 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283e6010 0x5648283e8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51eaffd640 1 -- 172.21.15.27:0/627626405 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f51ec02e030 con 0x7f51f41066b0 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9611640 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 0x5648283dacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 -- 172.21.15.27:0/2195606660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283e6010 msgr2=0x5648283e8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283e6010 0x5648283e8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 -- 172.21.15.27:0/2195606660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 msgr2=0x5648283dacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 0x5648283dacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 -- 172.21.15.27:0/2195606660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648282f8fb0 con 0x5648283db290 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9611640 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283e6010 0x5648283e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51f9e12640 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283db290 0x5648283db710 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f51f000bc30 tx=0x7f51f00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6cb9c4b87d89904 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51fa613640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f51e00a2d00 0x7f51e00a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0ee8c0 tx=0x7f51ec0dd040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.967+0000 7f51dffff640 1 -- 172.21.15.27:0/2195606660 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f001c480 con 0x5648283db290 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51dffff640 1 -- 172.21.15.27:0/2195606660 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f001cb00 con 0x5648283db290 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51dffff640 1 -- 172.21.15.27:0/2195606660 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0011b70 con 0x5648283db290 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 -- 172.21.15.27:0/2195606660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283db290 msgr2=0x5648283db710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283db290 0x5648283db710 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f51f000bc30 tx=0x7f51f00107a0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 -- 172.21.15.27:0/2195606660 shutdown_connections 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283e6010 0x5648283e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283db290 0x5648283db710 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 --2- 172.21.15.27:0/2195606660 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 0x5648283dacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 -- 172.21.15.27:0/2195606660 >> 172.21.15.27:0/2195606660 conn(0x5648283a3950 msgr2=0x5648283d77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 -- 172.21.15.27:0/2195606660 shutdown_connections 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.968+0000 7f51fed89740 1 -- 172.21.15.27:0/2195606660 wait complete. 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 1 -- start start 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 1 Processor -- start 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283db290 0x5648283e2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283dc980 0x5648283e34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283e39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5648283e8ea0 con 0x5648283e6010 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5648282fae90 con 0x5648283db290 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5648283e9020 con 0x5648283dc980 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51fed89740 10 client.0 _ll_register_callbacks cb 0x564828336970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51f9611640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283e39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51f9611640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283e39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35246/0 (socket says 172.21.15.27:35246) 2024-04-25T01:28:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.969+0000 7f51f9611640 1 -- 172.21.15.27:0/3121632734 learned_addr learned my addr 172.21.15.27:0/3121632734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9e12640 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283dc980 0x5648283e34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9611640 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283db290 0x5648283e2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9611640 1 -- 172.21.15.27:0/3121632734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283dc980 msgr2=0x5648283e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9611640 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283dc980 0x5648283e34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9611640 1 -- 172.21.15.27:0/3121632734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283db290 msgr2=0x5648283e2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9611640 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283db290 0x5648283e2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9611640 1 -- 172.21.15.27:0/3121632734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56482811b7e0 con 0x5648283e6010 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9e12640 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283dc980 0x5648283e34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51f9611640 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283e39e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f51e400c750 tx=0x7f51e400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51cc67e73e6d29f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51dd7fa640 1 -- 172.21.15.27:0/3121632734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e400fdc0 con 0x5648283e6010 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51dd7fa640 1 -- 172.21.15.27:0/3121632734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f51e4004030 con 0x5648283e6010 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51fed89740 1 -- 172.21.15.27:0/3121632734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51f001cb00 con 0x5648283e6010 2024-04-25T01:28:14.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51dd7fa640 1 -- 172.21.15.27:0/3121632734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f51e4004230 con 0x5648283e6010 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51fed89740 10 client.16293 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51fed89740 1 -- 172.21.15.27:0/3121632734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56482817bf70 con 0x5648283e6010 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.970+0000 7f51dd7fa640 1 -- 172.21.15.27:0/3121632734 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f001cb00 con 0x5648283e6010 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.971+0000 7f51dd7fa640 1 -- 172.21.15.27:0/3121632734 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f51e401d550 con 0x5648283e6010 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.971+0000 7f51fed89740 10 client.16293 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.971+0000 7f51fed89740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.971+0000 7f51fed89740 20 client.16293 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.971+0000 7f51fed89740 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5648283f6190 0x5648283f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.971+0000 7f51fed89740 4 client.16293 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.971+0000 7f51fed89740 1 -- 172.21.15.27:0/3121632734 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648283135a0 con 0x5648283f6190 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.971+0000 7f51f9e12640 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5648283f6190 0x5648283f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51f9e12640 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5648283f6190 0x5648283f85a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6906c8a6d4dac1d server_cookie=713b036a6dd038ad in_seq=0 out_seq=0 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51dd7fa640 10 client.16293 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51dd7fa640 1 -- 172.21.15.27:0/3121632734 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5648283135a0 con 0x5648283f6190 2024-04-25T01:28:14.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51dd7fa640 10 client.16293 handle_command_reply: tid=0 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51fed89740 1 client.16293 shutdown 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51fed89740 1 -- 172.21.15.27:0/3121632734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 msgr2=0x5648283e39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51fed89740 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283e39e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f51e400c750 tx=0x7f51e400cb90 comp rx=0 tx=0).stop 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51fed89740 1 -- 172.21.15.27:0/3121632734 shutdown_connections 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.972+0000 7f51fed89740 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5648283f6190 0x5648283f85a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.973+0000 7f51fed89740 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283e39e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.973+0000 7f51fed89740 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283dc980 0x5648283e34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.973+0000 7f51dd7fa640 0 client.16293 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.973+0000 7f51fed89740 1 --2- 172.21.15.27:0/3121632734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283db290 0x5648283e2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.973+0000 7f51fed89740 1 -- 172.21.15.27:0/3121632734 >> 172.21.15.27:0/3121632734 conn(0x5648283a3950 msgr2=0x5648283dd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.973+0000 7f51fed89740 1 -- 172.21.15.27:0/3121632734 shutdown_connections 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.973+0000 7f51fed89740 1 -- 172.21.15.27:0/3121632734 wait complete. 2024-04-25T01:28:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.973+0000 7f51fed89740 20 client.16293 trim_cache size 0 max 16384 2024-04-25T01:28:14.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51fed89740 1 Processor -- start 2024-04-25T01:28:14.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51fed89740 1 -- start start 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51fed89740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 0x5648283f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51fed89740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 0x5648283d67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51fed89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283fde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51fed89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41b1490 con 0x5648283e6010 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51fed89740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41aed60 con 0x5648283dc980 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51fed89740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41acbb0 con 0x5648283db290 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283fde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35258/0 (socket says 172.21.15.27:35258) 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 -- 172.21.15.27:0/2657343111 learned_addr learned my addr 172.21.15.27:0/2657343111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 0x5648283f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9e12640 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 0x5648283d67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 -- 172.21.15.27:0/2657343111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 msgr2=0x5648283f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 0x5648283f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 -- 172.21.15.27:0/2657343111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 msgr2=0x5648283d67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 0x5648283d67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.991+0000 7f51f9611640 1 -- 172.21.15.27:0/2657343111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564828013170 con 0x5648283e6010 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51f9611640 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283fde70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f51e40059f0 tx=0x7f51e400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f402709848566203 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51df7fe640 1 -- 172.21.15.27:0/2657343111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e401d630 con 0x5648283e6010 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51df7fe640 1 -- 172.21.15.27:0/2657343111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e400fbc0 con 0x5648283e6010 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51df7fe640 1 -- 172.21.15.27:0/2657343111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e402d930 con 0x5648283e6010 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 -- 172.21.15.27:0/2657343111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 msgr2=0x5648283fde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283fde70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f51e40059f0 tx=0x7f51e400cf90 comp rx=0 tx=0).stop 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 -- 172.21.15.27:0/2657343111 shutdown_connections 2024-04-25T01:28:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283e6010 0x5648283fde70 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283dc980 0x5648283d67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 --2- 172.21.15.27:0/2657343111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 0x5648283f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 -- 172.21.15.27:0/2657343111 >> 172.21.15.27:0/2657343111 conn(0x5648283a3950 msgr2=0x5648283d60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 -- 172.21.15.27:0/2657343111 shutdown_connections 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.992+0000 7f51fed89740 1 -- 172.21.15.27:0/2657343111 wait complete. 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 1 -- start start 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 1 Processor -- start 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 0x5648283ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283dc980 0x5648283ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283e6010 0x5648283eb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41ad220 con 0x5648283dc980 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41ae690 con 0x5648283e6010 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41ad8f0 con 0x5648283db290 2024-04-25T01:28:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51fed89740 10 client.0 _ll_register_callbacks cb 0x7f51e400ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51f9e12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283dc980 0x5648283ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51f9e12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283dc980 0x5648283ee7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35268/0 (socket says 172.21.15.27:35268) 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51f9e12640 1 -- 172.21.15.27:0/2177234301 learned_addr learned my addr 172.21.15.27:0/2177234301 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51f9611640 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 0x5648283ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.993+0000 7f51f9611640 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283e6010 0x5648283eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51f9e12640 1 -- 172.21.15.27:0/2177234301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 msgr2=0x5648283ee2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51f9e12640 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 0x5648283ee2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51f9e12640 1 -- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283e6010 msgr2=0x5648283eb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51f9e12640 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283e6010 0x5648283eb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51f9e12640 1 -- 172.21.15.27:0/2177234301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648283eb7f0 con 0x5648283dc980 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51f9611640 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283e6010 0x5648283eb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51f9e12640 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283dc980 0x5648283ee7f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x5648283dded0 tx=0x7f51f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb1f061ad747320 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51dd7fa640 1 -- 172.21.15.27:0/2177234301 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0005240 con 0x5648283dc980 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51fed89740 1 -- 172.21.15.27:0/2177234301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e400fbc0 con 0x5648283dc980 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51dd7fa640 1 -- 172.21.15.27:0/2177234301 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f51f00053e0 con 0x5648283dc980 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51dd7fa640 1 -- 172.21.15.27:0/2177234301 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f51f0039050 con 0x5648283dc980 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51fed89740 10 client.16299 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51fed89740 1 -- 172.21.15.27:0/2177234301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56482845cf70 con 0x5648283dc980 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51dd7fa640 1 -- 172.21.15.27:0/2177234301 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e400fbc0 con 0x5648283dc980 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.994+0000 7f51dd7fa640 1 -- 172.21.15.27:0/2177234301 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f51f0011a00 con 0x5648283dc980 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51fed89740 10 client.16299 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51fed89740 10 client.16299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51fed89740 20 client.16299 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51fed89740 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56482845d4d0 0x56482845f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51fed89740 4 client.16299 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:28:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51fed89740 1 -- 172.21.15.27:0/2177234301 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56482834c800 con 0x56482845d4d0 2024-04-25T01:28:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51f9611640 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56482845d4d0 0x56482845f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51f9611640 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56482845d4d0 0x56482845f8f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6f0213b026e3904 server_cookie=c79afd128250fc15 in_seq=0 out_seq=0 2024-04-25T01:28:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.995+0000 7f51dd7fa640 10 client.16299 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.996+0000 7f51dd7fa640 1 -- 172.21.15.27:0/2177234301 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56482834c800 con 0x56482845d4d0 2024-04-25T01:28:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.996+0000 7f51dd7fa640 10 client.16299 handle_command_reply: tid=0 2024-04-25T01:28:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 client.16299 shutdown 2024-04-25T01:28:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 -- 172.21.15.27:0/2177234301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283dc980 msgr2=0x5648283ee7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283dc980 0x5648283ee7f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x5648283dded0 tx=0x7f51f000b890 comp rx=0 tx=0).stop 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 -- 172.21.15.27:0/2177234301 shutdown_connections 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56482845d4d0 0x56482845f8f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5648283e6010 0x5648283eb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648283dc980 0x5648283ee7f0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51dd7fa640 0 client.16299 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 --2- 172.21.15.27:0/2177234301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5648283db290 0x5648283ee2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 -- 172.21.15.27:0/2177234301 >> 172.21.15.27:0/2177234301 conn(0x5648283a3950 msgr2=0x5648283d6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 -- 172.21.15.27:0/2177234301 shutdown_connections 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 1 -- 172.21.15.27:0/2177234301 wait complete. 2024-04-25T01:28:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fed89740 20 client.16299 trim_cache size 0 max 16384 2024-04-25T01:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "2e41fbcd-3868-4db8-82f1-32d0220f6d50", 2024-04-25T01:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:28:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.997+0000 7f51fc076640 1 -- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f51e00a2d00 msgr2=0x7f51e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f51e00a2d00 0x7f51e00a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0ee8c0 tx=0x7f51ec0dd040 comp rx=0 tx=0).stop 2024-04-25T01:28:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 -- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f41066b0 msgr2=0x7f51f41a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f41066b0 0x7f51f41a3e10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f51ec005cc0 tx=0x7f51ec003970 comp rx=0 tx=0).stop 2024-04-25T01:28:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 -- 172.21.15.27:0/627626405 shutdown_connections 2024-04-25T01:28:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f51e00a2d00 0x7f51e00a51c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f41066b0 0x7f51f41a3e10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51f4100fd0 0x7f51f41992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 --2- 172.21.15.27:0/627626405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f4100610 0x7f51f4198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 -- 172.21.15.27:0/627626405 >> 172.21.15.27:0/627626405 conn(0x7f51f4077740 msgr2=0x7f51f40ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 -- 172.21.15.27:0/627626405 shutdown_connections 2024-04-25T01:28:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:14.998+0000 7f51fc076640 1 -- 172.21.15.27:0/627626405 wait complete. 2024-04-25T01:28:14.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:28:14.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 -- 172.21.15.27:0/1021158543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd30107720 msgr2=0x7efd30107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 --2- 172.21.15.27:0/1021158543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd30107720 0x7efd30107b00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7efd20009930 tx=0x7efd2002a420 comp rx=0 tx=0).stop 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 -- 172.21.15.27:0/1021158543 shutdown_connections 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 --2- 172.21.15.27:0/1021158543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd301020d0 0x7efd3010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 --2- 172.21.15.27:0/1021158543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd30101710 0x7efd30101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 --2- 172.21.15.27:0/1021158543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd30107720 0x7efd30107b00 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 -- 172.21.15.27:0/1021158543 >> 172.21.15.27:0/1021158543 conn(0x7efd300fd180 msgr2=0x7efd300ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 -- 172.21.15.27:0/1021158543 shutdown_connections 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.113+0000 7efd354e1640 1 -- 172.21.15.27:0/1021158543 wait complete. 2024-04-25T01:28:15.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd354e1640 1 Processor -- start 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd354e1640 1 -- start start 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd354e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd30101710 0x7efd30198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd354e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd301020d0 0x7efd301985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd354e1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd30107720 0x7efd3019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd354e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd30111b30 con 0x7efd301020d0 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd354e1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efd301119b0 con 0x7efd30107720 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd354e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd30111cb0 con 0x7efd30101710 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd2e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd301020d0 0x7efd301985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd2e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd301020d0 0x7efd301985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35290/0 (socket says 172.21.15.27:35290) 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.114+0000 7efd2e7fc640 1 -- 172.21.15.27:0/1302424001 learned_addr learned my addr 172.21.15.27:0/1302424001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2effd640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd30101710 0x7efd30198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd30107720 0x7efd3019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2e7fc640 1 -- 172.21.15.27:0/1302424001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd30101710 msgr2=0x7efd30198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd30101710 0x7efd30198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2e7fc640 1 -- 172.21.15.27:0/1302424001 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd30107720 msgr2=0x7efd3019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd30107720 0x7efd3019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2e7fc640 1 -- 172.21.15.27:0/1302424001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3019d080 con 0x7efd301020d0 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd30107720 0x7efd3019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2effd640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd30101710 0x7efd30198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd301020d0 0x7efd301985d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7efd2400e9f0 tx=0x7efd2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd17fff640 1 -- 172.21.15.27:0/1302424001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2400cd80 con 0x7efd301020d0 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd354e1640 1 -- 172.21.15.27:0/1302424001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3019d370 con 0x7efd301020d0 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd17fff640 1 -- 172.21.15.27:0/1302424001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd24005500 con 0x7efd301020d0 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd17fff640 1 -- 172.21.15.27:0/1302424001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24010690 con 0x7efd301020d0 2024-04-25T01:28:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.115+0000 7efd354e1640 1 -- 172.21.15.27:0/1302424001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd301a4c00 con 0x7efd301020d0 2024-04-25T01:28:15.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.117+0000 7efd17fff640 1 -- 172.21.15.27:0/1302424001 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efd240046c0 con 0x7efd301020d0 2024-04-25T01:28:15.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd381f5740 1 Processor -- start 2024-04-25T01:28:15.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd381f5740 1 -- start start 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd381f5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267bba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd381f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd381f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd17fff640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efd180a2d00 0x7efd180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd381f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a6267206b0 con 0x55a6267c6620 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd381f5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a6266c8880 con 0x55a6267bb630 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd381f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a62671c670 con 0x55a6267bbf50 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efd180a2d00 0x7efd180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267bba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267bba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:49618/0 (socket says 172.21.15.27:49618) 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2f7fe640 1 -- 172.21.15.27:0/4263329926 learned_addr learned my addr 172.21.15.27:0/4263329926 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd17fff640 1 -- 172.21.15.27:0/1302424001 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7efd240cfd90 con 0x7efd301020d0 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2e7fc640 1 -- 172.21.15.27:0/4263329926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 msgr2=0x55a6267c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/4263329926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2e7fc640 1 -- 172.21.15.27:0/4263329926 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 msgr2=0x55a6267bba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/4263329926 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267bba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.118+0000 7efd2e7fc640 1 -- 172.21.15.27:0/4263329926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6264fe7d0 con 0x55a6267c6620 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/4263329926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c8a10 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7efd240cd0f0 tx=0x7efd24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8769d68e4d65057d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efd180a2d00 0x7efd180a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd280092f0 tx=0x7efd28009280 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd157fa640 1 -- 172.21.15.27:0/4263329926 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2409a480 con 0x55a6267c6620 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd157fa640 1 -- 172.21.15.27:0/4263329926 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd2409c5c0 con 0x55a6267c6620 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd157fa640 1 -- 172.21.15.27:0/4263329926 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd240a1a00 con 0x55a6267c6620 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 -- 172.21.15.27:0/4263329926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 msgr2=0x55a6267c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 --2- 172.21.15.27:0/4263329926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c8a10 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7efd240cd0f0 tx=0x7efd24004a40 comp rx=0 tx=0).stop 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 -- 172.21.15.27:0/4263329926 shutdown_connections 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 --2- 172.21.15.27:0/4263329926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c8a10 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 --2- 172.21.15.27:0/4263329926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 --2- 172.21.15.27:0/4263329926 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267bba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 -- 172.21.15.27:0/4263329926 >> 172.21.15.27:0/4263329926 conn(0x55a6267b7950 msgr2=0x55a6267b7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 -- 172.21.15.27:0/4263329926 shutdown_connections 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.119+0000 7efd381f5740 1 -- 172.21.15.27:0/4263329926 wait complete. 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 1 -- start start 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 1 Processor -- start 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267c3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a6267c9410 con 0x55a6267c6620 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a6266dbf50 con 0x55a6267bb630 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a6267c9590 con 0x55a6267bbf50 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd381f5740 10 client.0 _ll_register_callbacks cb 0x55a626717000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35316/0 (socket says 172.21.15.27:35316) 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd2f7fe640 1 -- 172.21.15.27:0/2773890248 learned_addr learned my addr 172.21.15.27:0/2773890248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd2effd640 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.120+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd2f7fe640 1 -- 172.21.15.27:0/2773890248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 msgr2=0x55a6267c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd2f7fe640 1 -- 172.21.15.27:0/2773890248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 msgr2=0x55a6267c3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267c3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd2f7fe640 1 -- 172.21.15.27:0/2773890248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6264343f0 con 0x55a6267c6620 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c3fc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7efd28012fd0 tx=0x7efd28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c78c47f877770ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd12ff5640 1 -- 172.21.15.27:0/2773890248 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28010ca0 con 0x55a6267c6620 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd12ff5640 1 -- 172.21.15.27:0/2773890248 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7efd28004b20 con 0x55a6267c6620 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd381f5740 1 -- 172.21.15.27:0/2773890248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd2409c5c0 con 0x55a6267c6620 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd12ff5640 1 -- 172.21.15.27:0/2773890248 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7efd28010e40 con 0x55a6267c6620 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd381f5740 10 client.16311 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd381f5740 1 -- 172.21.15.27:0/2773890248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a6263f40f0 con 0x55a6267c6620 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd12ff5640 1 -- 172.21.15.27:0/2773890248 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd2409c5c0 con 0x55a6267c6620 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd12ff5640 1 -- 172.21.15.27:0/2773890248 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7efd2800c030 con 0x55a6267c6620 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd381f5740 10 client.16311 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd381f5740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.121+0000 7efd381f5740 20 client.16311 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.122+0000 7efd381f5740 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6267d68f0 0x55a6267d8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.122+0000 7efd381f5740 4 client.16311 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.122+0000 7efd381f5740 1 -- 172.21.15.27:0/2773890248 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a62672dbf0 con 0x55a6267d68f0 2024-04-25T01:28:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.122+0000 7efd2effd640 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6267d68f0 0x55a6267d8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.122+0000 7efd12ff5640 10 client.16311 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.122+0000 7efd2effd640 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6267d68f0 0x55a6267d8d20 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d7522b1749d3bad server_cookie=e70b7dd7aabb65ec in_seq=0 out_seq=0 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd12ff5640 1 -- 172.21.15.27:0/2773890248 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a62672dbf0 con 0x55a6267d68f0 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd12ff5640 10 client.16311 handle_command_reply: tid=0 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 client.16311 shutdown 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 -- 172.21.15.27:0/2773890248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 msgr2=0x55a6267c3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c3fc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7efd28012fd0 tx=0x7efd28004ab0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 -- 172.21.15.27:0/2773890248 shutdown_connections 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6267d68f0 0x55a6267d8d20 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267c3fc0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bbf50 0x55a6267c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 --2- 172.21.15.27:0/2773890248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bb630 0x55a6267c3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 -- 172.21.15.27:0/2773890248 >> 172.21.15.27:0/2773890248 conn(0x55a6267b7950 msgr2=0x55a6267bd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd12ff5640 0 client.16311 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 -- 172.21.15.27:0/2773890248 shutdown_connections 2024-04-25T01:28:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.123+0000 7efd381f5740 1 -- 172.21.15.27:0/2773890248 wait complete. 2024-04-25T01:28:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.124+0000 7efd381f5740 20 client.16311 trim_cache size 0 max 16384 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd381f5740 1 Processor -- start 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd381f5740 1 -- start start 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd381f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 0x55a6267d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd381f5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 0x55a6267c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd381f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd381f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd301b1610 con 0x55a6267c6620 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd381f5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efd301aeee0 con 0x55a6267bbf50 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd381f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd301acd30 con 0x55a6267bb630 2024-04-25T01:28:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35332/0 (socket says 172.21.15.27:35332) 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2f7fe640 1 -- 172.21.15.27:0/2185381167 learned_addr learned my addr 172.21.15.27:0/2185381167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2effd640 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 0x55a6267d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 0x55a6267c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2f7fe640 1 -- 172.21.15.27:0/2185381167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 msgr2=0x55a6267d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 0x55a6267d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2f7fe640 1 -- 172.21.15.27:0/2185381167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 msgr2=0x55a6267c26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 0x55a6267c26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.141+0000 7efd2f7fe640 1 -- 172.21.15.27:0/2185381167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6267a9090 con 0x55a6267c6620 2024-04-25T01:28:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.142+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bddd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7efd28009770 tx=0x7efd28000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c659a145b3728a89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:15.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.144+0000 7efd14ff9640 1 -- 172.21.15.27:0/2185381167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28010ca0 con 0x55a6267c6620 2024-04-25T01:28:15.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.144+0000 7efd14ff9640 1 -- 172.21.15.27:0/2185381167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd28025070 con 0x55a6267c6620 2024-04-25T01:28:15.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.144+0000 7efd14ff9640 1 -- 172.21.15.27:0/2185381167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28020db0 con 0x55a6267c6620 2024-04-25T01:28:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- 172.21.15.27:0/2185381167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 msgr2=0x55a6267bddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bddd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7efd28009770 tx=0x7efd28000f40 comp rx=0 tx=0).stop 2024-04-25T01:28:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- 172.21.15.27:0/2185381167 shutdown_connections 2024-04-25T01:28:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bddd0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 0x55a6267c26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 --2- 172.21.15.27:0/2185381167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 0x55a6267d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- 172.21.15.27:0/2185381167 >> 172.21.15.27:0/2185381167 conn(0x55a6267b7950 msgr2=0x55a6267b6260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- 172.21.15.27:0/2185381167 shutdown_connections 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- 172.21.15.27:0/2185381167 wait complete. 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- start start 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 Processor -- start 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 0x55a6267ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 0x55a6267ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efd301ad3a0 con 0x55a6267c6620 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efd301ae810 con 0x55a6267bbf50 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.145+0000 7efd381f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd301ada70 con 0x55a6267bb630 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd381f5740 10 client.0 _ll_register_callbacks cb 0x7efd28012840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35336/0 (socket says 172.21.15.27:35336) 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 0x55a6267ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 -- 172.21.15.27:0/1022015144 learned_addr learned my addr 172.21.15.27:0/1022015144 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2e7fc640 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 0x55a6267ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 -- 172.21.15.27:0/1022015144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 msgr2=0x55a6267ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 0x55a6267ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 -- 172.21.15.27:0/1022015144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 msgr2=0x55a6267ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 0x55a6267ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 -- 172.21.15.27:0/1022015144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6267c05a0 con 0x55a6267c6620 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd2f7fe640 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bfd70 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7efd28012d20 tx=0x7efd28012a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28cadb1798ebdbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd12ff5640 1 -- 172.21.15.27:0/1022015144 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28043060 con 0x55a6267c6620 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd12ff5640 1 -- 172.21.15.27:0/1022015144 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7efd28020d30 con 0x55a6267c6620 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd381f5740 1 -- 172.21.15.27:0/1022015144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd28025070 con 0x55a6267c6620 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd12ff5640 1 -- 172.21.15.27:0/1022015144 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7efd2804a050 con 0x55a6267c6620 2024-04-25T01:28:15.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd381f5740 10 client.16317 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.146+0000 7efd381f5740 1 -- 172.21.15.27:0/1022015144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a62683ca50 con 0x55a6267c6620 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd12ff5640 1 -- 172.21.15.27:0/1022015144 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd28025070 con 0x55a6267c6620 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd12ff5640 1 -- 172.21.15.27:0/1022015144 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7efd28005510 con 0x55a6267c6620 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd381f5740 10 client.16317 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd381f5740 10 client.16317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd381f5740 20 client.16317 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd381f5740 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a62683d0e0 0x55a62683f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd381f5740 4 client.16317 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd381f5740 1 -- 172.21.15.27:0/1022015144 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a626727f40 con 0x55a62683d0e0 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd2effd640 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a62683d0e0 0x55a62683f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd12ff5640 10 client.16317 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.147+0000 7efd2effd640 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a62683d0e0 0x55a62683f500 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ff6d2570b00525 server_cookie=18dd5528728eb25d in_seq=0 out_seq=0 2024-04-25T01:28:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd12ff5640 1 -- 172.21.15.27:0/1022015144 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a626727f40 con 0x55a62683d0e0 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd12ff5640 10 client.16317 handle_command_reply: tid=0 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 client.16317 shutdown 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 -- 172.21.15.27:0/1022015144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 msgr2=0x55a6267bfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bfd70 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7efd28012d20 tx=0x7efd28012a70 comp rx=0 tx=0).stop 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 -- 172.21.15.27:0/1022015144 shutdown_connections 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a62683d0e0 0x55a62683f500 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6267c6620 0x55a6267bfd70 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6267bbf50 0x55a6267ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd12ff5640 0 client.16317 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 --2- 172.21.15.27:0/1022015144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6267bb630 0x55a6267ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 -- 172.21.15.27:0/1022015144 >> 172.21.15.27:0/1022015144 conn(0x55a6267b7950 msgr2=0x55a6267c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 -- 172.21.15.27:0/1022015144 shutdown_connections 2024-04-25T01:28:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 1 -- 172.21.15.27:0/1022015144 wait complete. 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.148+0000 7efd381f5740 20 client.16317 trim_cache size 0 max 16384 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout: "2e41fbcd-3868-4db8-82f1-32d0220f6d50": { 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "2e41fbcd-3868-4db8-82f1-32d0220f6d50", 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:28:15.132 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:28:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 -- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efd180a2d00 msgr2=0x7efd180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efd180a2d00 0x7efd180a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd280092f0 tx=0x7efd28009280 comp rx=0 tx=0).stop 2024-04-25T01:28:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 -- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd301020d0 msgr2=0x7efd301985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd301020d0 0x7efd301985d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7efd2400e9f0 tx=0x7efd2400edf0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 -- 172.21.15.27:0/1302424001 shutdown_connections 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd30107720 0x7efd3019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efd180a2d00 0x7efd180a51c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd301020d0 0x7efd301985d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 --2- 172.21.15.27:0/1302424001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd30101710 0x7efd30198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 -- 172.21.15.27:0/1302424001 >> 172.21.15.27:0/1302424001 conn(0x7efd300fd180 msgr2=0x7efd300fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 -- 172.21.15.27:0/1302424001 shutdown_connections 2024-04-25T01:28:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:15.149+0000 7efd354e1640 1 -- 172.21.15.27:0/1302424001 wait complete. 2024-04-25T01:28:15.138 INFO:tasks.cephfs.filesystem:scrub status for tag:2e41fbcd-3868-4db8-82f1-32d0220f6d50 - {'path': '/', 'tag': '2e41fbcd-3868-4db8-82f1-32d0220f6d50', 'options': 'recursive,force'} 2024-04-25T01:28:16.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:16 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:28:16.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:16 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:16 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:28:16.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:16 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:28:16.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:16 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:16 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:16 smithi027 ceph-mon[29197]: pgmap v88: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T01:28:16.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:16 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:28:16.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:16 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:16 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:28:16.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:16 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:28:16.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:16 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:16 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:16 smithi153 ceph-mon[32621]: pgmap v88: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T01:28:16.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:16 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:28:16.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:16 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:16 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:28:16.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:16 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:28:16.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:16 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:16 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:16.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:16 smithi139 ceph-mon[32760]: pgmap v88: 129 pgs: 129 active+clean; 1.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T01:28:18.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:18 smithi027 ceph-mon[29197]: pgmap v89: 129 pgs: 129 active+clean; 1.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-04-25T01:28:18.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:18 smithi153 ceph-mon[32621]: pgmap v89: 129 pgs: 129 active+clean; 1.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-04-25T01:28:18.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:18 smithi139 ceph-mon[32760]: pgmap v89: 129 pgs: 129 active+clean; 1.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-04-25T01:28:20.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:20 smithi027 ceph-mon[29197]: pgmap v90: 129 pgs: 129 active+clean; 1.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-04-25T01:28:20.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:20 smithi153 ceph-mon[32621]: pgmap v90: 129 pgs: 129 active+clean; 1.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-04-25T01:28:20.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:20 smithi139 ceph-mon[32760]: pgmap v90: 129 pgs: 129 active+clean; 1.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-04-25T01:28:22.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:22 smithi027 ceph-mon[29197]: pgmap v91: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:22.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:22 smithi153 ceph-mon[32621]: pgmap v91: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:22.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:22 smithi139 ceph-mon[32760]: pgmap v91: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:23.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:23 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:28:23.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:23 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:28:23.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:23 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:28:23.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:23 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:28:23.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:23 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:28:23.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:23 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:28:24.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:24 smithi027 ceph-mon[29197]: pgmap v92: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:28:24.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:24 smithi153 ceph-mon[32621]: pgmap v92: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:28:24.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:24 smithi139 ceph-mon[32760]: pgmap v92: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:28:26.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:26 smithi027 ceph-mon[29197]: pgmap v93: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:28:26.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:26 smithi153 ceph-mon[32621]: pgmap v93: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:28:26.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:26 smithi139 ceph-mon[32760]: pgmap v93: 129 pgs: 129 active+clean; 1.9 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:28:28.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:28 smithi027 ceph-mon[29197]: pgmap v94: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:28:28.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:28 smithi153 ceph-mon[32621]: pgmap v94: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:28:28.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:28 smithi139 ceph-mon[32760]: pgmap v94: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:28:30.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:30 smithi027 ceph-mon[29197]: pgmap v95: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:28:30.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:30 smithi153 ceph-mon[32621]: pgmap v95: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:28:30.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:30 smithi139 ceph-mon[32760]: pgmap v95: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-04-25T01:28:32.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:32 smithi027 ceph-mon[29197]: pgmap v96: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 KiB/s wr, 14 op/s 2024-04-25T01:28:32.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:32 smithi153 ceph-mon[32621]: pgmap v96: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 KiB/s wr, 14 op/s 2024-04-25T01:28:32.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:32 smithi139 ceph-mon[32760]: pgmap v96: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 KiB/s wr, 14 op/s 2024-04-25T01:28:33.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:33 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:33.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:33 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:33.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:33 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:33.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:33 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:28:33.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:33 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:33.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:33 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:33.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:33 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:33.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:33 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:28:33.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:33 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:33.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:33 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:33.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:33 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:33.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:33 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:28:34.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:34 smithi027 ceph-mon[29197]: pgmap v97: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:34.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:34 smithi153 ceph-mon[32621]: pgmap v97: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:34.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:34 smithi139 ceph-mon[32760]: pgmap v97: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:36.446 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:36 smithi027 ceph-mon[29197]: pgmap v98: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:36.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:36 smithi153 ceph-mon[32621]: pgmap v98: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:36.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:36 smithi139 ceph-mon[32760]: pgmap v98: 129 pgs: 129 active+clean; 1.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:38.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:38 smithi027 ceph-mon[29197]: pgmap v99: 129 pgs: 129 active+clean; 2.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:38.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:38 smithi153 ceph-mon[32621]: pgmap v99: 129 pgs: 129 active+clean; 2.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:38.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:38 smithi139 ceph-mon[32760]: pgmap v99: 129 pgs: 129 active+clean; 2.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:40.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:40 smithi027 ceph-mon[29197]: pgmap v100: 129 pgs: 129 active+clean; 2.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:40.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:40 smithi153 ceph-mon[32621]: pgmap v100: 129 pgs: 129 active+clean; 2.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:40.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:40 smithi139 ceph-mon[32760]: pgmap v100: 129 pgs: 129 active+clean; 2.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:42.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:42 smithi027 ceph-mon[29197]: pgmap v101: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.7 KiB/s wr, 14 op/s 2024-04-25T01:28:42.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:42 smithi153 ceph-mon[32621]: pgmap v101: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.7 KiB/s wr, 14 op/s 2024-04-25T01:28:42.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:42 smithi139 ceph-mon[32760]: pgmap v101: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.7 KiB/s wr, 14 op/s 2024-04-25T01:28:44.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:44 smithi027 ceph-mon[29197]: pgmap v102: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:44.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:44 smithi153 ceph-mon[32621]: pgmap v102: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:44.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:44 smithi139 ceph-mon[32760]: pgmap v102: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:45.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:28:45.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:28:45.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 -- 172.21.15.27:0/1499826330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d806a780 msgr2=0x7f74d810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 --2- 172.21.15.27:0/1499826330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d806a780 0x7f74d810e5f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009cd0 tx=0x7f74d002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 -- 172.21.15.27:0/1499826330 shutdown_connections 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 --2- 172.21.15.27:0/1499826330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d806a780 0x7f74d810e5f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 --2- 172.21.15.27:0/1499826330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d8069dc0 0x7f74d806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 --2- 172.21.15.27:0/1499826330 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74d8107730 0x7f74d8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 -- 172.21.15.27:0/1499826330 >> 172.21.15.27:0/1499826330 conn(0x7f74d80776a0 msgr2=0x7f74d8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 -- 172.21.15.27:0/1499826330 shutdown_connections 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.277+0000 7f74de55c640 1 -- 172.21.15.27:0/1499826330 wait complete. 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.278+0000 7f74de55c640 1 Processor -- start 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.278+0000 7f74de55c640 1 -- start start 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.278+0000 7f74de55c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d8069dc0 0x7f74d8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.278+0000 7f74de55c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d806a780 0x7f74d81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.278+0000 7f74de55c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74d8107730 0x7f74d819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.278+0000 7f74de55c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f74d8111b00 con 0x7f74d8069dc0 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.278+0000 7f74de55c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f74d8111980 con 0x7f74d8107730 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.278+0000 7f74de55c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74d8111c80 con 0x7f74d806a780 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d8069dc0 0x7f74d8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d8069dc0 0x7f74d8197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47832/0 (socket says 172.21.15.27:47832) 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 -- 172.21.15.27:0/1680647118 learned_addr learned my addr 172.21.15.27:0/1680647118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d77fe640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d806a780 0x7f74d81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74d8107730 0x7f74d819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 -- 172.21.15.27:0/1680647118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d806a780 msgr2=0x7f74d81984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d806a780 0x7f74d81984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 -- 172.21.15.27:0/1680647118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74d8107730 msgr2=0x7f74d819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74d8107730 0x7f74d819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 -- 172.21.15.27:0/1680647118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d819cf50 con 0x7f74d8069dc0 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74d8107730 0x7f74d819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d7fff640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d8069dc0 0x7f74d8197f60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f74c000d950 tx=0x7f74c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d57fa640 1 -- 172.21.15.27:0/1680647118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c0017070 con 0x7f74d8069dc0 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d57fa640 1 -- 172.21.15.27:0/1680647118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c0009c30 con 0x7f74d8069dc0 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74de55c640 1 -- 172.21.15.27:0/1680647118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d819d240 con 0x7f74d8069dc0 2024-04-25T01:28:45.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74d57fa640 1 -- 172.21.15.27:0/1680647118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c000f6b0 con 0x7f74d8069dc0 2024-04-25T01:28:45.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.279+0000 7f74de55c640 1 -- 172.21.15.27:0/1680647118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d81a4ad0 con 0x7f74d8069dc0 2024-04-25T01:28:45.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.281+0000 7f74d57fa640 1 -- 172.21.15.27:0/1680647118 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f74c00109d0 con 0x7f74d8069dc0 2024-04-25T01:28:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74e1260740 1 Processor -- start 2024-04-25T01:28:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74e1260740 1 -- start start 2024-04-25T01:28:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74d57fa640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f74c40a2d00 0x7f74c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74e1260740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86630 0x555f0fc86a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74e1260740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86f50 0x555f0fc910e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74e1260740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc91620 0x555f0fc93a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74e1260740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555f0fbeb6b0 con 0x555f0fc86f50 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74e1260740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555f0fb93880 con 0x555f0fc86630 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74e1260740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555f0fbe7670 con 0x555f0fc91620 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86f50 0x555f0fc910e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f74c40a2d00 0x7f74c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86f50 0x555f0fc910e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47840/0 (socket says 172.21.15.27:47840) 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74d77fe640 1 -- 172.21.15.27:0/601222685 learned_addr learned my addr 172.21.15.27:0/601222685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.282+0000 7f74d57fa640 1 -- 172.21.15.27:0/1680647118 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f74c00cf430 con 0x7f74d8069dc0 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74d77fe640 1 -- 172.21.15.27:0/601222685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc91620 msgr2=0x555f0fc93a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74d77fe640 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc91620 0x555f0fc93a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74d77fe640 1 -- 172.21.15.27:0/601222685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86630 msgr2=0x555f0fc86a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74d7fff640 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86630 0x555f0fc86a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74d77fe640 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86630 0x555f0fc86a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74d77fe640 1 -- 172.21.15.27:0/601222685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f0f9c97d0 con 0x555f0fc86f50 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74d7fff640 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86630 0x555f0fc86a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f74c40a2d00 0x7f74c40a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x555f0fc83ae0 tx=0x7f74d0009800 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74d77fe640 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86f50 0x555f0fc910e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f74c800dc80 tx=0x7f74c800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ea68282110f4c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74ce7fc640 1 -- 172.21.15.27:0/601222685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c8015070 con 0x555f0fc86f50 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74ce7fc640 1 -- 172.21.15.27:0/601222685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c801a940 con 0x555f0fc86f50 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.283+0000 7f74ce7fc640 1 -- 172.21.15.27:0/601222685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c8011a40 con 0x555f0fc86f50 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 -- 172.21.15.27:0/601222685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86f50 msgr2=0x555f0fc910e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86f50 0x555f0fc910e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f74c800dc80 tx=0x7f74c800f730 comp rx=0 tx=0).stop 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 -- 172.21.15.27:0/601222685 shutdown_connections 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc91620 0x555f0fc93a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86f50 0x555f0fc910e0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 --2- 172.21.15.27:0/601222685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86630 0x555f0fc86a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 -- 172.21.15.27:0/601222685 >> 172.21.15.27:0/601222685 conn(0x555f0fc82950 msgr2=0x555f0fc82d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 -- 172.21.15.27:0/601222685 shutdown_connections 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 -- 172.21.15.27:0/601222685 wait complete. 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 -- start start 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.284+0000 7f74e1260740 1 Processor -- start 2024-04-25T01:28:45.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74e1260740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74e1260740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc8eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74e1260740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74e1260740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555f0fc94410 con 0x555f0fc91620 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74e1260740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555f0fba6f50 con 0x555f0fc86f50 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74e1260740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555f0fc94590 con 0x555f0fc86630 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74e1260740 10 client.0 _ll_register_callbacks cb 0x555f0fbe2000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47854/0 (socket says 172.21.15.27:47854) 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 -- 172.21.15.27:0/2360465347 learned_addr learned my addr 172.21.15.27:0/2360465347 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74d7fff640 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 -- 172.21.15.27:0/2360465347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 msgr2=0x555f0fc8e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74d77fe640 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc8eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 -- 172.21.15.27:0/2360465347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 msgr2=0x555f0fc8eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc8eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 -- 172.21.15.27:0/2360465347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f0f8ff3f0 con 0x555f0fc91620 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74d7fff640 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8f040 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f74d0031e00 tx=0x7f74d00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96423ef4555fb5d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74b77fe640 1 -- 172.21.15.27:0/2360465347 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d004e070 con 0x555f0fc91620 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74b77fe640 1 -- 172.21.15.27:0/2360465347 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f74d00305a0 con 0x555f0fc91620 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.285+0000 7f74e1260740 1 -- 172.21.15.27:0/2360465347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74c801a940 con 0x555f0fc91620 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74b77fe640 1 -- 172.21.15.27:0/2360465347 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f74d005a050 con 0x555f0fc91620 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74e1260740 10 client.16329 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74e1260740 1 -- 172.21.15.27:0/2360465347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555f0f8bf0f0 con 0x555f0fc91620 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74b77fe640 1 -- 172.21.15.27:0/2360465347 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c801a940 con 0x555f0fc91620 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74b77fe640 1 -- 172.21.15.27:0/2360465347 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f74d0049070 con 0x555f0fc91620 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74e1260740 10 client.16329 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:45.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74e1260740 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74e1260740 20 client.16329 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74e1260740 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f0fca18b0 0x555f0fca3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74e1260740 4 client.16329 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74e1260740 1 -- 172.21.15.27:0/2360465347 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f0fbf8bf0 con 0x555f0fca18b0 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.286+0000 7f74d7fff640 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f0fca18b0 0x555f0fca3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.287+0000 7f74d7fff640 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f0fca18b0 0x555f0fca3d70 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8271db6a78333da2 server_cookie=5bed69a08faf0ec9 in_seq=0 out_seq=0 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.287+0000 7f74b77fe640 10 client.16329 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.287+0000 7f74b77fe640 1 -- 172.21.15.27:0/2360465347 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f0fbf8bf0 con 0x555f0fca18b0 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.287+0000 7f74b77fe640 10 client.16329 handle_command_reply: tid=0 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.287+0000 7f74e1260740 1 client.16329 shutdown 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 -- 172.21.15.27:0/2360465347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 msgr2=0x555f0fc8f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8f040 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f74d0031e00 tx=0x7f74d00095c0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 -- 172.21.15.27:0/2360465347 shutdown_connections 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f0fca18b0 0x555f0fca3d70 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8f040 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc8eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 --2- 172.21.15.27:0/2360465347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 -- 172.21.15.27:0/2360465347 >> 172.21.15.27:0/2360465347 conn(0x555f0fc82950 msgr2=0x555f0fc88650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74b77fe640 0 client.16329 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 -- 172.21.15.27:0/2360465347 shutdown_connections 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 1 -- 172.21.15.27:0/2360465347 wait complete. 2024-04-25T01:28:45.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.288+0000 7f74e1260740 20 client.16329 trim_cache size 0 max 16384 2024-04-25T01:28:45.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74e1260740 1 Processor -- start 2024-04-25T01:28:45.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74e1260740 1 -- start start 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74e1260740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86630 0x555f0fc90cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74e1260740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc9d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74e1260740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc91620 0x555f0fc8ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74e1260740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f74d81b14c0 con 0x555f0fc86630 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74e1260740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f74d81aed90 con 0x555f0fc86f50 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74e1260740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74d81acbe0 con 0x555f0fc91620 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86630 0x555f0fc90cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86630 0x555f0fc90cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47858/0 (socket says 172.21.15.27:47858) 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 -- 172.21.15.27:0/3160390262 learned_addr learned my addr 172.21.15.27:0/3160390262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d77fe640 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc9d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 -- 172.21.15.27:0/3160390262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc91620 msgr2=0x555f0fc8ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc91620 0x555f0fc8ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 -- 172.21.15.27:0/3160390262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 msgr2=0x555f0fc9d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc9d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 -- 172.21.15.27:0/3160390262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f0fc7e2a0 con 0x555f0fc86630 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d77fe640 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc9d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.306+0000 7f74d7fff640 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86630 0x555f0fc90cf0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f74c00cc850 tx=0x7f74c0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a27e6fb3aded4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74cdffb640 1 -- 172.21.15.27:0/3160390262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c0020040 con 0x555f0fc86630 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74cdffb640 1 -- 172.21.15.27:0/3160390262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c009b5c0 con 0x555f0fc86630 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74cdffb640 1 -- 172.21.15.27:0/3160390262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c00998c0 con 0x555f0fc86630 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 -- 172.21.15.27:0/3160390262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86630 msgr2=0x555f0fc90cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86630 0x555f0fc90cf0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f74c00cc850 tx=0x7f74c0005f90 comp rx=0 tx=0).stop 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 -- 172.21.15.27:0/3160390262 shutdown_connections 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc91620 0x555f0fc8ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc9d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 --2- 172.21.15.27:0/3160390262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc86630 0x555f0fc90cf0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 -- 172.21.15.27:0/3160390262 >> 172.21.15.27:0/3160390262 conn(0x555f0fc82950 msgr2=0x555f0fc8e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 -- 172.21.15.27:0/3160390262 shutdown_connections 2024-04-25T01:28:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 -- 172.21.15.27:0/3160390262 wait complete. 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.307+0000 7f74e1260740 1 -- start start 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74e1260740 1 Processor -- start 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74e1260740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74e1260740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc8b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74e1260740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74e1260740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f74d81ad250 con 0x555f0fc91620 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74e1260740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f74d81ae6c0 con 0x555f0fc86f50 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74e1260740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74d81ad920 con 0x555f0fc86630 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74e1260740 10 client.0 _ll_register_callbacks cb 0x7f74d0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74dcaf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74dcaf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47864/0 (socket says 172.21.15.27:47864) 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74dcaf9640 1 -- 172.21.15.27:0/3584890738 learned_addr learned my addr 172.21.15.27:0/3584890738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74d7fff640 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74d77fe640 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc8b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74dcaf9640 1 -- 172.21.15.27:0/3584890738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 msgr2=0x555f0fc8b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74dcaf9640 1 -- 172.21.15.27:0/3584890738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 msgr2=0x555f0fc8b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc8b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.308+0000 7f74dcaf9640 1 -- 172.21.15.27:0/3584890738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f0fd06eb0 con 0x555f0fc91620 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74dcaf9640 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8bcb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f74d0004010 tx=0x7f74d002b010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8816eb70911cf728 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74b77fe640 1 -- 172.21.15.27:0/3584890738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d004e050 con 0x555f0fc91620 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74b77fe640 1 -- 172.21.15.27:0/3584890738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f74d003dd80 con 0x555f0fc91620 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 1 -- 172.21.15.27:0/3584890738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74c009b5c0 con 0x555f0fc91620 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74b77fe640 1 -- 172.21.15.27:0/3584890738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f74d006b050 con 0x555f0fc91620 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 10 client.16335 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 1 -- 172.21.15.27:0/3584890738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555f0fd08070 con 0x555f0fc91620 2024-04-25T01:28:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74b77fe640 1 -- 172.21.15.27:0/3584890738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c009b5c0 con 0x555f0fc91620 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74b77fe640 1 -- 172.21.15.27:0/3584890738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f74d0049040 con 0x555f0fc91620 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 10 client.16335 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 10 client.16335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 20 client.16335 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f0fd087b0 0x555f0fd0abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 4 client.16335 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.309+0000 7f74e1260740 1 -- 172.21.15.27:0/3584890738 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555f0fbf2f40 con 0x555f0fd087b0 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74d7fff640 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f0fd087b0 0x555f0fd0abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74b77fe640 10 client.16335 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74d7fff640 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f0fd087b0 0x555f0fd0abd0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e95e365d2b110f65 server_cookie=67d73185a28d53d9 in_seq=0 out_seq=0 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74b77fe640 1 -- 172.21.15.27:0/3584890738 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555f0fbf2f40 con 0x555f0fd087b0 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74b77fe640 10 client.16335 handle_command_reply: tid=0 2024-04-25T01:28:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 client.16335 shutdown 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 -- 172.21.15.27:0/3584890738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 msgr2=0x555f0fc8bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8bcb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f74d0004010 tx=0x7f74d002b010 comp rx=0 tx=0).stop 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 -- 172.21.15.27:0/3584890738 shutdown_connections 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f0fd087b0 0x555f0fd0abd0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f0fc91620 0x555f0fc8bcb0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f0fc86f50 0x555f0fc8b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 --2- 172.21.15.27:0/3584890738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f0fc86630 0x555f0fc8b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74b77fe640 0 client.16335 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 -- 172.21.15.27:0/3584890738 >> 172.21.15.27:0/3584890738 conn(0x555f0fc82950 msgr2=0x555f0fc81af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.310+0000 7f74e1260740 1 -- 172.21.15.27:0/3584890738 shutdown_connections 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74e1260740 1 -- 172.21.15.27:0/3584890738 wait complete. 2024-04-25T01:28:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74e1260740 20 client.16335 trim_cache size 0 max 16384 2024-04-25T01:28:45.297 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:28:45.297 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:28:45.297 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:28:45.297 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:28:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 -- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f74c40a2d00 msgr2=0x7f74c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f74c40a2d00 0x7f74c40a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x555f0fc83ae0 tx=0x7f74d0009800 comp rx=0 tx=0).stop 2024-04-25T01:28:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 -- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d8069dc0 msgr2=0x7f74d8197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d8069dc0 0x7f74d8197f60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f74c000d950 tx=0x7f74c000de10 comp rx=0 tx=0).stop 2024-04-25T01:28:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 -- 172.21.15.27:0/1680647118 shutdown_connections 2024-04-25T01:28:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f74d8107730 0x7f74d819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74d806a780 0x7f74d81984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f74c40a2d00 0x7f74c40a51c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 --2- 172.21.15.27:0/1680647118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d8069dc0 0x7f74d8197f60 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 -- 172.21.15.27:0/1680647118 >> 172.21.15.27:0/1680647118 conn(0x7f74d80776a0 msgr2=0x7f74d80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 -- 172.21.15.27:0/1680647118 shutdown_connections 2024-04-25T01:28:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.311+0000 7f74de55c640 1 -- 172.21.15.27:0/1680647118 wait complete. 2024-04-25T01:28:45.301 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:28:45.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- 172.21.15.27:0/619485352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 msgr2=0x7f45dc10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/619485352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 0x7f45dc10d4f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009930 tx=0x7f45cc02a420 comp rx=0 tx=0).stop 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- 172.21.15.27:0/619485352 shutdown_connections 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/619485352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 0x7f45dc10d4f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/619485352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45dc100620 0x7f45dc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/619485352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45dc106630 0x7f45dc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- 172.21.15.27:0/619485352 >> 172.21.15.27:0/619485352 conn(0x7f45dc077740 msgr2=0x7f45dc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- 172.21.15.27:0/619485352 shutdown_connections 2024-04-25T01:28:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- 172.21.15.27:0/619485352 wait complete. 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 Processor -- start 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- start start 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45dc100620 0x7f45dc198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 0x7f45dc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45dc106630 0x7f45dc19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111b40 con 0x7f45dc100fe0 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1119c0 con 0x7f45dc100620 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.425+0000 7f45e2b1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111cc0 con 0x7f45dc106630 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 0x7f45dc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 0x7f45dc1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47882/0 (socket says 172.21.15.27:47882) 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 -- 172.21.15.27:0/2561122666 learned_addr learned my addr 172.21.15.27:0/2561122666 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45e08b9640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45dc100620 0x7f45dc198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45e10ba640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45dc106630 0x7f45dc19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 -- 172.21.15.27:0/2561122666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45dc106630 msgr2=0x7f45dc19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45dc106630 0x7f45dc19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 -- 172.21.15.27:0/2561122666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45dc100620 msgr2=0x7f45dc198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45dc100620 0x7f45dc198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 -- 172.21.15.27:0/2561122666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc19d020 con 0x7f45dc100fe0 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45dbfff640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 0x7f45dc1985c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f45c400d950 tx=0x7f45c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45d9ffb640 1 -- 172.21.15.27:0/2561122666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4017070 con 0x7f45dc100fe0 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45dc19d310 con 0x7f45dc100fe0 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45d9ffb640 1 -- 172.21.15.27:0/2561122666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c4009c30 con 0x7f45dc100fe0 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45d9ffb640 1 -- 172.21.15.27:0/2561122666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400f6b0 con 0x7f45dc100fe0 2024-04-25T01:28:45.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.426+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45dc1a4ba0 con 0x7f45dc100fe0 2024-04-25T01:28:45.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.428+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45dc068560 con 0x7f45dc100fe0 2024-04-25T01:28:45.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.428+0000 7f45d9ffb640 1 -- 172.21.15.27:0/2561122666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f45c40109d0 con 0x7f45dc100fe0 2024-04-25T01:28:45.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.429+0000 7f45d9ffb640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f45c80a2d00 0x7f45c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.429+0000 7f45e08b9640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f45c80a2d00 0x7f45c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.429+0000 7f45d9ffb640 1 -- 172.21.15.27:0/2561122666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f45c40cf430 con 0x7f45dc100fe0 2024-04-25T01:28:45.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.430+0000 7f45e08b9640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f45c80a2d00 0x7f45c80a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f45d4005fd0 tx=0x7f45d4009500 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.434+0000 7f45d9ffb640 1 -- 172.21.15.27:0/2561122666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f45c4009de0 con 0x7f45dc100fe0 2024-04-25T01:28:45.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.664+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45dc108b90 con 0x7f45dc100fe0 2024-04-25T01:28:45.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.665+0000 7f45d9ffb640 1 -- 172.21.15.27:0/2561122666 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11883 (secure 0 0 0) 0x7f45c409bcc0 con 0x7f45dc100fe0 2024-04-25T01:28:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:28:45.644 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:26:47.433054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24517,24521,24505]},"id":1}]} 2024-04-25T01:28:45.645 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f45c80a2d00 msgr2=0x7f45c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f45c80a2d00 0x7f45c80a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f45d4005fd0 tx=0x7f45d4009500 comp rx=0 tx=0).stop 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 msgr2=0x7f45dc1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 0x7f45dc1985c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f45c400d950 tx=0x7f45c400de10 comp rx=0 tx=0).stop 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 shutdown_connections 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45dc106630 0x7f45dc19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f45c80a2d00 0x7f45c80a51c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45dc100fe0 0x7f45dc1985c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 --2- 172.21.15.27:0/2561122666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45dc100620 0x7f45dc198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 >> 172.21.15.27:0/2561122666 conn(0x7f45dc077740 msgr2=0x7f45dc102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 shutdown_connections 2024-04-25T01:28:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.667+0000 7f45e2b1d640 1 -- 172.21.15.27:0/2561122666 wait complete. 2024-04-25T01:28:45.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:28:45.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 -- 172.21.15.27:0/3098762123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a78107730 msgr2=0x7f9a78107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/3098762123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a78107730 0x7f9a78107b10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009930 tx=0x7f9a6802a420 comp rx=0 tx=0).stop 2024-04-25T01:28:45.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 -- 172.21.15.27:0/3098762123 shutdown_connections 2024-04-25T01:28:45.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/3098762123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a7806a780 0x7f9a7810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/3098762123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a78069dc0 0x7f9a7806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/3098762123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a78107730 0x7f9a78107b10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 -- 172.21.15.27:0/3098762123 >> 172.21.15.27:0/3098762123 conn(0x7f9a780776a0 msgr2=0x7f9a78077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 -- 172.21.15.27:0/3098762123 shutdown_connections 2024-04-25T01:28:45.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.782+0000 7f9a7d05d640 1 -- 172.21.15.27:0/3098762123 wait complete. 2024-04-25T01:28:45.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a7d05d640 1 Processor -- start 2024-04-25T01:28:45.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a7d05d640 1 -- start start 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a7d05d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a78069dc0 0x7f9a78100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a7d05d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a7806a780 0x7f9a78100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a7d05d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a78107730 0x7f9a781010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a7d05d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9a78111b40 con 0x7f9a7806a780 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a7d05d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9a781119c0 con 0x7f9a78107730 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a7d05d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a78111cc0 con 0x7f9a78069dc0 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a6ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a7806a780 0x7f9a78100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a6ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a7806a780 0x7f9a78100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47898/0 (socket says 172.21.15.27:47898) 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.783+0000 7f9a6ffff640 1 -- 172.21.15.27:0/2240945670 learned_addr learned my addr 172.21.15.27:0/2240945670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a76d9d640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a78069dc0 0x7f9a78100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a6ffff640 1 -- 172.21.15.27:0/2240945670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a78069dc0 msgr2=0x7f9a78100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a7759e640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a78107730 0x7f9a781010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a6ffff640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a78069dc0 0x7f9a78100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a6ffff640 1 -- 172.21.15.27:0/2240945670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a78107730 msgr2=0x7f9a781010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a6ffff640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a78107730 0x7f9a781010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a6ffff640 1 -- 172.21.15.27:0/2240945670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a781a6500 con 0x7f9a7806a780 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a6ffff640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a7806a780 0x7f9a78100bb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000d770 tx=0x7f9a6000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a74d99640 1 -- 172.21.15.27:0/2240945670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60017070 con 0x7f9a7806a780 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a7d05d640 1 -- 172.21.15.27:0/2240945670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a781a67f0 con 0x7f9a7806a780 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a74d99640 1 -- 172.21.15.27:0/2240945670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a60005500 con 0x7f9a7806a780 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a74d99640 1 -- 172.21.15.27:0/2240945670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60004e00 con 0x7f9a7806a780 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.784+0000 7f9a7d05d640 1 -- 172.21.15.27:0/2240945670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a781a6d40 con 0x7f9a7806a780 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.786+0000 7f9a74d99640 1 -- 172.21.15.27:0/2240945670 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9a60004fa0 con 0x7f9a7806a780 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7fd6f740 1 Processor -- start 2024-04-25T01:28:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7fd6f740 1 -- start start 2024-04-25T01:28:45.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c790960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790ea0 0x55f15c79b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a74d99640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9a640a2d30 0x7f9a640a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c79b570 0x55f15c79d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f15c6f5670 con 0x55f15c790580 2024-04-25T01:28:45.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f15c69d970 con 0x55f15c790ea0 2024-04-25T01:28:45.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f15c6f1630 con 0x55f15c79b570 2024-04-25T01:28:45.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c790960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a6ffff640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9a640a2d30 0x7f9a640a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c790960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47902/0 (socket says 172.21.15.27:47902) 2024-04-25T01:28:45.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7759e640 1 -- 172.21.15.27:0/444479807 learned_addr learned my addr 172.21.15.27:0/444479807 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a76d9d640 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790ea0 0x55f15c79b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a74d99640 1 -- 172.21.15.27:0/2240945670 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f9a600cf1e0 con 0x7f9a7806a780 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.787+0000 7f9a7759e640 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c79b570 0x55f15c79d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a7759e640 1 -- 172.21.15.27:0/444479807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c79b570 msgr2=0x55f15c79d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a7759e640 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c79b570 0x55f15c79d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a7759e640 1 -- 172.21.15.27:0/444479807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790ea0 msgr2=0x55f15c79b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a7759e640 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790ea0 0x55f15c79b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a7759e640 1 -- 172.21.15.27:0/444479807 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f15c6c1400 con 0x55f15c790580 2024-04-25T01:28:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a6ffff640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9a640a2d30 0x7f9a640a51f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9a6009b070 tx=0x7f9a600bb040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a7759e640 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c790960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9a70014620 tx=0x7f9a70008680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3bbb5fae26585a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a6d7fa640 1 -- 172.21.15.27:0/444479807 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70012710 con 0x55f15c790580 2024-04-25T01:28:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a6d7fa640 1 -- 172.21.15.27:0/444479807 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7001a070 con 0x55f15c790580 2024-04-25T01:28:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.788+0000 7f9a6d7fa640 1 -- 172.21.15.27:0/444479807 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a700119a0 con 0x55f15c790580 2024-04-25T01:28:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/444479807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 msgr2=0x55f15c790960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c790960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9a70014620 tx=0x7f9a70008680 comp rx=0 tx=0).stop 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/444479807 shutdown_connections 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c79b570 0x55f15c79d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790ea0 0x55f15c79b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/444479807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c790960 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/444479807 >> 172.21.15.27:0/444479807 conn(0x55f15c78c8a0 msgr2=0x55f15c78cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/444479807 shutdown_connections 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/444479807 wait complete. 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- start start 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 Processor -- start 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c79aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c798190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 0x55f15c7986d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f15c79e360 con 0x55f15c790580 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f15c6b1050 con 0x55f15c79b570 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f15c79e4e0 con 0x55f15c790ea0 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7fd6f740 10 client.0 _ll_register_callbacks cb 0x55f15c408be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.789+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c79aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c79aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47910/0 (socket says 172.21.15.27:47910) 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 -- 172.21.15.27:0/4293667163 learned_addr learned my addr 172.21.15.27:0/4293667163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a76d9d640 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c798190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 0x55f15c7986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 -- 172.21.15.27:0/4293667163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 msgr2=0x55f15c798190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c798190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 -- 172.21.15.27:0/4293667163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 msgr2=0x55f15c7986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 0x55f15c7986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 -- 172.21.15.27:0/4293667163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f15c682dc0 con 0x55f15c790580 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7759e640 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c79aea0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9a700084d0 tx=0x7f9a70011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97f71d6d61ab89d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a5e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4293667163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70012510 con 0x55f15c790580 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4293667163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f9a700126b0 con 0x55f15c790580 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4293667163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a7001a070 con 0x55f15c790580 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4293667163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9a70004a40 con 0x55f15c790580 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7fd6f740 10 client.16353 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4293667163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f15c4093f0 con 0x55f15c790580 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.790+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4293667163 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7001a070 con 0x55f15c790580 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4293667163 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f9a70005020 con 0x55f15c790580 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a7fd6f740 10 client.16353 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a7fd6f740 10 client.16353 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a7fd6f740 20 client.16353 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f15c7ab4f0 0x55f15c7ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a7fd6f740 4 client.16353 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4293667163 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f15c702bb0 con 0x55f15c7ab4f0 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a76d9d640 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f15c7ab4f0 0x55f15c7ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a76d9d640 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f15c7ab4f0 0x55f15c7ad8e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=363e92bcb32ddc93 server_cookie=2acd20fbd72d2c0c in_seq=0 out_seq=0 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.791+0000 7f9a5e7fc640 10 client.16353 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4293667163 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f15c702bb0 con 0x55f15c7ab4f0 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a5e7fc640 10 client.16353 handle_command_reply: tid=0 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 client.16353 shutdown 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4293667163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 msgr2=0x55f15c79aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c79aea0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9a700084d0 tx=0x7f9a70011660 comp rx=0 tx=0).stop 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4293667163 shutdown_connections 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f15c7ab4f0 0x55f15c7ad8e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 0x55f15c7986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c798190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a5e7fc640 0 client.16353 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:28:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4293667163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c79aea0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4293667163 >> 172.21.15.27:0/4293667163 conn(0x55f15c78c8a0 msgr2=0x55f15c7926a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4293667163 shutdown_connections 2024-04-25T01:28:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.792+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4293667163 wait complete. 2024-04-25T01:28:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.793+0000 7f9a7fd6f740 20 client.16353 trim_cache size 0 max 16384 2024-04-25T01:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7fd6f740 1 Processor -- start 2024-04-25T01:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7fd6f740 1 -- start start 2024-04-25T01:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790580 0x55f15c7a7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c792eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c79b570 0x55f15c79a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9a781b3610 con 0x55f15c79b570 2024-04-25T01:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9a781b0ee0 con 0x55f15c790580 2024-04-25T01:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a781aed30 con 0x55f15c790ea0 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c79b570 0x55f15c79a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c79b570 0x55f15c79a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47912/0 (socket says 172.21.15.27:47912) 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.806+0000 7f9a7759e640 1 -- 172.21.15.27:0/1686184208 learned_addr learned my addr 172.21.15.27:0/1686184208 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a7759e640 1 -- 172.21.15.27:0/1686184208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 msgr2=0x55f15c792eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a7759e640 1 --2- 172.21.15.27:0/1686184208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c792eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a7759e640 1 -- 172.21.15.27:0/1686184208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790580 msgr2=0x55f15c7a7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a7759e640 1 --2- 172.21.15.27:0/1686184208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790580 0x55f15c7a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a7759e640 1 -- 172.21.15.27:0/1686184208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f15c4d37d0 con 0x55f15c79b570 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a7759e640 1 --2- 172.21.15.27:0/1686184208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c79b570 0x55f15c79a9d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9a70008830 tx=0x7f9a70008460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7048374369293f6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a6cff9640 1 -- 172.21.15.27:0/1686184208 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70011a50 con 0x55f15c79b570 2024-04-25T01:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a6cff9640 1 -- 172.21.15.27:0/1686184208 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7001a070 con 0x55f15c79b570 2024-04-25T01:28:45.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a6cff9640 1 -- 172.21.15.27:0/1686184208 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70004be0 con 0x55f15c79b570 2024-04-25T01:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/1686184208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c79b570 msgr2=0x55f15c79a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.807+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/1686184208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c79b570 0x55f15c79a9d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9a70008830 tx=0x7f9a70008460 comp rx=0 tx=0).stop 2024-04-25T01:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/1686184208 shutdown_connections 2024-04-25T01:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/1686184208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c79b570 0x55f15c79a9d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/1686184208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c792eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/1686184208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c790580 0x55f15c7a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/1686184208 >> 172.21.15.27:0/1686184208 conn(0x55f15c78c8a0 msgr2=0x55f15c7925f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/1686184208 shutdown_connections 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/1686184208 wait complete. 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 -- start start 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 Processor -- start 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c7a3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c794a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 0x55f15c794f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9a781af3a0 con 0x55f15c790580 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9a781b0810 con 0x55f15c79b570 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a781afa70 con 0x55f15c790ea0 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7fd6f740 10 client.0 _ll_register_callbacks cb 0x7f9a70014500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c7a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c7a3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47924/0 (socket says 172.21.15.27:47924) 2024-04-25T01:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a7759e640 1 -- 172.21.15.27:0/4221027343 learned_addr learned my addr 172.21.15.27:0/4221027343 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.808+0000 7f9a76d9d640 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c794a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7759e640 1 -- 172.21.15.27:0/4221027343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 msgr2=0x55f15c794a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7759e640 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c794a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7759e640 1 -- 172.21.15.27:0/4221027343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 msgr2=0x55f15c794f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7759e640 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 0x55f15c794f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7759e640 1 -- 172.21.15.27:0/4221027343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f15c795550 con 0x55f15c790580 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7759e640 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c7a3920 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f9a70005040 tx=0x7f9a700143f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336fbc8d0bc95fd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a5e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4221027343 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70011500 con 0x55f15c790580 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4221027343 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f9a70004be0 con 0x55f15c790580 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4221027343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a7001a070 con 0x55f15c790580 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4221027343 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9a70004de0 con 0x55f15c790580 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 10 client.16359 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4221027343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f15c811680 con 0x55f15c790580 2024-04-25T01:28:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4221027343 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7001a070 con 0x55f15c790580 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4221027343 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f9a7000e430 con 0x55f15c790580 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 10 client.16359 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 10 client.16359 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 20 client.16359 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f15c7ab4f0 0x55f15c813ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 4 client.16359 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.809+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4221027343 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f15c6fcf00 con 0x55f15c7ab4f0 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a76d9d640 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f15c7ab4f0 0x55f15c813ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a5e7fc640 10 client.16359 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a76d9d640 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f15c7ab4f0 0x55f15c813ca0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5470cfd022f4df56 server_cookie=ef3d363cfa76ac2b in_seq=0 out_seq=0 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a5e7fc640 1 -- 172.21.15.27:0/4221027343 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f15c6fcf00 con 0x55f15c7ab4f0 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a5e7fc640 10 client.16359 handle_command_reply: tid=0 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a7fd6f740 1 client.16359 shutdown 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4221027343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 msgr2=0x55f15c7a3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c7a3920 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f9a70005040 tx=0x7f9a700143f0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4221027343 shutdown_connections 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.810+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55f15c7ab4f0 0x55f15c813ca0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f15c79b570 0x55f15c794f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f15c790ea0 0x55f15c794a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7fd6f740 1 --2- 172.21.15.27:0/4221027343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f15c790580 0x55f15c7a3920 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a5e7fc640 0 client.16359 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4221027343 >> 172.21.15.27:0/4221027343 conn(0x55f15c78c8a0 msgr2=0x55f15c78b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4221027343 shutdown_connections 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7fd6f740 1 -- 172.21.15.27:0/4221027343 wait complete. 2024-04-25T01:28:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7fd6f740 20 client.16359 trim_cache size 0 max 16384 2024-04-25T01:28:45.798 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:28:45.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 -- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9a640a2d30 msgr2=0x7f9a640a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9a640a2d30 0x7f9a640a51f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9a6009b070 tx=0x7f9a600bb040 comp rx=0 tx=0).stop 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 -- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a7806a780 msgr2=0x7f9a78100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a7806a780 0x7f9a78100bb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000d770 tx=0x7f9a6000dc30 comp rx=0 tx=0).stop 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 -- 172.21.15.27:0/2240945670 shutdown_connections 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9a78107730 0x7f9a781010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9a640a2d30 0x7f9a640a51f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a7806a780 0x7f9a78100bb0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 --2- 172.21.15.27:0/2240945670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a78069dc0 0x7f9a78100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 -- 172.21.15.27:0/2240945670 >> 172.21.15.27:0/2240945670 conn(0x7f9a780776a0 msgr2=0x7f9a780fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 -- 172.21.15.27:0/2240945670 shutdown_connections 2024-04-25T01:28:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.811+0000 7f9a7d05d640 1 -- 172.21.15.27:0/2240945670 wait complete. 2024-04-25T01:28:45.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 -- 172.21.15.27:0/3897797159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc1020c0 msgr2=0x7f69bc10e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 --2- 172.21.15.27:0/3897797159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc1020c0 0x7f69bc10e5d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009cd0 tx=0x7f69b402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 -- 172.21.15.27:0/3897797159 shutdown_connections 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 --2- 172.21.15.27:0/3897797159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc1020c0 0x7f69bc10e5d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 --2- 172.21.15.27:0/3897797159 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69bc101700 0x7f69bc101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 --2- 172.21.15.27:0/3897797159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69bc107710 0x7f69bc107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 -- 172.21.15.27:0/3897797159 >> 172.21.15.27:0/3897797159 conn(0x7f69bc0fd1b0 msgr2=0x7f69bc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 -- 172.21.15.27:0/3897797159 shutdown_connections 2024-04-25T01:28:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 -- 172.21.15.27:0/3897797159 wait complete. 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 Processor -- start 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 -- start start 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc101700 0x7f69bc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.927+0000 7f69c462c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69bc1020c0 0x7f69bc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c462c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69bc107710 0x7f69bc19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c462c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc111b70 con 0x7f69bc101700 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c462c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1119f0 con 0x7f69bc107710 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c462c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc111cf0 con 0x7f69bc1020c0 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc101700 0x7f69bc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc101700 0x7f69bc198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47938/0 (socket says 172.21.15.27:47938) 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 -- 172.21.15.27:0/3855012308 learned_addr learned my addr 172.21.15.27:0/3855012308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69bc1020c0 0x7f69bc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69bc107710 0x7f69bc19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 -- 172.21.15.27:0/3855012308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69bc1020c0 msgr2=0x7f69bc198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69bc1020c0 0x7f69bc198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 -- 172.21.15.27:0/3855012308 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69bc107710 msgr2=0x7f69bc19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69bc107710 0x7f69bc19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 -- 172.21.15.27:0/3855012308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69bc19d000 con 0x7f69bc101700 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c23c8640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc101700 0x7f69bc198010 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c970 tx=0x7f69ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69b37fe640 1 -- 172.21.15.27:0/3855012308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac007bb0 con 0x7f69bc101700 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69b37fe640 1 -- 172.21.15.27:0/3855012308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69ac007d50 con 0x7f69bc101700 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c462c640 1 -- 172.21.15.27:0/3855012308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69bc19d2f0 con 0x7f69bc101700 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69b37fe640 1 -- 172.21.15.27:0/3855012308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00f450 con 0x7f69bc101700 2024-04-25T01:28:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.928+0000 7f69c462c640 1 -- 172.21.15.27:0/3855012308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69bc1a4b80 con 0x7f69bc101700 2024-04-25T01:28:45.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69b37fe640 1 -- 172.21.15.27:0/3855012308 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f69ac005020 con 0x7f69bc101700 2024-04-25T01:28:45.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69c740e740 1 Processor -- start 2024-04-25T01:28:45.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69c740e740 1 -- start start 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69c740e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5580 0x565456fe5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69c740e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69c740e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456ff0570 0x565456ff2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69c740e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565456f4a670 con 0x565456fe5ea0 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69c740e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x565456ef2970 con 0x565456ff0570 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.931+0000 7f69c740e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x565456f46630 con 0x565456fe5580 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69b37fe640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69a80a2d00 0x7f69a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47948/0 (socket says 172.21.15.27:47948) 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 -- 172.21.15.27:0/3553030112 learned_addr learned my addr 172.21.15.27:0/3553030112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5580 0x565456fe5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456ff0570 0x565456ff2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69a80a2d00 0x7f69a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69b37fe640 1 -- 172.21.15.27:0/3855012308 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f69ac0cf570 con 0x7f69bc101700 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 -- 172.21.15.27:0/3553030112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5580 msgr2=0x565456fe5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5580 0x565456fe5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 -- 172.21.15.27:0/3553030112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456ff0570 msgr2=0x565456ff2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456ff0570 0x565456ff2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 -- 172.21.15.27:0/3553030112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565456f16400 con 0x565456fe5ea0 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c23c8640 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff0030 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0cc8a0 tx=0x7f69ac004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0002001d06cb63e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69a80a2d00 0x7f69a80a51c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f69b4006010 tx=0x7f69b40095c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69b0bfe640 1 -- 172.21.15.27:0/3553030112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac01d070 con 0x565456fe5ea0 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69b0bfe640 1 -- 172.21.15.27:0/3553030112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69ac09be00 con 0x565456fe5ea0 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.932+0000 7f69b0bfe640 1 -- 172.21.15.27:0/3553030112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0ccbb0 con 0x565456fe5ea0 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 -- 172.21.15.27:0/3553030112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 msgr2=0x565456ff0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff0030 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0cc8a0 tx=0x7f69ac004a20 comp rx=0 tx=0).stop 2024-04-25T01:28:45.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 -- 172.21.15.27:0/3553030112 shutdown_connections 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456ff0570 0x565456ff2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff0030 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 --2- 172.21.15.27:0/3553030112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5580 0x565456fe5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 -- 172.21.15.27:0/3553030112 >> 172.21.15.27:0/3553030112 conn(0x565456fe18a0 msgr2=0x565456fe1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 -- 172.21.15.27:0/3553030112 shutdown_connections 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.933+0000 7f69c740e740 1 -- 172.21.15.27:0/3553030112 wait complete. 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 1 -- start start 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 1 Processor -- start 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456fefe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5ea0 0x565456fed180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456ff0570 0x565456fed6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565456ff3360 con 0x565456ff0570 2024-04-25T01:28:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x565456f06050 con 0x565456fe5580 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x565456ff34e0 con 0x565456fe5ea0 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c740e740 10 client.0 _ll_register_callbacks cb 0x565456c5dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456ff0570 0x565456fed6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456ff0570 0x565456fed6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47954/0 (socket says 172.21.15.27:47954) 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 -- 172.21.15.27:0/2699768635 learned_addr learned my addr 172.21.15.27:0/2699768635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5ea0 0x565456fed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c23c8640 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456fefe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 -- 172.21.15.27:0/2699768635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5ea0 msgr2=0x565456fed180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5ea0 0x565456fed180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 -- 172.21.15.27:0/2699768635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 msgr2=0x565456fefe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456fefe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 -- 172.21.15.27:0/2699768635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565456ed7dc0 con 0x565456ff0570 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c23c8640 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456fefe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456ff0570 0x565456fed6c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009cd0 tx=0x7f69b402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa3fa68ce8be886 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.934+0000 7f69a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69a5ffb640 1 -- 172.21.15.27:0/2699768635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4051070 con 0x565456ff0570 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69a5ffb640 1 -- 172.21.15.27:0/2699768635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f69b4004390 con 0x565456ff0570 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 1 -- 172.21.15.27:0/2699768635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69ac09be00 con 0x565456ff0570 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69a5ffb640 1 -- 172.21.15.27:0/2699768635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f69b4004590 con 0x565456ff0570 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 10 client.16371 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 1 -- 172.21.15.27:0/2699768635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565456c5e3f0 con 0x565456ff0570 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69a5ffb640 1 -- 172.21.15.27:0/2699768635 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69ac09be00 con 0x565456ff0570 2024-04-25T01:28:45.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69a5ffb640 1 -- 172.21.15.27:0/2699768635 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f69b4049040 con 0x565456ff0570 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 10 client.16371 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 10 client.16371 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 20 client.16371 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x565457000590 0x5654570029f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 4 client.16371 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.935+0000 7f69c740e740 1 -- 172.21.15.27:0/2699768635 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565456f57bb0 con 0x565457000590 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.936+0000 7f69c23c8640 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x565457000590 0x5654570029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.936+0000 7f69c23c8640 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x565457000590 0x5654570029f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=627306e8d1caa5ef server_cookie=c921aad4de29bf5f in_seq=0 out_seq=0 2024-04-25T01:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.936+0000 7f69a5ffb640 10 client.16371 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69a5ffb640 1 -- 172.21.15.27:0/2699768635 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565456f57bb0 con 0x565457000590 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69a5ffb640 10 client.16371 handle_command_reply: tid=0 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 client.16371 shutdown 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 -- 172.21.15.27:0/2699768635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456ff0570 msgr2=0x565456fed6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456ff0570 0x565456fed6c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009cd0 tx=0x7f69b402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 -- 172.21.15.27:0/2699768635 shutdown_connections 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x565457000590 0x5654570029f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456ff0570 0x565456fed6c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69a5ffb640 0 client.16371 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456fe5ea0 0x565456fed180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 --2- 172.21.15.27:0/2699768635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456fefe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 -- 172.21.15.27:0/2699768635 >> 172.21.15.27:0/2699768635 conn(0x565456fe18a0 msgr2=0x565456fe0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 -- 172.21.15.27:0/2699768635 shutdown_connections 2024-04-25T01:28:45.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 1 -- 172.21.15.27:0/2699768635 wait complete. 2024-04-25T01:28:45.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.937+0000 7f69c740e740 20 client.16371 trim_cache size 0 max 16384 2024-04-25T01:28:45.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.950+0000 7f69c740e740 1 Processor -- start 2024-04-25T01:28:45.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.950+0000 7f69c740e740 1 -- start start 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c740e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ffc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c740e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x5654570082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c740e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565457008800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c740e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1b15d0 con 0x565456fe5ea0 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c740e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1aeea0 con 0x565456fe5580 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c740e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1accf0 con 0x565456ff0570 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x5654570082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x5654570082c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47956/0 (socket says 172.21.15.27:47956) 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 -- 172.21.15.27:0/2696033768 learned_addr learned my addr 172.21.15.27:0/2696033768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565457008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c23c8640 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ffc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 -- 172.21.15.27:0/2696033768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 msgr2=0x565457008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565457008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 -- 172.21.15.27:0/2696033768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 msgr2=0x565456ffc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ffc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 -- 172.21.15.27:0/2696033768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565456d287d0 con 0x565456fe5ea0 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565457008800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c23c8640 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ffc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x5654570082c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f69b80045a0 tx=0x7f69b800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3054940b4391be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69a7fff640 1 -- 172.21.15.27:0/2696033768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8020070 con 0x565456fe5ea0 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69a7fff640 1 -- 172.21.15.27:0/2696033768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69b8003cd0 con 0x565456fe5ea0 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.951+0000 7f69a7fff640 1 -- 172.21.15.27:0/2696033768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b801b760 con 0x565456fe5ea0 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 -- 172.21.15.27:0/2696033768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 msgr2=0x5654570082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x5654570082c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f69b80045a0 tx=0x7f69b800bcb0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 -- 172.21.15.27:0/2696033768 shutdown_connections 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565457008800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x5654570082c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 --2- 172.21.15.27:0/2696033768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ffc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 -- 172.21.15.27:0/2696033768 >> 172.21.15.27:0/2696033768 conn(0x565456fe18a0 msgr2=0x565456fe02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 -- 172.21.15.27:0/2696033768 shutdown_connections 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 -- 172.21.15.27:0/2696033768 wait complete. 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 -- start start 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.952+0000 7f69c740e740 1 Processor -- start 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c740e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ff8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c740e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c740e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565456ff6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c740e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1ad360 con 0x565456fe5ea0 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c740e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1ae7d0 con 0x565456fe5580 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c740e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1ada30 con 0x565456ff0570 2024-04-25T01:28:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c740e740 10 client.0 _ll_register_callbacks cb 0x7f69b4031430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47962/0 (socket says 172.21.15.27:47962) 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 -- 172.21.15.27:0/1399114592 learned_addr learned my addr 172.21.15.27:0/1399114592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c23c8640 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ff8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565456ff6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 -- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 msgr2=0x565456ff6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565456ff6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 -- 172.21.15.27:0/1399114592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 msgr2=0x565456ff8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ff8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 -- 172.21.15.27:0/1399114592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565456ff6cc0 con 0x565456fe5ea0 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c2bc9640 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565456ff6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c1bc7640 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff5f30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f69b80058c0 tx=0x7f69b8003b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4ff104037113cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69a5ffb640 1 -- 172.21.15.27:0/1399114592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8029020 con 0x565456fe5ea0 2024-04-25T01:28:45.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69a5ffb640 1 -- 172.21.15.27:0/1399114592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f69b8011640 con 0x565456fe5ea0 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.953+0000 7f69c740e740 1 -- 172.21.15.27:0/1399114592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69b8003cd0 con 0x565456fe5ea0 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69a5ffb640 1 -- 172.21.15.27:0/1399114592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f69b8034050 con 0x565456fe5ea0 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c740e740 10 client.16377 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c740e740 1 -- 172.21.15.27:0/1399114592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56545706af50 con 0x565456fe5ea0 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69a5ffb640 1 -- 172.21.15.27:0/1399114592 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69b8003cd0 con 0x565456fe5ea0 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69a5ffb640 1 -- 172.21.15.27:0/1399114592 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f69b801a070 con 0x565456fe5ea0 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c740e740 10 client.16377 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c740e740 10 client.16377 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c740e740 20 client.16377 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c740e740 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56545706b5c0 0x56545706d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c740e740 4 client.16377 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c740e740 1 -- 172.21.15.27:0/1399114592 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565456f51f00 con 0x56545706b5c0 2024-04-25T01:28:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.954+0000 7f69c23c8640 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56545706b5c0 0x56545706d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c23c8640 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56545706b5c0 0x56545706d9e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18874bc852e3dc45 server_cookie=162428aa67435926 in_seq=0 out_seq=0 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69a5ffb640 10 client.16377 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69a5ffb640 1 -- 172.21.15.27:0/1399114592 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565456f51f00 con 0x56545706b5c0 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69a5ffb640 10 client.16377 handle_command_reply: tid=0 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 client.16377 shutdown 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 -- 172.21.15.27:0/1399114592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 msgr2=0x565456ff5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff5f30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f69b80058c0 tx=0x7f69b8003b50 comp rx=0 tx=0).stop 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 -- 172.21.15.27:0/1399114592 shutdown_connections 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56545706b5c0 0x56545706d9e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x565456ff0570 0x565456ff6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565456fe5ea0 0x565456ff5f30 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69a5ffb640 0 client.16377 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 --2- 172.21.15.27:0/1399114592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x565456fe5580 0x565456ff8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 -- 172.21.15.27:0/1399114592 >> 172.21.15.27:0/1399114592 conn(0x565456fe18a0 msgr2=0x565456fe0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 -- 172.21.15.27:0/1399114592 shutdown_connections 2024-04-25T01:28:45.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 1 -- 172.21.15.27:0/1399114592 wait complete. 2024-04-25T01:28:45.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.955+0000 7f69c740e740 20 client.16377 trim_cache size 0 max 16384 2024-04-25T01:28:45.942 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 -- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69a80a2d00 msgr2=0x7f69a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69a80a2d00 0x7f69a80a51c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f69b4006010 tx=0x7f69b40095c0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 -- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc101700 msgr2=0x7f69bc198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc101700 0x7f69bc198010 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c970 tx=0x7f69ac00ce30 comp rx=0 tx=0).stop 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 -- 172.21.15.27:0/3855012308 shutdown_connections 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69bc107710 0x7f69bc19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69bc1020c0 0x7f69bc198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f69a80a2d00 0x7f69a80a51c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 --2- 172.21.15.27:0/3855012308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69bc101700 0x7f69bc198010 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 -- 172.21.15.27:0/3855012308 >> 172.21.15.27:0/3855012308 conn(0x7f69bc0fd1b0 msgr2=0x7f69bc0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 -- 172.21.15.27:0/3855012308 shutdown_connections 2024-04-25T01:28:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:45.956+0000 7f69c462c640 1 -- 172.21.15.27:0/3855012308 wait complete. 2024-04-25T01:28:45.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:28:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 -- 172.21.15.27:0/1491503341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd20101710 msgr2=0x7fdd20101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 --2- 172.21.15.27:0/1491503341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd20101710 0x7fdd20101b90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009930 tx=0x7fdd1002a420 comp rx=0 tx=0).stop 2024-04-25T01:28:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 -- 172.21.15.27:0/1491503341 shutdown_connections 2024-04-25T01:28:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 --2- 172.21.15.27:0/1491503341 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd201020d0 0x7fdd2010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 --2- 172.21.15.27:0/1491503341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd20101710 0x7fdd20101b90 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 --2- 172.21.15.27:0/1491503341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd20107720 0x7fdd20107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 -- 172.21.15.27:0/1491503341 >> 172.21.15.27:0/1491503341 conn(0x7fdd200fd180 msgr2=0x7fdd200ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 -- 172.21.15.27:0/1491503341 shutdown_connections 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 -- 172.21.15.27:0/1491503341 wait complete. 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 Processor -- start 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.071+0000 7fdd27f32640 1 -- start start 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd27f32640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd20101710 0x7fdd20073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd27f32640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd201020d0 0x7fdd20073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd27f32640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd20107720 0x7fdd200744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd27f32640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd20111af0 con 0x7fdd201020d0 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd27f32640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdd20111970 con 0x7fdd20101710 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd27f32640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd20111c70 con 0x7fdd20107720 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd201020d0 0x7fdd20073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd201020d0 0x7fdd20073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47978/0 (socket says 172.21.15.27:47978) 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 -- 172.21.15.27:0/423470210 learned_addr learned my addr 172.21.15.27:0/423470210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd25cce640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd20101710 0x7fdd20073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd264cf640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd20107720 0x7fdd200744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 -- 172.21.15.27:0/423470210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd20107720 msgr2=0x7fdd200744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd20107720 0x7fdd200744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 -- 172.21.15.27:0/423470210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd20101710 msgr2=0x7fdd20073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd20101710 0x7fdd20073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 -- 172.21.15.27:0/423470210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd201a6510 con 0x7fdd201020d0 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd264cf640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd20107720 0x7fdd200744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd254cd640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd201020d0 0x7fdd20073f80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009230 tx=0x7fdd10004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd16ffd640 1 -- 172.21.15.27:0/423470210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1003f030 con 0x7fdd201020d0 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd16ffd640 1 -- 172.21.15.27:0/423470210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd10003ad0 con 0x7fdd201020d0 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd27f32640 1 -- 172.21.15.27:0/423470210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd201a67a0 con 0x7fdd201020d0 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd16ffd640 1 -- 172.21.15.27:0/423470210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10035070 con 0x7fdd201020d0 2024-04-25T01:28:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.072+0000 7fdd27f32640 1 -- 172.21.15.27:0/423470210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd201a6ce0 con 0x7fdd201020d0 2024-04-25T01:28:46.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd16ffd640 1 -- 172.21.15.27:0/423470210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdd10042030 con 0x7fdd201020d0 2024-04-25T01:28:46.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd2ac3d740 1 Processor -- start 2024-04-25T01:28:46.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd2ac3d740 1 -- start start 2024-04-25T01:28:46.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981c0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981cb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981cb570 0x55e0981cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e098125670 con 0x55e0981c0580 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e0980cd970 con 0x55e0981cb570 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.075+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e098121630 con 0x55e0981c0ea0 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981c0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981c0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47990/0 (socket says 172.21.15.27:47990) 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 -- 172.21.15.27:0/817542264 learned_addr learned my addr 172.21.15.27:0/817542264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd25cce640 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd16ffd640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdd0c0a2db0 0x7fdd0c0a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 -- 172.21.15.27:0/817542264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 msgr2=0x55e0981cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd254cd640 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981cb570 0x55e0981cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 -- 172.21.15.27:0/817542264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981cb570 msgr2=0x55e0981cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd16ffd640 1 -- 172.21.15.27:0/423470210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdd100efd70 con 0x7fdd201020d0 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981cb570 0x55e0981cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 -- 172.21.15.27:0/817542264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0980f1400 con 0x55e0981c0580 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd254cd640 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981cb570 0x55e0981cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdd0c0a2db0 0x7fdd0c0a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd264cf640 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981c0960 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800f4b0 tx=0x7fdd1800f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38040eb244b4a2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd0bfff640 1 -- 172.21.15.27:0/817542264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18018070 con 0x55e0981c0580 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd0bfff640 1 -- 172.21.15.27:0/817542264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd18021070 con 0x55e0981c0580 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.076+0000 7fdd0bfff640 1 -- 172.21.15.27:0/817542264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18025b20 con 0x55e0981c0580 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/817542264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 msgr2=0x55e0981c0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd264cf640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdd0c0a2db0 0x7fdd0c0a5270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd18020000 tx=0x7fdd1801e000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:46.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981c0960 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800f4b0 tx=0x7fdd1800f970 comp rx=0 tx=0).stop 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/817542264 shutdown_connections 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981cb570 0x55e0981cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/817542264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981c0960 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/817542264 >> 172.21.15.27:0/817542264 conn(0x55e0981bc8a0 msgr2=0x55e0981bcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/817542264 shutdown_connections 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.077+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/817542264 wait complete. 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 1 -- start start 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 1 Processor -- start 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981caf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981cb570 0x55e0981c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e0981ce360 con 0x55e0981c0580 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e0980e1050 con 0x55e0981c0ea0 2024-04-25T01:28:46.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0981ce4e0 con 0x55e0981cb570 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd2ac3d740 10 client.0 _ll_register_callbacks cb 0x55e097e38be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd25cce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd25cce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48000/0 (socket says 172.21.15.27:48000) 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd25cce640 1 -- 172.21.15.27:0/90562810 learned_addr learned my addr 172.21.15.27:0/90562810 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd264cf640 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981cb570 0x55e0981c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd254cd640 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd25cce640 1 -- 172.21.15.27:0/90562810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981cb570 msgr2=0x55e0981c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd25cce640 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981cb570 0x55e0981c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd25cce640 1 -- 172.21.15.27:0/90562810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 msgr2=0x55e0981c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd25cce640 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.078+0000 7fdd25cce640 1 -- 172.21.15.27:0/90562810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0980b2dc0 con 0x55e0981c0580 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd25cce640 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981caf00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c010a50 tx=0x7fdd1c010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657c83a98d5f7bd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd097fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd097fa640 1 -- 172.21.15.27:0/90562810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c01e060 con 0x55e0981c0580 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd097fa640 1 -- 172.21.15.27:0/90562810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdd1c00fbd0 con 0x55e0981c0580 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/90562810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd18021070 con 0x55e0981c0580 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd097fa640 1 -- 172.21.15.27:0/90562810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdd1c025050 con 0x55e0981c0580 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 10 client.16389 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/90562810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e097e393f0 con 0x55e0981c0580 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd097fa640 1 -- 172.21.15.27:0/90562810 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd18021070 con 0x55e0981c0580 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd097fa640 1 -- 172.21.15.27:0/90562810 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fdd1c018400 con 0x55e0981c0580 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 10 client.16389 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 10 client.16389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 20 client.16389 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e0981db5e0 0x55e0981dda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 4 client.16389 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.079+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/90562810 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e098132bb0 con 0x55e0981db5e0 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.080+0000 7fdd254cd640 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e0981db5e0 0x55e0981dda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.080+0000 7fdd097fa640 10 client.16389 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.080+0000 7fdd254cd640 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e0981db5e0 0x55e0981dda10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=946b536752b67634 server_cookie=a20a069b784a2eda in_seq=0 out_seq=0 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd097fa640 1 -- 172.21.15.27:0/90562810 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e098132bb0 con 0x55e0981db5e0 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd097fa640 10 client.16389 handle_command_reply: tid=0 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 client.16389 shutdown 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/90562810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 msgr2=0x55e0981caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981caf00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c010a50 tx=0x7fdd1c010f10 comp rx=0 tx=0).stop 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/90562810 shutdown_connections 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e0981db5e0 0x55e0981dda10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981cb570 0x55e0981c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/90562810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981c0580 0x55e0981caf00 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd097fa640 0 client.16389 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/90562810 >> 172.21.15.27:0/90562810 conn(0x55e0981bc8a0 msgr2=0x55e0981bbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/90562810 shutdown_connections 2024-04-25T01:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/90562810 wait complete. 2024-04-25T01:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.081+0000 7fdd2ac3d740 20 client.16389 trim_cache size 0 max 16384 2024-04-25T01:28:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.094+0000 7fdd2ac3d740 1 Processor -- start 2024-04-25T01:28:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd2ac3d740 1 -- start start 2024-04-25T01:28:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0580 0x55e0981caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981d3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd201b3680 con 0x55e0981cb570 2024-04-25T01:28:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdd201b0f50 con 0x55e0981c0580 2024-04-25T01:28:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd201aeda0 con 0x55e0981c0ea0 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48010/0 (socket says 172.21.15.27:48010) 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 -- 172.21.15.27:0/3640641157 learned_addr learned my addr 172.21.15.27:0/3640641157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd254cd640 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981d3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd25cce640 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0580 0x55e0981caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 -- 172.21.15.27:0/3640641157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 msgr2=0x55e0981d3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981d3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 -- 172.21.15.27:0/3640641157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0580 msgr2=0x55e0981caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0580 0x55e0981caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 -- 172.21.15.27:0/3640641157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e097f037d0 con 0x55e0981cb570 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd254cd640 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981d3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd25cce640 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0580 0x55e0981caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.095+0000 7fdd264cf640 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d3ca0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdd18011000 tx=0x7fdd18027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5678c7f6cd7a937 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd0b7fe640 1 -- 172.21.15.27:0/3640641157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18033070 con 0x55e0981cb570 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd0b7fe640 1 -- 172.21.15.27:0/3640641157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd18021040 con 0x55e0981cb570 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd0b7fe640 1 -- 172.21.15.27:0/3640641157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18018040 con 0x55e0981cb570 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/3640641157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 msgr2=0x55e0981d3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d3ca0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdd18011000 tx=0x7fdd18027000 comp rx=0 tx=0).stop 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/3640641157 shutdown_connections 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d3ca0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0ea0 0x55e0981d3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/3640641157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0580 0x55e0981caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/3640641157 >> 172.21.15.27:0/3640641157 conn(0x55e0981bc8a0 msgr2=0x55e0981bb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/3640641157 shutdown_connections 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.096+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/3640641157 wait complete. 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 1 -- start start 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 1 Processor -- start 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0580 0x55e0981d3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd201af410 con 0x55e0981cb570 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdd201b0880 con 0x55e0981c0ea0 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd201afae0 con 0x55e0981c0580 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd2ac3d740 10 client.0 _ll_register_callbacks cb 0x7fdd1c00a740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd264cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd264cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48022/0 (socket says 172.21.15.27:48022) 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd264cf640 1 -- 172.21.15.27:0/1879972323 learned_addr learned my addr 172.21.15.27:0/1879972323 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd25cce640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0580 0x55e0981d3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd254cd640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd264cf640 1 -- 172.21.15.27:0/1879972323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0580 msgr2=0x55e0981d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.097+0000 7fdd264cf640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0580 0x55e0981d3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd264cf640 1 -- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 msgr2=0x55e0981d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd264cf640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981d3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd264cf640 1 -- 172.21.15.27:0/1879972323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0981d2090 con 0x55e0981cb570 2024-04-25T01:28:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd25cce640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0580 0x55e0981d3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd254cd640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd264cf640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d19f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdd18031040 tx=0x7fdd18009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b368d4ee81465da9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd097fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd097fa640 1 -- 172.21.15.27:0/1879972323 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1803c060 con 0x55e0981cb570 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd097fa640 1 -- 172.21.15.27:0/1879972323 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdd1801ca70 con 0x55e0981cb570 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/1879972323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd18021040 con 0x55e0981cb570 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd097fa640 1 -- 172.21.15.27:0/1879972323 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdd18018030 con 0x55e0981cb570 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd2ac3d740 10 client.16395 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/1879972323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e0982426e0 con 0x55e0981cb570 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd097fa640 1 -- 172.21.15.27:0/1879972323 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd18021040 con 0x55e0981cb570 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd097fa640 1 -- 172.21.15.27:0/1879972323 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fdd18036070 con 0x55e0981cb570 2024-04-25T01:28:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd2ac3d740 10 client.16395 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd2ac3d740 10 client.16395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.098+0000 7fdd2ac3d740 20 client.16395 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e098242c10 0x55e098245030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd2ac3d740 4 client.16395 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/1879972323 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e09812cf00 con 0x55e098242c10 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd25cce640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e098242c10 0x55e098245030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd25cce640 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e098242c10 0x55e098245030 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ba6309888ea80c8 server_cookie=dd67514ade7a5446 in_seq=0 out_seq=0 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd097fa640 10 client.16395 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd097fa640 1 -- 172.21.15.27:0/1879972323 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e09812cf00 con 0x55e098242c10 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd097fa640 10 client.16395 handle_command_reply: tid=0 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.099+0000 7fdd2ac3d740 1 client.16395 shutdown 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/1879972323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 msgr2=0x55e0981d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d19f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdd18031040 tx=0x7fdd18009d10 comp rx=0 tx=0).stop 2024-04-25T01:28:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/1879972323 shutdown_connections 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e098242c10 0x55e098245030 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0981cb570 0x55e0981d19f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e0981c0ea0 0x55e0981d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 --2- 172.21.15.27:0/1879972323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0981c0580 0x55e0981d3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd097fa640 0 client.16395 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/1879972323 >> 172.21.15.27:0/1879972323 conn(0x55e0981bc8a0 msgr2=0x55e0981bbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/1879972323 shutdown_connections 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 1 -- 172.21.15.27:0/1879972323 wait complete. 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd2ac3d740 20 client.16395 trim_cache size 0 max 16384 2024-04-25T01:28:46.087 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:28:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 -- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdd0c0a2db0 msgr2=0x7fdd0c0a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdd0c0a2db0 0x7fdd0c0a5270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd18020000 tx=0x7fdd1801e000 comp rx=0 tx=0).stop 2024-04-25T01:28:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 -- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd201020d0 msgr2=0x7fdd20073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd201020d0 0x7fdd20073f80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009230 tx=0x7fdd10004270 comp rx=0 tx=0).stop 2024-04-25T01:28:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 -- 172.21.15.27:0/423470210 shutdown_connections 2024-04-25T01:28:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd20107720 0x7fdd200744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdd0c0a2db0 0x7fdd0c0a5270 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd201020d0 0x7fdd20073f80 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 --2- 172.21.15.27:0/423470210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd20101710 0x7fdd20073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 -- 172.21.15.27:0/423470210 >> 172.21.15.27:0/423470210 conn(0x7fdd200fd180 msgr2=0x7fdd200fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 -- 172.21.15.27:0/423470210 shutdown_connections 2024-04-25T01:28:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:46.100+0000 7fdd27f32640 1 -- 172.21.15.27:0/423470210 wait complete. 2024-04-25T01:28:46.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:46 smithi027 ceph-mon[29197]: pgmap v103: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:46.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:46 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2561122666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:28:46.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:46 smithi153 ceph-mon[32621]: pgmap v103: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:46.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:46 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2561122666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:28:46.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:46 smithi139 ceph-mon[32760]: pgmap v103: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:28:46.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:46 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2561122666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:28:47.090 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:28:47.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:28:47.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:28:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 -- 172.21.15.27:0/3906462176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c410bd00 msgr2=0x7f53c410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 --2- 172.21.15.27:0/3906462176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c410bd00 0x7f53c410e1a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009ca0 tx=0x7f53bc02a9e0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 -- 172.21.15.27:0/3906462176 shutdown_connections 2024-04-25T01:28:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 --2- 172.21.15.27:0/3906462176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c410bd00 0x7f53c410e1a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 --2- 172.21.15.27:0/3906462176 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c41032e0 0x7f53c410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 --2- 172.21.15.27:0/3906462176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c41029c0 0x7f53c4102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 -- 172.21.15.27:0/3906462176 >> 172.21.15.27:0/3906462176 conn(0x7f53c4077740 msgr2=0x7f53c4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 -- 172.21.15.27:0/3906462176 shutdown_connections 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 -- 172.21.15.27:0/3906462176 wait complete. 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 Processor -- start 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.231+0000 7f53cc561640 1 -- start start 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53cc561640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c41029c0 0x7f53c4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53cc561640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c41032e0 0x7f53c4198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53cc561640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c410bd00 0x7f53c419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53cc561640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53c4111b00 con 0x7f53c41032e0 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53cc561640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53c4111980 con 0x7f53c410bd00 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53cc561640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53c4111c80 con 0x7f53c41029c0 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c41032e0 0x7f53c4198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c41032e0 0x7f53c4198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48040/0 (socket says 172.21.15.27:48040) 2024-04-25T01:28:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 -- 172.21.15.27:0/1777111554 learned_addr learned my addr 172.21.15.27:0/1777111554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c41029c0 0x7f53c4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53caafe640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c410bd00 0x7f53c419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 -- 172.21.15.27:0/1777111554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c41029c0 msgr2=0x7f53c4198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c41029c0 0x7f53c4198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 -- 172.21.15.27:0/1777111554 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c410bd00 msgr2=0x7f53c419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c410bd00 0x7f53c419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 -- 172.21.15.27:0/1777111554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c419cfc0 con 0x7f53c41032e0 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53caafe640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c410bd00 0x7f53c419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53c9afc640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c41032e0 0x7f53c4198560 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f53b400d950 tx=0x7f53b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53bb7fe640 1 -- 172.21.15.27:0/1777111554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4017030 con 0x7f53c41032e0 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.232+0000 7f53bb7fe640 1 -- 172.21.15.27:0/1777111554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b4004020 con 0x7f53c41032e0 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.233+0000 7f53bb7fe640 1 -- 172.21.15.27:0/1777111554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b40047a0 con 0x7f53c41032e0 2024-04-25T01:28:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.233+0000 7f53cc561640 1 -- 172.21.15.27:0/1777111554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c419d2b0 con 0x7f53c41032e0 2024-04-25T01:28:47.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.233+0000 7f53cc561640 1 -- 172.21.15.27:0/1777111554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c41a4b40 con 0x7f53c41032e0 2024-04-25T01:28:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.235+0000 7f53bb7fe640 1 -- 172.21.15.27:0/1777111554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f53b40041d0 con 0x7f53c41032e0 2024-04-25T01:28:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.236+0000 7f53cf26f740 1 Processor -- start 2024-04-25T01:28:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.236+0000 7f53cf26f740 1 -- start start 2024-04-25T01:28:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.236+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c2ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.236+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2d290 0x555619c2d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.236+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c3e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.236+0000 7f53cf26f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555619b912c0 con 0x555619c2d290 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.236+0000 7f53cf26f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555619b39910 con 0x555619c3c420 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.236+0000 7f53cf26f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555619b8d280 con 0x555619c2e980 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53bb7fe640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2d80 0x7f53b00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2d290 0x555619c2d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53caafe640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2d80 0x7f53b00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2d290 0x555619c2d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48056/0 (socket says 172.21.15.27:48056) 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 -- 172.21.15.27:0/1542498380 learned_addr learned my addr 172.21.15.27:0/1542498380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53caafe640 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c2ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53bb7fe640 1 -- 172.21.15.27:0/1777111554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f53b40cf0b0 con 0x7f53c41032e0 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53c9afc640 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c3e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 -- 172.21.15.27:0/1542498380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 msgr2=0x555619c2ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c2ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 -- 172.21.15.27:0/1542498380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 msgr2=0x555619c3e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c3e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 -- 172.21.15.27:0/1542498380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555619b4afb0 con 0x555619c2d290 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53caafe640 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c2ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2d290 0x555619c2d710 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009d20 tx=0x7f53c0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67dbda22062e269b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53caafe640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2d80 0x7f53b00a5240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009430 tx=0x7f53bc03a040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53b8bfe640 1 -- 172.21.15.27:0/1542498380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c001e070 con 0x555619c2d290 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.237+0000 7f53b8bfe640 1 -- 172.21.15.27:0/1542498380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c0019b10 con 0x555619c2d290 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53b8bfe640 1 -- 172.21.15.27:0/1542498380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c000ebf0 con 0x555619c2d290 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 -- 172.21.15.27:0/1542498380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2d290 msgr2=0x555619c2d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2d290 0x555619c2d710 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009d20 tx=0x7f53c0011810 comp rx=0 tx=0).stop 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 -- 172.21.15.27:0/1542498380 shutdown_connections 2024-04-25T01:28:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c3e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2d290 0x555619c2d710 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 --2- 172.21.15.27:0/1542498380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c2ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 -- 172.21.15.27:0/1542498380 >> 172.21.15.27:0/1542498380 conn(0x555619bf5950 msgr2=0x555619c297f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 -- 172.21.15.27:0/1542498380 shutdown_connections 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.238+0000 7f53cf26f740 1 -- 172.21.15.27:0/1542498380 wait complete. 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 1 -- start start 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 1 Processor -- start 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 0x555619c39320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c39860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c39da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555619c3f2b0 con 0x555619c2e980 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555619b4ce90 con 0x555619c3c420 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555619c3f430 con 0x555619c2d290 2024-04-25T01:28:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53cf26f740 10 client.0 _ll_register_callbacks cb 0x555619b88970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53c9afc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c39860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53c9afc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c39860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48058/0 (socket says 172.21.15.27:48058) 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53c9afc640 1 -- 172.21.15.27:0/297213530 learned_addr learned my addr 172.21.15.27:0/297213530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53caafe640 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c39da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53c9afc640 1 -- 172.21.15.27:0/297213530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 msgr2=0x555619c39320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 0x555619c39320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53c9afc640 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 0x555619c39320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53c9afc640 1 -- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 msgr2=0x555619c39da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53c9afc640 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c39da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53c9afc640 1 -- 172.21.15.27:0/297213530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55561996d7e0 con 0x555619c2e980 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.239+0000 7f53caafe640 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c39da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53c9afc640 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c39860 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f53b4005e00 tx=0x7f53b409b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e929e9c5da4ac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53adffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53adffb640 1 -- 172.21.15.27:0/297213530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b409b240 con 0x555619c2e980 2024-04-25T01:28:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53adffb640 1 -- 172.21.15.27:0/297213530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f53b40bb0e0 con 0x555619c2e980 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 1 -- 172.21.15.27:0/297213530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53c0019b10 con 0x555619c2e980 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53adffb640 1 -- 172.21.15.27:0/297213530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f53b409f5f0 con 0x555619c2e980 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 10 client.16407 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 1 -- 172.21.15.27:0/297213530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5556199cdf70 con 0x555619c2e980 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53adffb640 1 -- 172.21.15.27:0/297213530 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c0019b10 con 0x555619c2e980 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53adffb640 1 -- 172.21.15.27:0/297213530 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f53b40ccbc0 con 0x555619c2e980 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 10 client.16407 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 20 client.16407 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555619c4c540 0x555619c4e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 4 client.16407 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.240+0000 7f53cf26f740 1 -- 172.21.15.27:0/297213530 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555619b655a0 con 0x555619c4c540 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.241+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555619c4c540 0x555619c4e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.241+0000 7f53adffb640 10 client.16407 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:47.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.241+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555619c4c540 0x555619c4e950 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e7ddb1c57819eba server_cookie=deb0de76e0faa850 in_seq=0 out_seq=0 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53adffb640 1 -- 172.21.15.27:0/297213530 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555619b655a0 con 0x555619c4c540 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53adffb640 10 client.16407 handle_command_reply: tid=0 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 client.16407 shutdown 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 -- 172.21.15.27:0/297213530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 msgr2=0x555619c39860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c39860 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f53b4005e00 tx=0x7f53b409b0d0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 -- 172.21.15.27:0/297213530 shutdown_connections 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555619c4c540 0x555619c4e950 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c39da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c39860 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 --2- 172.21.15.27:0/297213530 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 0x555619c39320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53adffb640 0 client.16407 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 -- 172.21.15.27:0/297213530 >> 172.21.15.27:0/297213530 conn(0x555619bf5950 msgr2=0x555619c2f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 -- 172.21.15.27:0/297213530 shutdown_connections 2024-04-25T01:28:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.242+0000 7f53cf26f740 1 -- 172.21.15.27:0/297213530 wait complete. 2024-04-25T01:28:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.243+0000 7f53cf26f740 20 client.16407 trim_cache size 0 max 16384 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.260+0000 7f53cf26f740 1 Processor -- start 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.260+0000 7f53cf26f740 1 -- start start 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 0x555619c481b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c287d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c54220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53cf26f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53c41af350 con 0x555619c2e980 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53cf26f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53c41acc20 con 0x555619c3c420 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53cf26f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53c41aaa70 con 0x555619c2d290 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c287d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c287d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48070/0 (socket says 172.21.15.27:48070) 2024-04-25T01:28:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 -- 172.21.15.27:0/1160328262 learned_addr learned my addr 172.21.15.27:0/1160328262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/1160328262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 0x555619c481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 -- 172.21.15.27:0/1160328262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 msgr2=0x555619c481b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 --2- 172.21.15.27:0/1160328262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 0x555619c481b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 -- 172.21.15.27:0/1160328262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 msgr2=0x555619c54220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 --2- 172.21.15.27:0/1160328262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c54220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 -- 172.21.15.27:0/1160328262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555619865170 con 0x555619c2e980 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53c9afc640 1 --2- 172.21.15.27:0/1160328262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c287d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f53b40994f0 tx=0x7f53b40cf460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db42ea5a9f6893c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53affff640 1 -- 172.21.15.27:0/1160328262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b409b240 con 0x555619c2e980 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53affff640 1 -- 172.21.15.27:0/1160328262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b4096f90 con 0x555619c2e980 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.261+0000 7f53affff640 1 -- 172.21.15.27:0/1160328262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b40ccbc0 con 0x555619c2e980 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- 172.21.15.27:0/1160328262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 msgr2=0x555619c287d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 --2- 172.21.15.27:0/1160328262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c287d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f53b40994f0 tx=0x7f53b40cf460 comp rx=0 tx=0).stop 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- 172.21.15.27:0/1160328262 shutdown_connections 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 --2- 172.21.15.27:0/1160328262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c3c420 0x555619c54220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 --2- 172.21.15.27:0/1160328262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c2e980 0x555619c287d0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 --2- 172.21.15.27:0/1160328262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2d290 0x555619c481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- 172.21.15.27:0/1160328262 >> 172.21.15.27:0/1160328262 conn(0x555619bf5950 msgr2=0x555619c280c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- 172.21.15.27:0/1160328262 shutdown_connections 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- 172.21.15.27:0/1160328262 wait complete. 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- start start 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 Processor -- start 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c2d290 0x555619c446c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c44c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c3c420 0x555619c355b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53c41ab0e0 con 0x555619c3c420 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53c41ac550 con 0x555619c2d290 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.262+0000 7f53cf26f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53c41ab7b0 con 0x555619c2e980 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53cf26f740 10 client.0 _ll_register_callbacks cb 0x7f53b40cf3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c3c420 0x555619c355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c3c420 0x555619c355b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48084/0 (socket says 172.21.15.27:48084) 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 -- 172.21.15.27:0/3231764479 learned_addr learned my addr 172.21.15.27:0/3231764479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53c9afc640 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c44c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c2d290 0x555619c446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 -- 172.21.15.27:0/3231764479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 msgr2=0x555619c44c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c44c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 -- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c2d290 msgr2=0x555619c446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c2d290 0x555619c446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 -- 172.21.15.27:0/3231764479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555619c35af0 con 0x555619c3c420 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c2d290 0x555619c446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53caafe640 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c3c420 0x555619c355b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f53bc035000 tx=0x7f53bc009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3844099118a0d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53adffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53adffb640 1 -- 172.21.15.27:0/3231764479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc033030 con 0x555619c3c420 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53adffb640 1 -- 172.21.15.27:0/3231764479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f53bc041070 con 0x555619c3c420 2024-04-25T01:28:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53cf26f740 1 -- 172.21.15.27:0/3231764479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53b4096f90 con 0x555619c3c420 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53adffb640 1 -- 172.21.15.27:0/3231764479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f53bc02e760 con 0x555619c3c420 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53cf26f740 10 client.16413 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.263+0000 7f53cf26f740 1 -- 172.21.15.27:0/3231764479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555619cb3310 con 0x555619c3c420 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53adffb640 1 -- 172.21.15.27:0/3231764479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b4096f90 con 0x555619c3c420 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53adffb640 1 -- 172.21.15.27:0/3231764479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f53bc051070 con 0x555619c3c420 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53cf26f740 10 client.16413 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53cf26f740 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53cf26f740 20 client.16413 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53cf26f740 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555619cb3af0 0x555619cb5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53cf26f740 4 client.16413 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53cf26f740 1 -- 172.21.15.27:0/3231764479 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555619b9e800 con 0x555619cb3af0 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555619cb3af0 0x555619cb5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53adffb640 10 client.16413 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.264+0000 7f53ca2fd640 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555619cb3af0 0x555619cb5f10 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d1e290cb9ba98d0 server_cookie=4c501b8d0d4272a2 in_seq=0 out_seq=0 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.265+0000 7f53adffb640 1 -- 172.21.15.27:0/3231764479 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555619b9e800 con 0x555619cb3af0 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.265+0000 7f53adffb640 10 client.16413 handle_command_reply: tid=0 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.265+0000 7f53cf26f740 1 client.16413 shutdown 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 -- 172.21.15.27:0/3231764479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c3c420 msgr2=0x555619c355b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c3c420 0x555619c355b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f53bc035000 tx=0x7f53bc009360 comp rx=0 tx=0).stop 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 -- 172.21.15.27:0/3231764479 shutdown_connections 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555619cb3af0 0x555619cb5f10 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555619c3c420 0x555619c355b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555619c2e980 0x555619c44c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 --2- 172.21.15.27:0/3231764479 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555619c2d290 0x555619c446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53adffb640 0 client.16413 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 -- 172.21.15.27:0/3231764479 >> 172.21.15.27:0/3231764479 conn(0x555619bf5950 msgr2=0x555619c28950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 -- 172.21.15.27:0/3231764479 shutdown_connections 2024-04-25T01:28:47.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 1 -- 172.21.15.27:0/3231764479 wait complete. 2024-04-25T01:28:47.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cf26f740 20 client.16413 trim_cache size 0 max 16384 2024-04-25T01:28:47.251 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:28:47.251 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:28:47.251 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "fd01e8cc-8201-40e5-8e61-6dd2d1aea597", 2024-04-25T01:28:47.251 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:28:47.251 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:28:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 -- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2d80 msgr2=0x7f53b00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2d80 0x7f53b00a5240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009430 tx=0x7f53bc03a040 comp rx=0 tx=0).stop 2024-04-25T01:28:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 -- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c41032e0 msgr2=0x7f53c4198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c41032e0 0x7f53c4198560 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f53b400d950 tx=0x7f53b400de10 comp rx=0 tx=0).stop 2024-04-25T01:28:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 -- 172.21.15.27:0/1777111554 shutdown_connections 2024-04-25T01:28:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53c410bd00 0x7f53c419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53b00a2d80 0x7f53b00a5240 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c41032e0 0x7f53c4198560 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 --2- 172.21.15.27:0/1777111554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53c41029c0 0x7f53c4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.266+0000 7f53cc561640 1 -- 172.21.15.27:0/1777111554 >> 172.21.15.27:0/1777111554 conn(0x7f53c4077740 msgr2=0x7f53c40fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.267+0000 7f53cc561640 1 -- 172.21.15.27:0/1777111554 shutdown_connections 2024-04-25T01:28:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.267+0000 7f53cc561640 1 -- 172.21.15.27:0/1777111554 wait complete. 2024-04-25T01:28:47.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:28:47.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:28:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3538535018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100fd0 msgr2=0x7fdcf810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3538535018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100fd0 0x7fdcf810d4e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009cd0 tx=0x7fdcf002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3538535018 shutdown_connections 2024-04-25T01:28:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3538535018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100fd0 0x7fdcf810d4e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3538535018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcf8100610 0x7fdcf8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3538535018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf81066b0 0x7fdcf8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3538535018 >> 172.21.15.27:0/3538535018 conn(0x7fdcf8077740 msgr2=0x7fdcf8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3538535018 shutdown_connections 2024-04-25T01:28:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.382+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3538535018 wait complete. 2024-04-25T01:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcfc9c8640 1 Processor -- start 2024-04-25T01:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcfc9c8640 1 -- start start 2024-04-25T01:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcfc9c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100610 0x7fdcf8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcfc9c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf8100fd0 0x7fdcf81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcfc9c8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcf81066b0 0x7fdcf819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcfc9c8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf81109f0 con 0x7fdcf8100610 2024-04-25T01:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcfc9c8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf8110870 con 0x7fdcf81066b0 2024-04-25T01:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcfc9c8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf8110b70 con 0x7fdcf8100fd0 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100610 0x7fdcf8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100610 0x7fdcf8197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48092/0 (socket says 172.21.15.27:48092) 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf659c640 1 -- 172.21.15.27:0/3904650291 learned_addr learned my addr 172.21.15.27:0/3904650291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf8100fd0 0x7fdcf81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf659c640 1 -- 172.21.15.27:0/3904650291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf8100fd0 msgr2=0x7fdcf81984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf8100fd0 0x7fdcf81984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf659c640 1 -- 172.21.15.27:0/3904650291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcf81066b0 msgr2=0x7fdcf819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcf81066b0 0x7fdcf819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf659c640 1 -- 172.21.15.27:0/3904650291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf819cf50 con 0x7fdcf8100610 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.383+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf8100fd0 0x7fdcf81984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.384+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100610 0x7fdcf8197f60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fdce800e9c0 tx=0x7fdce800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.384+0000 7fdce77fe640 1 -- 172.21.15.27:0/3904650291 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce800cd80 con 0x7fdcf8100610 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.384+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3904650291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf819d240 con 0x7fdcf8100610 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.384+0000 7fdce77fe640 1 -- 172.21.15.27:0/3904650291 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdce80035f0 con 0x7fdcf8100610 2024-04-25T01:28:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.384+0000 7fdce77fe640 1 -- 172.21.15.27:0/3904650291 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8010690 con 0x7fdcf8100610 2024-04-25T01:28:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.384+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3904650291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf81a4ad0 con 0x7fdcf8100610 2024-04-25T01:28:47.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.386+0000 7fdcff7b3740 1 Processor -- start 2024-04-25T01:28:47.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.386+0000 7fdce77fe640 1 -- 172.21.15.27:0/3904650291 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdce80037a0 con 0x7fdcf8100610 2024-04-25T01:28:47.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.386+0000 7fdcff7b3740 1 -- start start 2024-04-25T01:28:47.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6987a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef69920e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef6994a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cef68ec6b0 con 0x55cef6987630 2024-04-25T01:28:47.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cef6894880 con 0x55cef6992620 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cef68e8670 con 0x55cef6987f50 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6987a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6987a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48102/0 (socket says 172.21.15.27:48102) 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 -- 172.21.15.27:0/4031771543 learned_addr learned my addr 172.21.15.27:0/4031771543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdce77fe640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdce00a2d00 0x7fdce00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf659c640 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef69920e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef6994a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 -- 172.21.15.27:0/4031771543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 msgr2=0x55cef69920e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef69920e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 -- 172.21.15.27:0/4031771543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 msgr2=0x55cef6994a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef6994a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 -- 172.21.15.27:0/4031771543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cef66ca7d0 con 0x55cef6987630 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf659c640 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef69920e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef6994a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdce77fe640 1 -- 172.21.15.27:0/3904650291 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdce80cfc90 con 0x7fdcf8100610 2024-04-25T01:28:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdce00a2d00 0x7fdce00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.387+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6987a10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009df0 tx=0x7fdcf0005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582e8f3350be2070 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdce4ff9640 1 -- 172.21.15.27:0/4031771543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf004d070 con 0x55cef6987630 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdce4ff9640 1 -- 172.21.15.27:0/4031771543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcf0043070 con 0x55cef6987630 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdce4ff9640 1 -- 172.21.15.27:0/4031771543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0030b00 con 0x55cef6987630 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdce00a2d00 0x7fdce00a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcf004a000 tx=0x7fdcf004d000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 -- 172.21.15.27:0/4031771543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 msgr2=0x55cef6987a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6987a10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009df0 tx=0x7fdcf0005ef0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 -- 172.21.15.27:0/4031771543 shutdown_connections 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef6994a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef69920e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/4031771543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6987a10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 -- 172.21.15.27:0/4031771543 >> 172.21.15.27:0/4031771543 conn(0x55cef6983950 msgr2=0x55cef6983d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 -- 172.21.15.27:0/4031771543 shutdown_connections 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.388+0000 7fdcff7b3740 1 -- 172.21.15.27:0/4031771543 wait complete. 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 1 -- start start 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 1 Processor -- start 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef698f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef698faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef698ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cef6995410 con 0x55cef6987630 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55cef68a7f50 con 0x55cef6992620 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55cef6995590 con 0x55cef6987f50 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcff7b3740 10 client.0 _ll_register_callbacks cb 0x55cef68e3000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcf659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef698f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcf659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef698f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48116/0 (socket says 172.21.15.27:48116) 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcf659c640 1 -- 172.21.15.27:0/3169780373 learned_addr learned my addr 172.21.15.27:0/3169780373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.389+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef698faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef698ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcf659c640 1 -- 172.21.15.27:0/3169780373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 msgr2=0x55cef698faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef698faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcf659c640 1 -- 172.21.15.27:0/3169780373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 msgr2=0x55cef698ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef698ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcf659c640 1 -- 172.21.15.27:0/3169780373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cef66003f0 con 0x55cef6987630 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef698faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef698f560 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdce809c480 tx=0x7fdce809b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc085067dbc2aeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcddffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcddffb640 1 -- 172.21.15.27:0/3169780373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce809c520 con 0x55cef6987630 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcddffb640 1 -- 172.21.15.27:0/3169780373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdce80ba690 con 0x55cef6987630 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3169780373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf0043070 con 0x55cef6987630 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcddffb640 1 -- 172.21.15.27:0/3169780373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdce80a4a90 con 0x55cef6987630 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcff7b3740 10 client.16425 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3169780373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cef65c00f0 con 0x55cef6987630 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcddffb640 1 -- 172.21.15.27:0/3169780373 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcf0043070 con 0x55cef6987630 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcddffb640 1 -- 172.21.15.27:0/3169780373 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fdce80a4600 con 0x55cef6987630 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcff7b3740 10 client.16425 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcff7b3740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.390+0000 7fdcff7b3740 20 client.16425 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.391+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55cef69a2970 0x55cef69a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.391+0000 7fdcff7b3740 4 client.16425 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.391+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3169780373 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cef68f9bf0 con 0x55cef69a2970 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.391+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55cef69a2970 0x55cef69a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.391+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55cef69a2970 0x55cef69a4e00 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7a1eccf51002091 server_cookie=f4e067659dcd31b in_seq=0 out_seq=0 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.391+0000 7fdcddffb640 10 client.16425 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcddffb640 1 -- 172.21.15.27:0/3169780373 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cef68f9bf0 con 0x55cef69a2970 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcddffb640 10 client.16425 handle_command_reply: tid=0 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 client.16425 shutdown 2024-04-25T01:28:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3169780373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 msgr2=0x55cef698f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef698f560 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdce809c480 tx=0x7fdce809b780 comp rx=0 tx=0).stop 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3169780373 shutdown_connections 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55cef69a2970 0x55cef69a4e00 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef698ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987f50 0x55cef698faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3169780373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef698f560 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3169780373 >> 172.21.15.27:0/3169780373 conn(0x55cef6983950 msgr2=0x55cef6989750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcddffb640 0 client.16425 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3169780373 shutdown_connections 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.392+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3169780373 wait complete. 2024-04-25T01:28:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.393+0000 7fdcff7b3740 20 client.16425 trim_cache size 0 max 16384 2024-04-25T01:28:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcff7b3740 1 Processor -- start 2024-04-25T01:28:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcff7b3740 1 -- start start 2024-04-25T01:28:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6991c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6987f50 0x55cef699e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6992620 0x55cef698e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf81b14e0 con 0x55cef6987630 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf81aedb0 con 0x55cef6987f50 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf81acc00 con 0x55cef6992620 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6991c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6991c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48132/0 (socket says 172.21.15.27:48132) 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf659c640 1 -- 172.21.15.27:0/2111652945 learned_addr learned my addr 172.21.15.27:0/2111652945 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf659c640 1 -- 172.21.15.27:0/2111652945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6992620 msgr2=0x55cef698e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6987f50 0x55cef699e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6992620 0x55cef698e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf659c640 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6992620 0x55cef698e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf659c640 1 -- 172.21.15.27:0/2111652945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6987f50 msgr2=0x55cef699e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf659c640 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6987f50 0x55cef699e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf659c640 1 -- 172.21.15.27:0/2111652945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cef6975090 con 0x55cef6987630 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6992620 0x55cef698e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.410+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6987f50 0x55cef699e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcf659c640 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6991c90 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdce80cd100 tx=0x7fdce80a00b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d05ddc6a430b903 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcdffff640 1 -- 172.21.15.27:0/2111652945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce809bbf0 con 0x55cef6987630 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcdffff640 1 -- 172.21.15.27:0/2111652945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdce809c520 con 0x55cef6987630 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcdffff640 1 -- 172.21.15.27:0/2111652945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce809b4d0 con 0x55cef6987630 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 -- 172.21.15.27:0/2111652945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 msgr2=0x55cef6991c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6991c90 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdce80cd100 tx=0x7fdce80a00b0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 -- 172.21.15.27:0/2111652945 shutdown_connections 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6992620 0x55cef698e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6987f50 0x55cef699e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/2111652945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987630 0x55cef6991c90 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 -- 172.21.15.27:0/2111652945 >> 172.21.15.27:0/2111652945 conn(0x55cef6983950 msgr2=0x55cef6982260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 -- 172.21.15.27:0/2111652945 shutdown_connections 2024-04-25T01:28:47.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.411+0000 7fdcff7b3740 1 -- 172.21.15.27:0/2111652945 wait complete. 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 1 -- start start 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 1 Processor -- start 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987630 0x55cef699a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987f50 0x55cef698bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef698c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf81ad270 con 0x55cef6987f50 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf81ae6e0 con 0x55cef6992620 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf81ad940 con 0x55cef6987630 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcff7b3740 10 client.0 _ll_register_callbacks cb 0x7fdce8098120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf5d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987f50 0x55cef698bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf5d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987f50 0x55cef698bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48136/0 (socket says 172.21.15.27:48136) 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf5d9b640 1 -- 172.21.15.27:0/3348264672 learned_addr learned my addr 172.21.15.27:0/3348264672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf6d9d640 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef698c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987630 0x55cef699a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf5d9b640 1 -- 172.21.15.27:0/3348264672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987630 msgr2=0x55cef699a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987630 0x55cef699a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf5d9b640 1 -- 172.21.15.27:0/3348264672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 msgr2=0x55cef698c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef698c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.412+0000 7fdcf5d9b640 1 -- 172.21.15.27:0/3348264672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cef698c640 con 0x55cef6987f50 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcf5d9b640 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987f50 0x55cef698bbc0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x55cef6984ae0 tx=0x7fdcec012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e0ce4ef090772b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcddffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcddffb640 1 -- 172.21.15.27:0/3348264672 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcec01e070 con 0x55cef6987f50 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3348264672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdce809c520 con 0x55cef6987f50 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcddffb640 1 -- 172.21.15.27:0/3348264672 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdcec005050 con 0x55cef6987f50 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcddffb640 1 -- 172.21.15.27:0/3348264672 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdcec02a050 con 0x55cef6987f50 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 10 client.16431 fetch_fsmap learned FSMap version 26 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3348264672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cef6a08c00 con 0x55cef6987f50 2024-04-25T01:28:47.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcddffb640 1 -- 172.21.15.27:0/3348264672 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdce809c520 con 0x55cef6987f50 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcddffb640 1 -- 172.21.15.27:0/3348264672 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fdcec00f040 con 0x55cef6987f50 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 10 client.16431 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 20 client.16431 populate_metadata read hostname 'smithi027' 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55cef6a09360 0x55cef6a0b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 4 client.16431 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.413+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3348264672 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cef68f3f40 con 0x55cef6a09360 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55cef6a09360 0x55cef6a0b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcf659c640 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55cef6a09360 0x55cef6a0b780 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c38bd6e19564dac1 server_cookie=f01ce9e93411a0c8 in_seq=0 out_seq=0 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcddffb640 10 client.16431 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcddffb640 1 -- 172.21.15.27:0/3348264672 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cef68f3f40 con 0x55cef6a09360 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcddffb640 10 client.16431 handle_command_reply: tid=0 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 client.16431 shutdown 2024-04-25T01:28:47.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3348264672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987f50 msgr2=0x55cef698bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987f50 0x55cef698bbc0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x55cef6984ae0 tx=0x7fdcec012820 comp rx=0 tx=0).stop 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3348264672 shutdown_connections 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55cef6a09360 0x55cef6a0b780 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cef6992620 0x55cef698c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cef6987f50 0x55cef698bbc0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 --2- 172.21.15.27:0/3348264672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55cef6987630 0x55cef699a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3348264672 >> 172.21.15.27:0/3348264672 conn(0x55cef6983950 msgr2=0x55cef698fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcddffb640 0 client.16431 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3348264672 shutdown_connections 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 1 -- 172.21.15.27:0/3348264672 wait complete. 2024-04-25T01:28:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.414+0000 7fdcff7b3740 20 client.16431 trim_cache size 0 max 16384 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout: "fd01e8cc-8201-40e5-8e61-6dd2d1aea597": { 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "fd01e8cc-8201-40e5-8e61-6dd2d1aea597", 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:28:47.401 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:28:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdce00a2d00 msgr2=0x7fdce00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdce00a2d00 0x7fdce00a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcf004a000 tx=0x7fdcf004d000 comp rx=0 tx=0).stop 2024-04-25T01:28:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100610 msgr2=0x7fdcf8197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100610 0x7fdcf8197f60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fdce800e9c0 tx=0x7fdce800ee80 comp rx=0 tx=0).stop 2024-04-25T01:28:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3904650291 shutdown_connections 2024-04-25T01:28:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcf81066b0 0x7fdcf819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf8100fd0 0x7fdcf81984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdce00a2d00 0x7fdce00a51c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 --2- 172.21.15.27:0/3904650291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8100610 0x7fdcf8197f60 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3904650291 >> 172.21.15.27:0/3904650291 conn(0x7fdcf8077740 msgr2=0x7fdcf80ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3904650291 shutdown_connections 2024-04-25T01:28:47.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:28:47.415+0000 7fdcfc9c8640 1 -- 172.21.15.27:0/3904650291 wait complete. 2024-04-25T01:28:47.405 INFO:tasks.cephfs.filesystem:scrub status for tag:fd01e8cc-8201-40e5-8e61-6dd2d1aea597 - {'path': '/', 'tag': 'fd01e8cc-8201-40e5-8e61-6dd2d1aea597', 'options': 'recursive,force'} 2024-04-25T01:28:48.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:48 smithi027 ceph-mon[29197]: pgmap v104: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:48.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:48 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:28:48.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:48 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:48.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:48 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:28:48.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:48 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:28:48.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:48 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:48.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:48 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:48.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:48 smithi153 ceph-mon[32621]: pgmap v104: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:48.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:48 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:28:48.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:48 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:48.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:48 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:28:48.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:48 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:28:48.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:48 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:48.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:48 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:48.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:48 smithi139 ceph-mon[32760]: pgmap v104: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:48.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:48 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:28:48.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:48 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:48.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:48 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:28:48.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:48 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:28:48.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:48 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:48.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:48 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:28:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:50 smithi027 ceph-mon[29197]: pgmap v105: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:50.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:50 smithi153 ceph-mon[32621]: pgmap v105: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:50.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:50 smithi139 ceph-mon[32760]: pgmap v105: 129 pgs: 129 active+clean; 2.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:28:52.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:52 smithi027 ceph-mon[29197]: pgmap v106: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:52.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:52 smithi153 ceph-mon[32621]: pgmap v106: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:52.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:52 smithi139 ceph-mon[32760]: pgmap v106: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:64/600: open fd = 3 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:65/600: open fd = 3 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:66/600: open fd = 3 2024-04-25T01:28:52.651 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:67/600: open fd = 3 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:68/600: open fd = 3 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.652 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:69/600: open fd = 3 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:70/600: open fd = 3 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:71/600: open fd = 3 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.653 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:72/600: open fd = 3 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:73/600: open fd = 3 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:74/600: open fd = 3 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.654 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:75/600: open fd = 3 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:76/600: open fd = 3 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:77/600: open fd = 3 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.655 INFO:tasks.workunit.client.0.smithi027.stdout:78/600: open fd = 3 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:79/600: open fd = 3 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:80/600: open fd = 3 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.656 INFO:tasks.workunit.client.0.smithi027.stdout:81/600: open fd = 3 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:82/600: open fd = 3 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:83/600: open fd = 3 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.657 INFO:tasks.workunit.client.0.smithi027.stdout:84/600: open fd = 3 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:85/600: open fd = 3 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:86/600: open fd = 3 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.658 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:87/600: open fd = 3 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:88/600: open fd = 3 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:89/600: open fd = 3 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.659 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:90/600: open fd = 3 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:91/600: open fd = 3 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:92/600: open fd = 3 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.660 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:93/600: open fd = 3 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:94/600: open fd = 3 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:95/600: open fd = 3 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.661 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:96/600: open fd = 3 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:97/600: open fd = 3 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:98/600: open fd = 3 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.662 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:99/600: open fd = 3 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:100/600: open fd = 3 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:101/600: open fd = 3 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.663 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:102/600: open fd = 3 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:103/600: open fd = 3 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:104/600: open fd = 3 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.664 INFO:tasks.workunit.client.0.smithi027.stdout:105/600: open fd = 3 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:106/600: open fd = 3 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:107/600: open fd = 3 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:108/600: open fd = 3 2024-04-25T01:28:52.665 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:109/600: open fd = 3 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:110/600: open fd = 3 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:111/600: open fd = 3 2024-04-25T01:28:52.666 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:112/600: open fd = 3 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:113/600: open fd = 3 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:114/600: open fd = 3 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.667 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:115/600: open fd = 3 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:116/600: open fd = 3 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:117/600: open fd = 3 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.668 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:118/600: open fd = 3 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:119/600: open fd = 3 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:120/600: open fd = 3 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.669 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:121/600: open fd = 3 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:122/600: open fd = 3 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:123/600: open fd = 3 2024-04-25T01:28:52.670 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:124/600: open fd = 3 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:125/600: open fd = 3 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:28:52.671 INFO:tasks.workunit.client.0.smithi027.stdout:126/600: open fd = 3 2024-04-25T01:28:52.672 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:28:53.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:53 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:28:53.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:53 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:28:53.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:53 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:28:53.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:53 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:28:53.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:53 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:28:53.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:53 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:28:54.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:54 smithi027 ceph-mon[29197]: pgmap v107: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:28:54.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:54 smithi027 ceph-mon[29197]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:28:54.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:54 smithi153 ceph-mon[32621]: pgmap v107: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:28:54.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:54 smithi153 ceph-mon[32621]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:28:54.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:54 smithi139 ceph-mon[32760]: pgmap v107: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:28:54.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:54 smithi139 ceph-mon[32760]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:28:56.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:56 smithi027 ceph-mon[29197]: pgmap v108: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:28:56.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:56 smithi153 ceph-mon[32621]: pgmap v108: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:28:56.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:56 smithi139 ceph-mon[32760]: pgmap v108: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:28:58.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:28:58 smithi027 ceph-mon[29197]: pgmap v109: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:28:58.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:28:58 smithi153 ceph-mon[32621]: pgmap v109: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:28:58.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:28:58 smithi139 ceph-mon[32760]: pgmap v109: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:29:00.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:00 smithi153 ceph-mon[32621]: pgmap v110: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:29:00.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:00 smithi139 ceph-mon[32760]: pgmap v110: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:29:00.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:00 smithi027 ceph-mon[29197]: pgmap v110: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:29:02.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:02 smithi153 ceph-mon[32621]: pgmap v111: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:29:02.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:02 smithi139 ceph-mon[32760]: pgmap v111: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:29:02.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:02 smithi027 ceph-mon[29197]: pgmap v111: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 29 op/s 2024-04-25T01:29:04.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:04 smithi153 ceph-mon[32621]: pgmap v112: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:29:04.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:04 smithi139 ceph-mon[32760]: pgmap v112: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:29:04.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:04 smithi027 ceph-mon[29197]: pgmap v112: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:29:06.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:06 smithi153 ceph-mon[32621]: pgmap v113: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:29:06.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:06 smithi139 ceph-mon[32760]: pgmap v113: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:29:06.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:06 smithi027 ceph-mon[29197]: pgmap v113: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 24 op/s 2024-04-25T01:29:08.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:08 smithi153 ceph-mon[32621]: pgmap v114: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:29:08.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:08 smithi139 ceph-mon[32760]: pgmap v114: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:29:08.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:08 smithi027 ceph-mon[29197]: pgmap v114: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:29:10.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:10 smithi027 ceph-mon[29197]: pgmap v115: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-25T01:29:10.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:10 smithi153 ceph-mon[32621]: pgmap v115: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-25T01:29:10.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:10 smithi139 ceph-mon[32760]: pgmap v115: 129 pgs: 129 active+clean; 2.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-25T01:29:12.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:12 smithi153 ceph-mon[32621]: pgmap v116: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-25T01:29:12.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:12 smithi139 ceph-mon[32760]: pgmap v116: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-25T01:29:12.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:12 smithi027 ceph-mon[29197]: pgmap v116: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 29 op/s 2024-04-25T01:29:14.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:14 smithi153 ceph-mon[32621]: pgmap v117: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 9 op/s 2024-04-25T01:29:14.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:14 smithi139 ceph-mon[32760]: pgmap v117: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 9 op/s 2024-04-25T01:29:14.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:14 smithi027 ceph-mon[29197]: pgmap v117: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 9 op/s 2024-04-25T01:29:16.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:16 smithi153 ceph-mon[32621]: pgmap v118: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 9 op/s 2024-04-25T01:29:16.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:16 smithi139 ceph-mon[32760]: pgmap v118: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 9 op/s 2024-04-25T01:29:16.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:16 smithi027 ceph-mon[29197]: pgmap v118: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 9 op/s 2024-04-25T01:29:17.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:29:17.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:29:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 -- 172.21.15.27:0/1910550988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 msgr2=0x7f3994106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 --2- 172.21.15.27:0/1910550988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 0x7f3994106a70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f398c009900 tx=0x7f398c02a410 comp rx=0 tx=0).stop 2024-04-25T01:29:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 -- 172.21.15.27:0/1910550988 shutdown_connections 2024-04-25T01:29:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 --2- 172.21.15.27:0/1910550988 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3994101330 0x7f399410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 --2- 172.21.15.27:0/1910550988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994100990 0x7f3994100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 --2- 172.21.15.27:0/1910550988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 0x7f3994106a70 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 -- 172.21.15.27:0/1910550988 >> 172.21.15.27:0/1910550988 conn(0x7f3994076440 msgr2=0x7f3994076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 -- 172.21.15.27:0/1910550988 shutdown_connections 2024-04-25T01:29:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.544+0000 7f3999921640 1 -- 172.21.15.27:0/1910550988 wait complete. 2024-04-25T01:29:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3999921640 1 Processor -- start 2024-04-25T01:29:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3999921640 1 -- start start 2024-04-25T01:29:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3999921640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994100990 0x7f3994197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3999921640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3994101330 0x7f3994198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3999921640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 0x7f399419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3999921640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f39941109a0 con 0x7f3994106690 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3999921640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3994110820 con 0x7f3994101330 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3999921640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3994110b20 con 0x7f3994100990 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f39937fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 0x7f399419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f39937fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 0x7f399419c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39484/0 (socket says 172.21.15.27:39484) 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f39937fe640 1 -- 172.21.15.27:0/647951318 learned_addr learned my addr 172.21.15.27:0/647951318 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f3992ffd640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994100990 0x7f3994197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.545+0000 7f39927fc640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3994101330 0x7f3994198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f39937fe640 1 -- 172.21.15.27:0/647951318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994100990 msgr2=0x7f3994197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f39937fe640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994100990 0x7f3994197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f39937fe640 1 -- 172.21.15.27:0/647951318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3994101330 msgr2=0x7f3994198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f39937fe640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3994101330 0x7f3994198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f39937fe640 1 -- 172.21.15.27:0/647951318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399419cec0 con 0x7f3994106690 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f39937fe640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 0x7f399419c790 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f398400eaa0 tx=0x7f398400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f399891f640 1 -- 172.21.15.27:0/647951318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984004d20 con 0x7f3994106690 2024-04-25T01:29:17.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f399891f640 1 -- 172.21.15.27:0/647951318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3984005020 con 0x7f3994106690 2024-04-25T01:29:17.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f3999921640 1 -- 172.21.15.27:0/647951318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399419d1b0 con 0x7f3994106690 2024-04-25T01:29:17.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f399891f640 1 -- 172.21.15.27:0/647951318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39840116d0 con 0x7f3994106690 2024-04-25T01:29:17.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.546+0000 7f3999921640 1 -- 172.21.15.27:0/647951318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39941a4a40 con 0x7f3994106690 2024-04-25T01:29:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.548+0000 7f399891f640 1 -- 172.21.15.27:0/647951318 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f39840051d0 con 0x7f3994106690 2024-04-25T01:29:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399c62a740 1 Processor -- start 2024-04-25T01:29:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399c62a740 1 -- start start 2024-04-25T01:29:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399c62a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c6447aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399c62a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c644850e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399c62a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c64485620 0x558c64487a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399c62a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558c643df6b0 con 0x558c6447a630 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399c62a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558c64387880 con 0x558c64485620 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399c62a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558c643db670 con 0x558c6447af50 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c6447aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c6447aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39492/0 (socket says 172.21.15.27:39492) 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39927fc640 1 -- 172.21.15.27:0/4113704032 learned_addr learned my addr 172.21.15.27:0/4113704032 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399891f640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f397c0a2d00 0x7f397c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f3992ffd640 1 --2- 172.21.15.27:0/4113704032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c644850e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39937fe640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f397c0a2d00 0x7f397c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39927fc640 1 -- 172.21.15.27:0/4113704032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 msgr2=0x558c644850e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39927fc640 1 --2- 172.21.15.27:0/4113704032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c644850e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39927fc640 1 -- 172.21.15.27:0/4113704032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c64485620 msgr2=0x558c64487a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39927fc640 1 --2- 172.21.15.27:0/4113704032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c64485620 0x558c64487a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f39927fc640 1 -- 172.21.15.27:0/4113704032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c641bd7d0 con 0x558c6447a630 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.549+0000 7f399891f640 1 -- 172.21.15.27:0/647951318 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f39840cf9a0 con 0x7f3994106690 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f39927fc640 1 --2- 172.21.15.27:0/4113704032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c6447aa10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f398800ad40 tx=0x7f39880117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e025942e85b458ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f3979ffb640 1 -- 172.21.15.27:0/4113704032 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988012450 con 0x558c6447a630 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f39937fe640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f397c0a2d00 0x7f397c0a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f398400efd0 tx=0x7f3984004f50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f3979ffb640 1 -- 172.21.15.27:0/4113704032 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3988012ad0 con 0x558c6447a630 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f3979ffb640 1 -- 172.21.15.27:0/4113704032 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398800faa0 con 0x558c6447a630 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f399c62a740 1 -- 172.21.15.27:0/4113704032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 msgr2=0x558c6447aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f399c62a740 1 --2- 172.21.15.27:0/4113704032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c6447aa10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f398800ad40 tx=0x7f39880117b0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f399c62a740 1 -- 172.21.15.27:0/4113704032 shutdown_connections 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f399c62a740 1 --2- 172.21.15.27:0/4113704032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c64485620 0x558c64487a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.550+0000 7f399c62a740 1 --2- 172.21.15.27:0/4113704032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c644850e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 --2- 172.21.15.27:0/4113704032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c6447aa10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 -- 172.21.15.27:0/4113704032 >> 172.21.15.27:0/4113704032 conn(0x558c64476950 msgr2=0x558c64476d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 -- 172.21.15.27:0/4113704032 shutdown_connections 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 -- 172.21.15.27:0/4113704032 wait complete. 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 -- start start 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 Processor -- start 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c644825e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 0x558c64482b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 0x558c64483060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558c64488410 con 0x558c6447a630 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558c6439af50 con 0x558c6447af50 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.551+0000 7f399c62a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558c64488590 con 0x558c64485620 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f399c62a740 10 client.0 _ll_register_callbacks cb 0x558c643d6000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c644825e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c644825e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39504/0 (socket says 172.21.15.27:39504) 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 -- 172.21.15.27:0/387164343 learned_addr learned my addr 172.21.15.27:0/387164343 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:17.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 0x558c64483060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f3992ffd640 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 0x558c64482b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 -- 172.21.15.27:0/387164343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 msgr2=0x558c64483060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 0x558c64483060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 -- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 msgr2=0x558c64482b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 0x558c64482b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 -- 172.21.15.27:0/387164343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c640f33f0 con 0x558c6447a630 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f3992ffd640 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 0x558c64482b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f39927fc640 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c644825e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f398800a350 tx=0x7f3988011300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dc49e95f819d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f3972ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f3972ffd640 1 -- 172.21.15.27:0/387164343 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398800fbf0 con 0x558c6447a630 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f3972ffd640 1 -- 172.21.15.27:0/387164343 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f398800fd90 con 0x558c6447a630 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f399c62a740 1 -- 172.21.15.27:0/387164343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3988012ad0 con 0x558c6447a630 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f3972ffd640 1 -- 172.21.15.27:0/387164343 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f398800a9a0 con 0x558c6447a630 2024-04-25T01:29:17.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f399c62a740 10 client.16443 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.552+0000 7f399c62a740 1 -- 172.21.15.27:0/387164343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558c640b30f0 con 0x558c6447a630 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f3972ffd640 1 -- 172.21.15.27:0/387164343 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3988012ad0 con 0x558c6447a630 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f3972ffd640 1 -- 172.21.15.27:0/387164343 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3988027020 con 0x558c6447a630 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f399c62a740 10 client.16443 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f399c62a740 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f399c62a740 20 client.16443 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f399c62a740 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558c64495810 0x558c64497c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f399c62a740 4 client.16443 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f399c62a740 1 -- 172.21.15.27:0/387164343 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c643ecbf0 con 0x558c64495810 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.553+0000 7f3992ffd640 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558c64495810 0x558c64497c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.554+0000 7f3992ffd640 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558c64495810 0x558c64497c00 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba5e5427a45797ec server_cookie=d49a462c668478ee in_seq=0 out_seq=0 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.554+0000 7f3972ffd640 10 client.16443 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:17.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.554+0000 7f3972ffd640 1 -- 172.21.15.27:0/387164343 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c643ecbf0 con 0x558c64495810 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.554+0000 7f3972ffd640 10 client.16443 handle_command_reply: tid=0 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 client.16443 shutdown 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 -- 172.21.15.27:0/387164343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 msgr2=0x558c644825e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c644825e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f398800a350 tx=0x7f3988011300 comp rx=0 tx=0).stop 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 -- 172.21.15.27:0/387164343 shutdown_connections 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558c64495810 0x558c64497c00 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 0x558c64483060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 0x558c64482b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f3972ffd640 0 client.16443 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 --2- 172.21.15.27:0/387164343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c644825e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 -- 172.21.15.27:0/387164343 >> 172.21.15.27:0/387164343 conn(0x558c64476950 msgr2=0x558c6447c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 -- 172.21.15.27:0/387164343 shutdown_connections 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 1 -- 172.21.15.27:0/387164343 wait complete. 2024-04-25T01:29:17.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.555+0000 7f399c62a740 20 client.16443 trim_cache size 0 max 16384 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f399c62a740 1 Processor -- start 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f399c62a740 1 -- start start 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f399c62a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c64484d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f399c62a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 0x558c644912d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f399c62a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 0x558c64475970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f399c62a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f39941b1450 con 0x558c6447a630 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f399c62a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f39941aed20 con 0x558c6447af50 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f399c62a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39941acb70 con 0x558c64485620 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c64484d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c64484d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39518/0 (socket says 172.21.15.27:39518) 2024-04-25T01:29:17.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 -- 172.21.15.27:0/2913956534 learned_addr learned my addr 172.21.15.27:0/2913956534 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 -- 172.21.15.27:0/2913956534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 msgr2=0x558c64475970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 --2- 172.21.15.27:0/2913956534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 0x558c64475970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 -- 172.21.15.27:0/2913956534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 msgr2=0x558c644912d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 --2- 172.21.15.27:0/2913956534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 0x558c644912d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 -- 172.21.15.27:0/2913956534 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c64488530 con 0x558c6447a630 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.573+0000 7f39927fc640 1 --2- 172.21.15.27:0/2913956534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c64484d10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f398800e3f0 tx=0x7f398800a380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d353b6a0f04f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f39797fa640 1 -- 172.21.15.27:0/2913956534 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398803a070 con 0x558c6447a630 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f39797fa640 1 -- 172.21.15.27:0/2913956534 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f398801e030 con 0x558c6447a630 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f39797fa640 1 -- 172.21.15.27:0/2913956534 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398800fc50 con 0x558c6447a630 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 -- 172.21.15.27:0/2913956534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 msgr2=0x558c64484d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 --2- 172.21.15.27:0/2913956534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c64484d10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f398800e3f0 tx=0x7f398800a380 comp rx=0 tx=0).stop 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 -- 172.21.15.27:0/2913956534 shutdown_connections 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 --2- 172.21.15.27:0/2913956534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c64485620 0x558c64475970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 --2- 172.21.15.27:0/2913956534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447af50 0x558c644912d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 --2- 172.21.15.27:0/2913956534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c6447a630 0x558c64484d10 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 -- 172.21.15.27:0/2913956534 >> 172.21.15.27:0/2913956534 conn(0x558c64476950 msgr2=0x558c64475260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 -- 172.21.15.27:0/2913956534 shutdown_connections 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 -- 172.21.15.27:0/2913956534 wait complete. 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.574+0000 7f399c62a740 1 -- start start 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f399c62a740 1 Processor -- start 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f399c62a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447a630 0x558c6447f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f399c62a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c6447f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f399c62a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c64485620 0x558c6447fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f399c62a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f39941ad1e0 con 0x558c64485620 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f399c62a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f39941ae650 con 0x558c6447a630 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f399c62a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39941ad8b0 con 0x558c6447af50 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f399c62a740 10 client.0 _ll_register_callbacks cb 0x7f398c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c64485620 0x558c6447fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c64485620 0x558c6447fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39522/0 (socket says 172.21.15.27:39522) 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 -- 172.21.15.27:0/3057444958 learned_addr learned my addr 172.21.15.27:0/3057444958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447a630 0x558c6447f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f3992ffd640 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c6447f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 -- 172.21.15.27:0/3057444958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 msgr2=0x558c6447f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c6447f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 -- 172.21.15.27:0/3057444958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447a630 msgr2=0x558c6447f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447a630 0x558c6447f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f39927fc640 1 -- 172.21.15.27:0/3057444958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c644faa40 con 0x558c64485620 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.575+0000 7f3992ffd640 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c6447f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f39927fc640 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c64485620 0x558c6447fcb0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f398800a350 tx=0x7f3988005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d209c759b7da15f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f3972ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f3972ffd640 1 -- 172.21.15.27:0/3057444958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398803a070 con 0x558c64485620 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f3972ffd640 1 -- 172.21.15.27:0/3057444958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f398800fc00 con 0x558c64485620 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 1 -- 172.21.15.27:0/3057444958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f398801e030 con 0x558c64485620 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f3972ffd640 1 -- 172.21.15.27:0/3057444958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3988011cb0 con 0x558c64485620 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 10 client.16449 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 1 -- 172.21.15.27:0/3057444958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558c644fbc90 con 0x558c64485620 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f3972ffd640 1 -- 172.21.15.27:0/3057444958 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f398801e030 con 0x558c64485620 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f3972ffd640 1 -- 172.21.15.27:0/3057444958 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3988027050 con 0x558c64485620 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 10 client.16449 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 10 client.16449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:17.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 20 client.16449 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558c644fc320 0x558c644fe740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 4 client.16449 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.576+0000 7f399c62a740 1 -- 172.21.15.27:0/3057444958 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c643e6f40 con 0x558c644fc320 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f3992ffd640 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558c644fc320 0x558c644fe740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f3992ffd640 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558c644fc320 0x558c644fe740 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12c916f454e01519 server_cookie=454310c756f9fc2a in_seq=0 out_seq=0 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f3972ffd640 10 client.16449 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f3972ffd640 1 -- 172.21.15.27:0/3057444958 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558c643e6f40 con 0x558c644fc320 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f3972ffd640 10 client.16449 handle_command_reply: tid=0 2024-04-25T01:29:17.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 client.16449 shutdown 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 -- 172.21.15.27:0/3057444958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c64485620 msgr2=0x558c6447fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c64485620 0x558c6447fcb0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f398800a350 tx=0x7f3988005b10 comp rx=0 tx=0).stop 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 -- 172.21.15.27:0/3057444958 shutdown_connections 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x558c644fc320 0x558c644fe740 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558c64485620 0x558c6447fcb0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558c6447af50 0x558c6447f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 --2- 172.21.15.27:0/3057444958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558c6447a630 0x558c6447f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f399c62a740 1 -- 172.21.15.27:0/3057444958 >> 172.21.15.27:0/3057444958 conn(0x558c64476950 msgr2=0x558c64475af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.577+0000 7f3972ffd640 0 client.16449 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f399c62a740 1 -- 172.21.15.27:0/3057444958 shutdown_connections 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f399c62a740 1 -- 172.21.15.27:0/3057444958 wait complete. 2024-04-25T01:29:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f399c62a740 20 client.16449 trim_cache size 0 max 16384 2024-04-25T01:29:17.566 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:29:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:29:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:29:17.566 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 -- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f397c0a2d00 msgr2=0x7f397c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f397c0a2d00 0x7f397c0a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f398400efd0 tx=0x7f3984004f50 comp rx=0 tx=0).stop 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 -- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 msgr2=0x7f399419c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 0x7f399419c790 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f398400eaa0 tx=0x7f398400eea0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 -- 172.21.15.27:0/647951318 shutdown_connections 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f397c0a2d00 0x7f397c0a51c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3994106690 0x7f399419c790 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3994101330 0x7f3994198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 --2- 172.21.15.27:0/647951318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3994100990 0x7f3994197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 -- 172.21.15.27:0/647951318 >> 172.21.15.27:0/647951318 conn(0x7f3994076440 msgr2=0x7f3994104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:17.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 -- 172.21.15.27:0/647951318 shutdown_connections 2024-04-25T01:29:17.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.578+0000 7f3999921640 1 -- 172.21.15.27:0/647951318 wait complete. 2024-04-25T01:29:17.568 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:29:17.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:29:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.691+0000 7fb105f22640 1 -- 172.21.15.27:0/1870649868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100103340 msgr2=0x7fb10010b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 --2- 172.21.15.27:0/1870649868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100103340 0x7fb10010b820 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009900 tx=0x7fb0f002a480 comp rx=0 tx=0).stop 2024-04-25T01:29:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 -- 172.21.15.27:0/1870649868 shutdown_connections 2024-04-25T01:29:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 --2- 172.21.15.27:0/1870649868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb10010bd60 0x7fb10010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 --2- 172.21.15.27:0/1870649868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100103340 0x7fb10010b820 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 --2- 172.21.15.27:0/1870649868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb100102a20 0x7fb100102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 -- 172.21.15.27:0/1870649868 >> 172.21.15.27:0/1870649868 conn(0x7fb1000777a0 msgr2=0x7fb100077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 -- 172.21.15.27:0/1870649868 shutdown_connections 2024-04-25T01:29:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 -- 172.21.15.27:0/1870649868 wait complete. 2024-04-25T01:29:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 Processor -- start 2024-04-25T01:29:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.692+0000 7fb105f22640 1 -- start start 2024-04-25T01:29:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb105f22640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100102a20 0x7fb100197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb105f22640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb100103340 0x7fb100198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb105f22640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb10010bd60 0x7fb10019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb105f22640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb100111b30 con 0x7fb100102a20 2024-04-25T01:29:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb105f22640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb1001119b0 con 0x7fb10010bd60 2024-04-25T01:29:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb105f22640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb100111cb0 con 0x7fb100103340 2024-04-25T01:29:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100102a20 0x7fb100197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100102a20 0x7fb100197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39534/0 (socket says 172.21.15.27:39534) 2024-04-25T01:29:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 -- 172.21.15.27:0/4059080284 learned_addr learned my addr 172.21.15.27:0/4059080284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0fffff640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb10010bd60 0x7fb10019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0feffd640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb100103340 0x7fb100198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 -- 172.21.15.27:0/4059080284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb100103340 msgr2=0x7fb100198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb100103340 0x7fb100198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 -- 172.21.15.27:0/4059080284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb10010bd60 msgr2=0x7fb10019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb10010bd60 0x7fb10019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 -- 172.21.15.27:0/4059080284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10019cfc0 con 0x7fb100102a20 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0ff7fe640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100102a20 0x7fb100197fd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e9f0 tx=0x7fb0f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0fcff9640 1 -- 172.21.15.27:0/4059080284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f400cd80 con 0x7fb100102a20 2024-04-25T01:29:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0fcff9640 1 -- 172.21.15.27:0/4059080284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0f4005500 con 0x7fb100102a20 2024-04-25T01:29:17.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10019d2b0 con 0x7fb100102a20 2024-04-25T01:29:17.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.693+0000 7fb0fcff9640 1 -- 172.21.15.27:0/4059080284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4010690 con 0x7fb100102a20 2024-04-25T01:29:17.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.694+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1001a4b40 con 0x7fb100102a20 2024-04-25T01:29:17.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.694+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1000689b0 con 0x7fb100102a20 2024-04-25T01:29:17.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.696+0000 7fb0fcff9640 1 -- 172.21.15.27:0/4059080284 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb0f40046c0 con 0x7fb100102a20 2024-04-25T01:29:17.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.696+0000 7fb0fcff9640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb0e80a2d00 0x7fb0e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:17.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.696+0000 7fb0fcff9640 1 -- 172.21.15.27:0/4059080284 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb0f40cfc00 con 0x7fb100102a20 2024-04-25T01:29:17.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.696+0000 7fb0feffd640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb0e80a2d00 0x7fb0e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:17.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.697+0000 7fb0feffd640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb0e80a2d00 0x7fb0e80a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009230 tx=0x7fb0f00058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.701+0000 7fb0fcff9640 1 -- 172.21.15.27:0/4059080284 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0f40d6080 con 0x7fb100102a20 2024-04-25T01:29:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.925+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb100106f40 con 0x7fb100102a20 2024-04-25T01:29:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.926+0000 7fb0fcff9640 1 -- 172.21.15.27:0/4059080284 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7fb0f409c490 con 0x7fb100102a20 2024-04-25T01:29:17.905 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:29:17.905 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:29:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:29:17.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb0e80a2d00 msgr2=0x7fb0e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb0e80a2d00 0x7fb0e80a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009230 tx=0x7fb0f00058d0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100102a20 msgr2=0x7fb100197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100102a20 0x7fb100197fd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e9f0 tx=0x7fb0f400edf0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 shutdown_connections 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb10010bd60 0x7fb10019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb100103340 0x7fb100198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb0e80a2d00 0x7fb0e80a51c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 --2- 172.21.15.27:0/4059080284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb100102a20 0x7fb100197fd0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 >> 172.21.15.27:0/4059080284 conn(0x7fb1000777a0 msgr2=0x7fb1000fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 shutdown_connections 2024-04-25T01:29:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:17.928+0000 7fb105f22640 1 -- 172.21.15.27:0/4059080284 wait complete. 2024-04-25T01:29:17.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:29:18.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.043+0000 7f9cf47ce640 1 -- 172.21.15.27:0/3705112659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec074330 msgr2=0x7f9cec10dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.043+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/3705112659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec074330 0x7f9cec10dae0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009900 tx=0x7f9cdc02a480 comp rx=0 tx=0).stop 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.043+0000 7f9cf47ce640 1 -- 172.21.15.27:0/3705112659 shutdown_connections 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.043+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/3705112659 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cec10e020 0x7f9cec110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.043+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/3705112659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec074330 0x7f9cec10dae0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.043+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/3705112659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cec073a10 0x7f9cec073df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.043+0000 7f9cf47ce640 1 -- 172.21.15.27:0/3705112659 >> 172.21.15.27:0/3705112659 conn(0x7f9cec078b90 msgr2=0x7f9cec078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 -- 172.21.15.27:0/3705112659 shutdown_connections 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 -- 172.21.15.27:0/3705112659 wait complete. 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 Processor -- start 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 -- start start 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec073a10 0x7f9cec19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cec074330 0x7f9cec19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cec10e020 0x7f9cec19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113d20 con 0x7f9cec073a10 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113ba0 con 0x7f9cec10e020 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf47ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113ea0 con 0x7f9cec074330 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf256a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec073a10 0x7f9cec19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf256a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec073a10 0x7f9cec19a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58204/0 (socket says 172.21.15.27:58204) 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.044+0000 7f9cf256a640 1 -- 172.21.15.27:0/2064549198 learned_addr learned my addr 172.21.15.27:0/2064549198 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cec10e020 0x7f9cec19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf1d69640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cec074330 0x7f9cec19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf256a640 1 -- 172.21.15.27:0/2064549198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cec074330 msgr2=0x7f9cec19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf256a640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cec074330 0x7f9cec19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf256a640 1 -- 172.21.15.27:0/2064549198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cec10e020 msgr2=0x7f9cec19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf256a640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cec10e020 0x7f9cec19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf256a640 1 -- 172.21.15.27:0/2064549198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec19f290 con 0x7f9cec073a10 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cec10e020 0x7f9cec19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf256a640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec073a10 0x7f9cec19a2a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800e9f0 tx=0x7f9ce800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9ce37fe640 1 -- 172.21.15.27:0/2064549198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce800cd80 con 0x7f9cec073a10 2024-04-25T01:29:18.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9ce37fe640 1 -- 172.21.15.27:0/2064549198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce8005500 con 0x7f9cec073a10 2024-04-25T01:29:18.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9ce37fe640 1 -- 172.21.15.27:0/2064549198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8010690 con 0x7f9cec073a10 2024-04-25T01:29:18.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf47ce640 1 -- 172.21.15.27:0/2064549198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec19f580 con 0x7f9cec073a10 2024-04-25T01:29:18.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.045+0000 7f9cf47ce640 1 -- 172.21.15.27:0/2064549198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec1a6e10 con 0x7f9cec073a10 2024-04-25T01:29:18.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.047+0000 7f9ce37fe640 1 -- 172.21.15.27:0/2064549198 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9ce80046c0 con 0x7f9cec073a10 2024-04-25T01:29:18.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.047+0000 7f9cf75bc740 1 Processor -- start 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf75bc740 1 -- start start 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb025960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb030030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb032960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558fcaf8a670 con 0x558fcb025ea0 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558fcaf32970 con 0x558fcb030570 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558fcaf86630 con 0x558fcb025580 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf256a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb030030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf256a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb030030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58212/0 (socket says 172.21.15.27:58212) 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf256a640 1 -- 172.21.15.27:0/1735262359 learned_addr learned my addr 172.21.15.27:0/1735262359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf1d69640 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb032960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb025960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9ce37fe640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9cd80a2d80 0x7f9cd80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf256a640 1 -- 172.21.15.27:0/1735262359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 msgr2=0x558fcb025960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf256a640 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb025960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf256a640 1 -- 172.21.15.27:0/1735262359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 msgr2=0x558fcb032960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9cf256a640 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb032960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.048+0000 7f9ce37fe640 1 -- 172.21.15.27:0/2064549198 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f9ce80cfcd0 con 0x7f9cec073a10 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.049+0000 7f9cf256a640 1 -- 172.21.15.27:0/1735262359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fcaf56400 con 0x558fcb025ea0 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.049+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb025960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.049+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9cd80a2d80 0x7f9cd80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.049+0000 7f9cf256a640 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb030030 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80cd000 tx=0x7f9ce8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5801f3fe7093dce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.049+0000 7f9ce0ff9640 1 -- 172.21.15.27:0/1735262359 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce809bc30 con 0x558fcb025ea0 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.049+0000 7f9ce0ff9640 1 -- 172.21.15.27:0/1735262359 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce80d1030 con 0x558fcb025ea0 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.049+0000 7f9ce0ff9640 1 -- 172.21.15.27:0/1735262359 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce809b7c0 con 0x558fcb025ea0 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- 172.21.15.27:0/1735262359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 msgr2=0x558fcb030030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9cd80a2d80 0x7f9cd80a5240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9cec06b010 tx=0x7f9ce4004a40 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb030030 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80cd000 tx=0x7f9ce8004a40 comp rx=0 tx=0).stop 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- 172.21.15.27:0/1735262359 shutdown_connections 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb032960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb030030 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/1735262359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb025960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- 172.21.15.27:0/1735262359 >> 172.21.15.27:0/1735262359 conn(0x558fcb0218a0 msgr2=0x558fcb021c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- 172.21.15.27:0/1735262359 shutdown_connections 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- 172.21.15.27:0/1735262359 wait complete. 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- start start 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 Processor -- start 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb02ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb02d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb02d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558fcb033300 con 0x558fcb025ea0 2024-04-25T01:29:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558fcaf46050 con 0x558fcb030570 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.050+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558fcb033480 con 0x558fcb025580 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf75bc740 10 client.0 _ll_register_callbacks cb 0x558fcac9dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb02d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb02d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58226/0 (socket says 172.21.15.27:58226) 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 -- 172.21.15.27:0/187006765 learned_addr learned my addr 172.21.15.27:0/187006765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf256a640 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb02ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb02d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 -- 172.21.15.27:0/187006765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 msgr2=0x558fcb02ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb02ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 -- 172.21.15.27:0/187006765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 msgr2=0x558fcb02d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb02d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 -- 172.21.15.27:0/187006765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fcaf17dc0 con 0x558fcb025ea0 2024-04-25T01:29:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb02d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf1d69640 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb02d200 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc02a9a0 tx=0x7f9cdc0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4807ad14c17445 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/187006765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc035040 con 0x558fcb025ea0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/187006765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f9cdc004240 con 0x558fcb025ea0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf75bc740 1 -- 172.21.15.27:0/187006765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ce80d1030 con 0x558fcb025ea0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/187006765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9cdc031a10 con 0x558fcb025ea0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf75bc740 10 client.16467 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.051+0000 7f9cf75bc740 1 -- 172.21.15.27:0/187006765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558fcac9e3f0 con 0x558fcb025ea0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/187006765 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce80d1030 con 0x558fcb025ea0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/187006765 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9cdc004ac0 con 0x558fcb025ea0 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cf75bc740 10 client.16467 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cf75bc740 10 client.16467 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:29:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cf75bc740 20 client.16467 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558fcb0405a0 0x558fcb0429c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cf75bc740 4 client.16467 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cf75bc740 1 -- 172.21.15.27:0/187006765 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fcaf97bb0 con 0x558fcb0405a0 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cf256a640 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558fcb0405a0 0x558fcb0429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cd5ffb640 10 client.16467 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.052+0000 7f9cf256a640 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558fcb0405a0 0x558fcb0429c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a889cb703af1d871 server_cookie=394c6d052be2122a in_seq=0 out_seq=0 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/187006765 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fcaf97bb0 con 0x558fcb0405a0 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cd5ffb640 10 client.16467 handle_command_reply: tid=0 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 client.16467 shutdown 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 -- 172.21.15.27:0/187006765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 msgr2=0x558fcb02d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb02d200 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc02a9a0 tx=0x7f9cdc0317e0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 -- 172.21.15.27:0/187006765 shutdown_connections 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558fcb0405a0 0x558fcb0429c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb02d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb02d200 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cd5ffb640 0 client.16467 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:29:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/187006765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025580 0x558fcb02ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 -- 172.21.15.27:0/187006765 >> 172.21.15.27:0/187006765 conn(0x558fcb0218a0 msgr2=0x558fcb0276a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 -- 172.21.15.27:0/187006765 shutdown_connections 2024-04-25T01:29:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 1 -- 172.21.15.27:0/187006765 wait complete. 2024-04-25T01:29:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.053+0000 7f9cf75bc740 20 client.16467 trim_cache size 0 max 16384 2024-04-25T01:29:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf75bc740 1 Processor -- start 2024-04-25T01:29:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf75bc740 1 -- start start 2024-04-25T01:29:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025580 0x558fcb02fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025ea0 0x558fcb048290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb0487d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1b37a0 con 0x558fcb025580 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1b1070 con 0x558fcb030570 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1aeec0 con 0x558fcb025ea0 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025580 0x558fcb02fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025580 0x558fcb02fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58236/0 (socket says 172.21.15.27:58236) 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 -- 172.21.15.27:0/209608742 learned_addr learned my addr 172.21.15.27:0/209608742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 -- 172.21.15.27:0/209608742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025ea0 msgr2=0x558fcb048290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 --2- 172.21.15.27:0/209608742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025ea0 0x558fcb048290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 -- 172.21.15.27:0/209608742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 msgr2=0x558fcb0487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 --2- 172.21.15.27:0/209608742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb0487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 -- 172.21.15.27:0/209608742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fcad687d0 con 0x558fcb025580 2024-04-25T01:29:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cf256a640 1 --2- 172.21.15.27:0/209608742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025580 0x558fcb02fa40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80cd0f0 tx=0x7f9ce8005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc86e5a24683fb2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cd7fff640 1 -- 172.21.15.27:0/209608742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8055030 con 0x558fcb025580 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.067+0000 7f9cd7fff640 1 -- 172.21.15.27:0/209608742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce80d1040 con 0x558fcb025580 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cd7fff640 1 -- 172.21.15.27:0/209608742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce809b9a0 con 0x558fcb025580 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 -- 172.21.15.27:0/209608742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025580 msgr2=0x558fcb02fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/209608742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025580 0x558fcb02fa40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80cd0f0 tx=0x7f9ce8005730 comp rx=0 tx=0).stop 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 -- 172.21.15.27:0/209608742 shutdown_connections 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/209608742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb030570 0x558fcb0487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/209608742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb025ea0 0x558fcb048290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/209608742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025580 0x558fcb02fa40 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 -- 172.21.15.27:0/209608742 >> 172.21.15.27:0/209608742 conn(0x558fcb0218a0 msgr2=0x558fcb027560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 -- 172.21.15.27:0/209608742 shutdown_connections 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 -- 172.21.15.27:0/209608742 wait complete. 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 -- start start 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.068+0000 7f9cf75bc740 1 Processor -- start 2024-04-25T01:29:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb025580 0x558fcb034720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb034c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf75bc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb030570 0x558fcb0a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1af530 con 0x558fcb025ea0 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1b09a0 con 0x558fcb025580 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf75bc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1afc00 con 0x558fcb030570 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf75bc740 10 client.0 _ll_register_callbacks cb 0x7f9cdc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb034c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb034c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58240/0 (socket says 172.21.15.27:58240) 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 -- 172.21.15.27:0/3408847165 learned_addr learned my addr 172.21.15.27:0/3408847165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf256a640 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb025580 0x558fcb034720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf2d6b640 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb030570 0x558fcb0a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 -- 172.21.15.27:0/3408847165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb030570 msgr2=0x558fcb0a5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb030570 0x558fcb0a5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 -- 172.21.15.27:0/3408847165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb025580 msgr2=0x558fcb034720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb025580 0x558fcb034720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 -- 172.21.15.27:0/3408847165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fcb0a6040 con 0x558fcb025ea0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf1d69640 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb034c60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc006010 tx=0x7f9cdc004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1633e6d2fa1f0a6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/3408847165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc035070 con 0x558fcb025ea0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/3408847165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f9cdc031db0 con 0x558fcb025ea0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.069+0000 7f9cf75bc740 1 -- 172.21.15.27:0/3408847165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ce80d1040 con 0x558fcb025ea0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/3408847165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9cdc005d50 con 0x558fcb025ea0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf75bc740 10 client.16473 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf75bc740 1 -- 172.21.15.27:0/3408847165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558fcb0a7240 con 0x558fcb025ea0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/3408847165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce80d1040 con 0x558fcb025ea0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/3408847165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9cdc045500 con 0x558fcb025ea0 2024-04-25T01:29:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf75bc740 10 client.16473 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf75bc740 10 client.16473 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf75bc740 20 client.16473 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558fcb0a79b0 0x558fcb0a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf75bc740 4 client.16473 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf75bc740 1 -- 172.21.15.27:0/3408847165 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558fcaf91f00 con 0x558fcb0a79b0 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.070+0000 7f9cf256a640 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558fcb0a79b0 0x558fcb0a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cd5ffb640 10 client.16473 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf256a640 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558fcb0a79b0 0x558fcb0a9dd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a17af2069780660 server_cookie=75f9d2e7f1257052 in_seq=0 out_seq=0 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cd5ffb640 1 -- 172.21.15.27:0/3408847165 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558fcaf91f00 con 0x558fcb0a79b0 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cd5ffb640 10 client.16473 handle_command_reply: tid=0 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 client.16473 shutdown 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 -- 172.21.15.27:0/3408847165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 msgr2=0x558fcb034c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb034c60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc006010 tx=0x7f9cdc004970 comp rx=0 tx=0).stop 2024-04-25T01:29:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 -- 172.21.15.27:0/3408847165 shutdown_connections 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558fcb0a79b0 0x558fcb0a9dd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558fcb030570 0x558fcb0a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cd5ffb640 0 client.16473 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fcb025ea0 0x558fcb034c60 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 --2- 172.21.15.27:0/3408847165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558fcb025580 0x558fcb034720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 -- 172.21.15.27:0/3408847165 >> 172.21.15.27:0/3408847165 conn(0x558fcb0218a0 msgr2=0x558fcb0202b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 -- 172.21.15.27:0/3408847165 shutdown_connections 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 1 -- 172.21.15.27:0/3408847165 wait complete. 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.071+0000 7f9cf75bc740 20 client.16473 trim_cache size 0 max 16384 2024-04-25T01:29:18.057 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:29:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 -- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9cd80a2d80 msgr2=0x7f9cd80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9cd80a2d80 0x7f9cd80a5240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9cec06b010 tx=0x7f9ce4004a40 comp rx=0 tx=0).stop 2024-04-25T01:29:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 -- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec073a10 msgr2=0x7f9cec19a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec073a10 0x7f9cec19a2a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800e9f0 tx=0x7f9ce800edf0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 -- 172.21.15.27:0/2064549198 shutdown_connections 2024-04-25T01:29:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cec10e020 0x7f9cec19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cec074330 0x7f9cec19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9cd80a2d80 0x7f9cd80a5240 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 --2- 172.21.15.27:0/2064549198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cec073a10 0x7f9cec19a2a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 -- 172.21.15.27:0/2064549198 >> 172.21.15.27:0/2064549198 conn(0x7f9cec078b90 msgr2=0x7f9cec0fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 -- 172.21.15.27:0/2064549198 shutdown_connections 2024-04-25T01:29:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.072+0000 7f9cf47ce640 1 -- 172.21.15.27:0/2064549198 wait complete. 2024-04-25T01:29:18.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:29:18.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 -- 172.21.15.27:0/2284013207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c1011e0 msgr2=0x7f556c10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 --2- 172.21.15.27:0/2284013207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c1011e0 0x7f556c10d500 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5568009cd0 tx=0x7f556802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 -- 172.21.15.27:0/2284013207 shutdown_connections 2024-04-25T01:29:18.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 --2- 172.21.15.27:0/2284013207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c1011e0 0x7f556c10d500 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 --2- 172.21.15.27:0/2284013207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f556c100820 0x7f556c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 --2- 172.21.15.27:0/2284013207 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f556c1066d0 0x7f556c106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 -- 172.21.15.27:0/2284013207 >> 172.21.15.27:0/2284013207 conn(0x7f556c077410 msgr2=0x7f556c077820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 -- 172.21.15.27:0/2284013207 shutdown_connections 2024-04-25T01:29:18.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.187+0000 7f557279e640 1 -- 172.21.15.27:0/2284013207 wait complete. 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557279e640 1 Processor -- start 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557279e640 1 -- start start 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557279e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c100820 0x7f556c198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557279e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f556c1011e0 0x7f556c199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557279e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f556c1066d0 0x7f556c1a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557279e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f556c110a10 con 0x7f556c100820 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557279e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f556c110890 con 0x7f556c1011e0 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557279e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f556c110b90 con 0x7f556c1066d0 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c100820 0x7f556c198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c100820 0x7f556c198cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58254/0 (socket says 172.21.15.27:58254) 2024-04-25T01:29:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 -- 172.21.15.27:0/102129346 learned_addr learned my addr 172.21.15.27:0/102129346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f5570f9b640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f556c1011e0 0x7f556c199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 -- 172.21.15.27:0/102129346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f556c1066d0 msgr2=0x7f556c1a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f556c1066d0 0x7f556c1a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 -- 172.21.15.27:0/102129346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f556c1011e0 msgr2=0x7f556c199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f556c1011e0 0x7f556c199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 -- 172.21.15.27:0/102129346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556c1a4290 con 0x7f556c100820 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f5570f9b640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f556c1011e0 0x7f556c199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.188+0000 7f557179c640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c100820 0x7f556c198cd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f556400c970 tx=0x7f556400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.189+0000 7f55627fc640 1 -- 172.21.15.27:0/102129346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5564004220 con 0x7f556c100820 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.189+0000 7f557279e640 1 -- 172.21.15.27:0/102129346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556c1a4460 con 0x7f556c100820 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.189+0000 7f55627fc640 1 -- 172.21.15.27:0/102129346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55640043c0 con 0x7f556c100820 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.189+0000 7f55627fc640 1 -- 172.21.15.27:0/102129346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556400f450 con 0x7f556c100820 2024-04-25T01:29:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.189+0000 7f557279e640 1 -- 172.21.15.27:0/102129346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f556c1a4980 con 0x7f556c100820 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.191+0000 7f55627fc640 1 -- 172.21.15.27:0/102129346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f55640079b0 con 0x7f556c100820 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.191+0000 7f55767ee740 1 Processor -- start 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.191+0000 7f55767ee740 1 -- start start 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f55767ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c255710 0x55bc6c255af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f55767ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c256030 0x55bc6c2601c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f55767ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c262af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f55767ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bc6c1ba670 con 0x55bc6c260700 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f55767ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bc6c162970 con 0x55bc6c256030 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f55767ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bc6c1b6630 con 0x55bc6c255710 2024-04-25T01:29:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f55627fc640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f55580a2d00 0x7f55580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c262af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c262af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58258/0 (socket says 172.21.15.27:58258) 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 -- 172.21.15.27:0/1691176532 learned_addr learned my addr 172.21.15.27:0/1691176532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5571f9d640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f55580a2d00 0x7f55580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f557179c640 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c256030 0x55bc6c2601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5571f9d640 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c255710 0x55bc6c255af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f55627fc640 1 -- 172.21.15.27:0/102129346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f55640cf5a0 con 0x7f556c100820 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 -- 172.21.15.27:0/1691176532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c255710 msgr2=0x55bc6c255af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c255710 0x55bc6c255af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 -- 172.21.15.27:0/1691176532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c256030 msgr2=0x55bc6c2601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c256030 0x55bc6c2601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 -- 172.21.15.27:0/1691176532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc6c186400 con 0x55bc6c260700 2024-04-25T01:29:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5571f9d640 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c255710 0x55bc6c255af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5570f9b640 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c262af0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f555c00bbe0 tx=0x7f555c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbe0b14b89d0e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.192+0000 7f5571f9d640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f55580a2d00 0x7f55580a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5568009a60 tx=0x7f5568005ef0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55577fe640 1 -- 172.21.15.27:0/1691176532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c017620 con 0x55bc6c260700 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55577fe640 1 -- 172.21.15.27:0/1691176532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f555c017ca0 con 0x55bc6c260700 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55577fe640 1 -- 172.21.15.27:0/1691176532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c00fd40 con 0x55bc6c260700 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 -- 172.21.15.27:0/1691176532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 msgr2=0x55bc6c262af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c262af0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f555c00bbe0 tx=0x7f555c00d710 comp rx=0 tx=0).stop 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 -- 172.21.15.27:0/1691176532 shutdown_connections 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c262af0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c256030 0x55bc6c2601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 --2- 172.21.15.27:0/1691176532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c255710 0x55bc6c255af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 -- 172.21.15.27:0/1691176532 >> 172.21.15.27:0/1691176532 conn(0x55bc6c251a30 msgr2=0x55bc6c251e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 -- 172.21.15.27:0/1691176532 shutdown_connections 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.193+0000 7f55767ee740 1 -- 172.21.15.27:0/1691176532 wait complete. 2024-04-25T01:29:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 1 -- start start 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 1 Processor -- start 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c255710 0x55bc6c260060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c260700 0x55bc6c25d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bc6c2634f0 con 0x55bc6c255710 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bc6c176050 con 0x55bc6c260700 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bc6c263670 con 0x55bc6c256030 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f55767ee740 10 client.0 _ll_register_callbacks cb 0x55bc6becdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f557179c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c255710 0x55bc6c260060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f557179c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c255710 0x55bc6c260060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58274/0 (socket says 172.21.15.27:58274) 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f557179c640 1 -- 172.21.15.27:0/1181311481 learned_addr learned my addr 172.21.15.27:0/1181311481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f5571f9d640 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c260700 0x55bc6c25d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.194+0000 7f5570f9b640 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f557179c640 1 -- 172.21.15.27:0/1181311481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 msgr2=0x55bc6c25d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f557179c640 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f557179c640 1 -- 172.21.15.27:0/1181311481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c260700 msgr2=0x55bc6c25d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f557179c640 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c260700 0x55bc6c25d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f557179c640 1 -- 172.21.15.27:0/1181311481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc6c147dc0 con 0x55bc6c255710 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f5571f9d640 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c260700 0x55bc6c25d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f557179c640 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c255710 0x55bc6c260060 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f55640cc9c0 tx=0x7f556406a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d799b8bbfc100f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f5554ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f5554ff9640 1 -- 172.21.15.27:0/1181311481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556401d030 con 0x55bc6c255710 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f5554ff9640 1 -- 172.21.15.27:0/1181311481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5564097da0 con 0x55bc6c255710 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 1 -- 172.21.15.27:0/1181311481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f555c017ca0 con 0x55bc6c255710 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f5554ff9640 1 -- 172.21.15.27:0/1181311481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f55640667c0 con 0x55bc6c255710 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 10 client.16485 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:18.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 1 -- 172.21.15.27:0/1181311481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bc6bece3f0 con 0x55bc6c255710 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f5554ff9640 1 -- 172.21.15.27:0/1181311481 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f555c017ca0 con 0x55bc6c255710 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f5554ff9640 1 -- 172.21.15.27:0/1181311481 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f55640cf1b0 con 0x55bc6c255710 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 10 client.16485 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 10 client.16485 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 20 client.16485 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc6c2705e0 0x55bc6c272a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 4 client.16485 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.195+0000 7f55767ee740 1 -- 172.21.15.27:0/1181311481 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc6c1c7bb0 con 0x55bc6c2705e0 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.196+0000 7f5570f9b640 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc6c2705e0 0x55bc6c272a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.196+0000 7f5570f9b640 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc6c2705e0 0x55bc6c272a30 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6ecd74c32a254344 server_cookie=97f5b301a7a22bff in_seq=0 out_seq=0 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.196+0000 7f5554ff9640 10 client.16485 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f5554ff9640 1 -- 172.21.15.27:0/1181311481 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc6c1c7bb0 con 0x55bc6c2705e0 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f5554ff9640 10 client.16485 handle_command_reply: tid=0 2024-04-25T01:29:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 client.16485 shutdown 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 -- 172.21.15.27:0/1181311481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c255710 msgr2=0x55bc6c260060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c255710 0x55bc6c260060 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f55640cc9c0 tx=0x7f556406a660 comp rx=0 tx=0).stop 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 -- 172.21.15.27:0/1181311481 shutdown_connections 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc6c2705e0 0x55bc6c272a30 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c260700 0x55bc6c25d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 --2- 172.21.15.27:0/1181311481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c255710 0x55bc6c260060 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 -- 172.21.15.27:0/1181311481 >> 172.21.15.27:0/1181311481 conn(0x55bc6c251a30 msgr2=0x55bc6c257830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f5554ff9640 0 client.16485 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 -- 172.21.15.27:0/1181311481 shutdown_connections 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 1 -- 172.21.15.27:0/1181311481 wait complete. 2024-04-25T01:29:18.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.197+0000 7f55767ee740 20 client.16485 trim_cache size 0 max 16384 2024-04-25T01:29:18.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f55767ee740 1 Processor -- start 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f55767ee740 1 -- start start 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f55767ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 0x55bc6c250b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f55767ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f55767ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2688c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f55767ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f556c1b12b0 con 0x55bc6c260700 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f55767ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f556c1aeb80 con 0x55bc6c255710 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f55767ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f556c1ac9d0 con 0x55bc6c256030 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5571f9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5571f9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2688c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58284/0 (socket says 172.21.15.27:58284) 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5571f9d640 1 -- 172.21.15.27:0/887445413 learned_addr learned my addr 172.21.15.27:0/887445413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5570f9b640 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f557179c640 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 0x55bc6c250b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5571f9d640 1 -- 172.21.15.27:0/887445413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 msgr2=0x55bc6c25fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5571f9d640 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5571f9d640 1 -- 172.21.15.27:0/887445413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 msgr2=0x55bc6c250b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5571f9d640 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 0x55bc6c250b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5571f9d640 1 -- 172.21.15.27:0/887445413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc6bf987d0 con 0x55bc6c260700 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.211+0000 7f5570f9b640 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f5571f9d640 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2688c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5568009cd0 tx=0x7f556803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bdacb4f6364f6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f5556ffd640 1 -- 172.21.15.27:0/887445413 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568043040 con 0x55bc6c260700 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f5556ffd640 1 -- 172.21.15.27:0/887445413 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5568036050 con 0x55bc6c260700 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f5556ffd640 1 -- 172.21.15.27:0/887445413 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556803b660 con 0x55bc6c260700 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 -- 172.21.15.27:0/887445413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 msgr2=0x55bc6c2688c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2688c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5568009cd0 tx=0x7f556803c000 comp rx=0 tx=0).stop 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 -- 172.21.15.27:0/887445413 shutdown_connections 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2688c0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c25fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 --2- 172.21.15.27:0/887445413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 0x55bc6c250b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 -- 172.21.15.27:0/887445413 >> 172.21.15.27:0/887445413 conn(0x55bc6c251a30 msgr2=0x55bc6c26b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 -- 172.21.15.27:0/887445413 shutdown_connections 2024-04-25T01:29:18.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.212+0000 7f55767ee740 1 -- 172.21.15.27:0/887445413 wait complete. 2024-04-25T01:29:18.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 1 -- start start 2024-04-25T01:29:18.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 1 Processor -- start 2024-04-25T01:29:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 0x55bc6c266b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c267080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f556c1ad040 con 0x55bc6c260700 2024-04-25T01:29:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f556c1ae4b0 con 0x55bc6c255710 2024-04-25T01:29:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f556c1ad710 con 0x55bc6c256030 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f55767ee740 10 client.0 _ll_register_callbacks cb 0x7f5564003920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f5571f9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f5571f9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58300/0 (socket says 172.21.15.27:58300) 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f5571f9d640 1 -- 172.21.15.27:0/1522906685 learned_addr learned my addr 172.21.15.27:0/1522906685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f557179c640 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 0x55bc6c266b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.213+0000 7f5570f9b640 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c267080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5571f9d640 1 -- 172.21.15.27:0/1522906685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 msgr2=0x55bc6c267080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5571f9d640 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c267080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5571f9d640 1 -- 172.21.15.27:0/1522906685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 msgr2=0x55bc6c266b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5571f9d640 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 0x55bc6c266b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5571f9d640 1 -- 172.21.15.27:0/1522906685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc6c2d0110 con 0x55bc6c260700 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5571f9d640 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2cfb20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f556802ae00 tx=0x7f5568005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e60ce5f719738fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5554ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5554ff9640 1 -- 172.21.15.27:0/1522906685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568043040 con 0x55bc6c260700 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5554ff9640 1 -- 172.21.15.27:0/1522906685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5568003d80 con 0x55bc6c260700 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f55767ee740 1 -- 172.21.15.27:0/1522906685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5568036050 con 0x55bc6c260700 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5554ff9640 1 -- 172.21.15.27:0/1522906685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5568068050 con 0x55bc6c260700 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f55767ee740 10 client.16491 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f55767ee740 1 -- 172.21.15.27:0/1522906685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bc6c2d7490 con 0x55bc6c260700 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.214+0000 7f5554ff9640 1 -- 172.21.15.27:0/1522906685 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5568036050 con 0x55bc6c260700 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f5554ff9640 1 -- 172.21.15.27:0/1522906685 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5568030070 con 0x55bc6c260700 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f55767ee740 10 client.16491 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f55767ee740 10 client.16491 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f55767ee740 20 client.16491 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f55767ee740 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc6c2d7b00 0x55bc6c2d9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f55767ee740 4 client.16491 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:29:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f55767ee740 1 -- 172.21.15.27:0/1522906685 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc6c1c1f00 con 0x55bc6c2d7b00 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f557179c640 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc6c2d7b00 0x55bc6c2d9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f557179c640 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc6c2d7b00 0x55bc6c2d9ef0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57440e6d0d0f529e server_cookie=1f4d4667175db22e in_seq=0 out_seq=0 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f5554ff9640 10 client.16491 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f5554ff9640 1 -- 172.21.15.27:0/1522906685 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc6c1c1f00 con 0x55bc6c2d7b00 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f5554ff9640 10 client.16491 handle_command_reply: tid=0 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.215+0000 7f55767ee740 1 client.16491 shutdown 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 -- 172.21.15.27:0/1522906685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 msgr2=0x55bc6c2cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2cfb20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f556802ae00 tx=0x7f5568005b50 comp rx=0 tx=0).stop 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 -- 172.21.15.27:0/1522906685 shutdown_connections 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc6c2d7b00 0x55bc6c2d9ef0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc6c260700 0x55bc6c2cfb20 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc6c256030 0x55bc6c267080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f5554ff9640 0 client.16491 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:29:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 --2- 172.21.15.27:0/1522906685 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc6c255710 0x55bc6c266b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 -- 172.21.15.27:0/1522906685 >> 172.21.15.27:0/1522906685 conn(0x55bc6c251a30 msgr2=0x55bc6c2509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 -- 172.21.15.27:0/1522906685 shutdown_connections 2024-04-25T01:29:18.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 1 -- 172.21.15.27:0/1522906685 wait complete. 2024-04-25T01:29:18.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f55767ee740 20 client.16491 trim_cache size 0 max 16384 2024-04-25T01:29:18.201 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 -- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f55580a2d00 msgr2=0x7f55580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f55580a2d00 0x7f55580a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5568009a60 tx=0x7f5568005ef0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 -- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c100820 msgr2=0x7f556c198cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c100820 0x7f556c198cd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f556400c970 tx=0x7f556400ce30 comp rx=0 tx=0).stop 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 -- 172.21.15.27:0/102129346 shutdown_connections 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f556c1066d0 0x7f556c1a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f556c1011e0 0x7f556c199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f55580a2d00 0x7f55580a51c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 --2- 172.21.15.27:0/102129346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f556c100820 0x7f556c198cd0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 -- 172.21.15.27:0/102129346 >> 172.21.15.27:0/102129346 conn(0x7f556c077410 msgr2=0x7f556c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.216+0000 7f557279e640 1 -- 172.21.15.27:0/102129346 shutdown_connections 2024-04-25T01:29:18.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.217+0000 7f557279e640 1 -- 172.21.15.27:0/102129346 wait complete. 2024-04-25T01:29:18.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:29:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 -- 172.21.15.27:0/2014092930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 msgr2=0x7f4e6006a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/2014092930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 0x7f4e6006a240 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009930 tx=0x7f4e4c02a420 comp rx=0 tx=0).stop 2024-04-25T01:29:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 -- 172.21.15.27:0/2014092930 shutdown_connections 2024-04-25T01:29:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/2014092930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e6006a780 0x7f4e6010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/2014092930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 0x7f4e6006a240 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/2014092930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e60107730 0x7f4e60107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 -- 172.21.15.27:0/2014092930 >> 172.21.15.27:0/2014092930 conn(0x7f4e600776a0 msgr2=0x7f4e60077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 -- 172.21.15.27:0/2014092930 shutdown_connections 2024-04-25T01:29:18.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 -- 172.21.15.27:0/2014092930 wait complete. 2024-04-25T01:29:18.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.332+0000 7f4e66d9d640 1 Processor -- start 2024-04-25T01:29:18.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e66d9d640 1 -- start start 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 0x7f4e60100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e6006a780 0x7f4e60100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e60107730 0x7f4e601010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e66d9d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e60111b40 con 0x7f4e60069dc0 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e66d9d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4e601119c0 con 0x7f4e6006a780 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e66d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e60111cc0 con 0x7f4e60107730 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 0x7f4e60100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 0x7f4e60100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58314/0 (socket says 172.21.15.27:58314) 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 -- 172.21.15.27:0/1556740956 learned_addr learned my addr 172.21.15.27:0/1556740956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e6006a780 0x7f4e60100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 -- 172.21.15.27:0/1556740956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e60107730 msgr2=0x7f4e601010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e60107730 0x7f4e601010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 -- 172.21.15.27:0/1556740956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e6006a780 msgr2=0x7f4e60100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e6006a780 0x7f4e60100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 -- 172.21.15.27:0/1556740956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e601a64a0 con 0x7f4e60069dc0 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e6006a780 0x7f4e60100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e64b39640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 0x7f4e60100670 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4e5400d950 tx=0x7f4e5400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e5dffb640 1 -- 172.21.15.27:0/1556740956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e54017030 con 0x7f4e60069dc0 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e5dffb640 1 -- 172.21.15.27:0/1556740956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e54004020 con 0x7f4e60069dc0 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e66d9d640 1 -- 172.21.15.27:0/1556740956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e601a6790 con 0x7f4e60069dc0 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.333+0000 7f4e5dffb640 1 -- 172.21.15.27:0/1556740956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e540047a0 con 0x7f4e60069dc0 2024-04-25T01:29:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.334+0000 7f4e66d9d640 1 -- 172.21.15.27:0/1556740956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e601a6d00 con 0x7f4e60069dc0 2024-04-25T01:29:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e5dffb640 1 -- 172.21.15.27:0/1556740956 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4e540041d0 con 0x7f4e60069dc0 2024-04-25T01:29:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e69b81740 1 Processor -- start 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e69b81740 1 -- start start 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e5dffb640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4e500a2d80 0x7f4e500a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd85960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd90030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd92960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e69b81740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c54dcea670 con 0x55c54dd85580 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e69b81740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c54dc92970 con 0x55c54dd90570 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.336+0000 7f4e69b81740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c54dce6630 con 0x55c54dd85ea0 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e6533a640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4e500a2d80 0x7f4e500a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd85960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e5dffb640 1 -- 172.21.15.27:0/1556740956 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4e540cf1b0 con 0x7f4e60069dc0 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd85960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58318/0 (socket says 172.21.15.27:58318) 2024-04-25T01:29:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 -- 172.21.15.27:0/3208930614 learned_addr learned my addr 172.21.15.27:0/3208930614 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd90030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e6533a640 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd92960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 -- 172.21.15.27:0/3208930614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 msgr2=0x55c54dd90030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd90030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 -- 172.21.15.27:0/3208930614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 msgr2=0x55c54dd92960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd92960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 -- 172.21.15.27:0/3208930614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c54dcb6400 con 0x55c54dd85580 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e6533a640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4e500a2d80 0x7f4e500a5240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4e58005fd0 tx=0x7f4e580138e0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e64b39640 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd85960 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4e54097d10 tx=0x7f4e54018cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5bce7b8133a576 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e4affd640 1 -- 172.21.15.27:0/3208930614 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5409ae80 con 0x55c54dd85580 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e4affd640 1 -- 172.21.15.27:0/3208930614 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e5409ba40 con 0x55c54dd85580 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.337+0000 7f4e4affd640 1 -- 172.21.15.27:0/3208930614 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5409aca0 con 0x55c54dd85580 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 -- 172.21.15.27:0/3208930614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 msgr2=0x55c54dd85960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd85960 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4e54097d10 tx=0x7f4e54018cc0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 -- 172.21.15.27:0/3208930614 shutdown_connections 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd92960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd90030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 --2- 172.21.15.27:0/3208930614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd85960 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 -- 172.21.15.27:0/3208930614 >> 172.21.15.27:0/3208930614 conn(0x55c54dd818a0 msgr2=0x55c54dd81c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 -- 172.21.15.27:0/3208930614 shutdown_connections 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 -- 172.21.15.27:0/3208930614 wait complete. 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 -- start start 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.338+0000 7f4e69b81740 1 Processor -- start 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85580 0x55c54dd8fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd8d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e69b81740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c54dd93360 con 0x55c54dd85ea0 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e69b81740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c54dca6050 con 0x55c54dd90570 2024-04-25T01:29:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e69b81740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c54dd934e0 con 0x55c54dd85580 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e69b81740 10 client.0 _ll_register_callbacks cb 0x55c54d9fdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58326/0 (socket says 172.21.15.27:58326) 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 -- 172.21.15.27:0/2108324627 learned_addr learned my addr 172.21.15.27:0/2108324627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e6533a640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd8d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e64b39640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85580 0x55c54dd8fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 -- 172.21.15.27:0/2108324627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85580 msgr2=0x55c54dd8fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85580 0x55c54dd8fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 -- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 msgr2=0x55c54dd8d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd8d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 -- 172.21.15.27:0/2108324627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54dc77dc0 con 0x55c54dd85ea0 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e6533a640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd8d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e64b39640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85580 0x55c54dd8fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8d1b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009930 tx=0x7f4e4c031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9e4521792391f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.339+0000 7f4e3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2108324627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c035040 con 0x55c54dd85ea0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2108324627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4e4c004240 con 0x55c54dd85ea0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 1 -- 172.21.15.27:0/2108324627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e5409ba40 con 0x55c54dd85ea0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2108324627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4e4c04e050 con 0x55c54dd85ea0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 10 client.16503 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 1 -- 172.21.15.27:0/2108324627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c54d9fe3f0 con 0x55c54dd85ea0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2108324627 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e5409ba40 con 0x55c54dd85ea0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2108324627 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4e4c004b00 con 0x55c54dd85ea0 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 10 client.16503 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 10 client.16503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 20 client.16503 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c54dda05c0 0x55c54dda2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 4 client.16503 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.340+0000 7f4e69b81740 1 -- 172.21.15.27:0/2108324627 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c54dcf7bb0 con 0x55c54dda05c0 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.341+0000 7f4e64b39640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c54dda05c0 0x55c54dda2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.341+0000 7f4e3ffff640 10 client.16503 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.341+0000 7f4e64b39640 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c54dda05c0 0x55c54dda2a60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4ab83d61e0df815 server_cookie=1924f660aec3367c in_seq=0 out_seq=0 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2108324627 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c54dcf7bb0 con 0x55c54dda05c0 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e3ffff640 10 client.16503 handle_command_reply: tid=0 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 client.16503 shutdown 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 -- 172.21.15.27:0/2108324627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 msgr2=0x55c54dd8d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8d1b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009930 tx=0x7f4e4c031920 comp rx=0 tx=0).stop 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 -- 172.21.15.27:0/2108324627 shutdown_connections 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c54dda05c0 0x55c54dda2a60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd8d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8d1b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e3ffff640 0 client.16503 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2108324627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85580 0x55c54dd8fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 -- 172.21.15.27:0/2108324627 >> 172.21.15.27:0/2108324627 conn(0x55c54dd818a0 msgr2=0x55c54dd876a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 -- 172.21.15.27:0/2108324627 shutdown_connections 2024-04-25T01:29:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 1 -- 172.21.15.27:0/2108324627 wait complete. 2024-04-25T01:29:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.342+0000 7f4e69b81740 20 client.16503 trim_cache size 0 max 16384 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e69b81740 1 Processor -- start 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e69b81740 1 -- start start 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd85580 0x55c54dd9c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd90570 0x55c54dda8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e69b81740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e601b36a0 con 0x55c54dd85ea0 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e69b81740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4e601b0f70 con 0x55c54dd85580 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e69b81740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e601aedc0 con 0x55c54dd90570 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58336/0 (socket says 172.21.15.27:58336) 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 -- 172.21.15.27:0/2932306149 learned_addr learned my addr 172.21.15.27:0/2932306149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 -- 172.21.15.27:0/2932306149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd90570 msgr2=0x55c54dda8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2932306149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd90570 0x55c54dda8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 -- 172.21.15.27:0/2932306149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd85580 msgr2=0x55c54dd9c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2932306149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd85580 0x55c54dd9c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 -- 172.21.15.27:0/2932306149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c54dac87d0 con 0x55c54dd85ea0 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2932306149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8c1b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c003640 tx=0x7f4e4c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8630fdc7dd8c01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.356+0000 7f4e4a7fc640 1 -- 172.21.15.27:0/2932306149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c035070 con 0x55c54dd85ea0 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e4a7fc640 1 -- 172.21.15.27:0/2932306149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4c005000 con 0x55c54dd85ea0 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e4a7fc640 1 -- 172.21.15.27:0/2932306149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c003dc0 con 0x55c54dd85ea0 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 -- 172.21.15.27:0/2932306149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 msgr2=0x55c54dd8c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2932306149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8c1b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c003640 tx=0x7f4e4c001e50 comp rx=0 tx=0).stop 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 -- 172.21.15.27:0/2932306149 shutdown_connections 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2932306149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd90570 0x55c54dda8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2932306149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85ea0 0x55c54dd8c1b0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2932306149 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd85580 0x55c54dd9c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 -- 172.21.15.27:0/2932306149 >> 172.21.15.27:0/2932306149 conn(0x55c54dd818a0 msgr2=0x55c54dd8baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 -- 172.21.15.27:0/2932306149 shutdown_connections 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 -- 172.21.15.27:0/2932306149 wait complete. 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 -- start start 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.357+0000 7f4e69b81740 1 Processor -- start 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd98760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd98ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e69b81740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd95cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e69b81740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e601af430 con 0x55c54dd85580 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e69b81740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4e601b08a0 con 0x55c54dd90570 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e69b81740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e601afb00 con 0x55c54dd85ea0 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e69b81740 10 client.0 _ll_register_callbacks cb 0x7f4e4c0315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd98760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd98760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58344/0 (socket says 172.21.15.27:58344) 2024-04-25T01:29:18.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 -- 172.21.15.27:0/2625956210 learned_addr learned my addr 172.21.15.27:0/2625956210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd98ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e6533a640 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd95cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 -- 172.21.15.27:0/2625956210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 msgr2=0x55c54dd98ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd98ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 -- 172.21.15.27:0/2625956210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 msgr2=0x55c54dd95cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd95cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 -- 172.21.15.27:0/2625956210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54dd964d0 con 0x55c54dd85580 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd98ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e64b39640 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd98760 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4e540977a0 tx=0x7f4e54095cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a798e207ade65c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.358+0000 7f4e3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2625956210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5409ba40 con 0x55c54dd85580 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2625956210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4e540979b0 con 0x55c54dd85580 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 1 -- 172.21.15.27:0/2625956210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e4c005000 con 0x55c54dd85580 2024-04-25T01:29:18.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2625956210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4e5409b340 con 0x55c54dd85580 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 10 client.16509 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 1 -- 172.21.15.27:0/2625956210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c54de06870 con 0x55c54dd85580 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2625956210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4c005000 con 0x55c54dd85580 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2625956210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4e54062020 con 0x55c54dd85580 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 10 client.16509 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 10 client.16509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 20 client.16509 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c54de06bb0 0x55c54de08fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 4 client.16509 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e69b81740 1 -- 172.21.15.27:0/2625956210 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c54dcf1f00 con 0x55c54de06bb0 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.359+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c54de06bb0 0x55c54de08fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e5ffff640 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c54de06bb0 0x55c54de08fa0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70d940954adae7b7 server_cookie=a890a8363fe6fbb4 in_seq=0 out_seq=0 2024-04-25T01:29:18.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e3ffff640 10 client.16509 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e3ffff640 1 -- 172.21.15.27:0/2625956210 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c54dcf1f00 con 0x55c54de06bb0 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e3ffff640 10 client.16509 handle_command_reply: tid=0 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 client.16509 shutdown 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 -- 172.21.15.27:0/2625956210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 msgr2=0x55c54dd98760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd98760 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4e540977a0 tx=0x7f4e54095cd0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 -- 172.21.15.27:0/2625956210 shutdown_connections 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c54de06bb0 0x55c54de08fa0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c54dd90570 0x55c54dd95cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c54dd85ea0 0x55c54dd98ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 --2- 172.21.15.27:0/2625956210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c54dd85580 0x55c54dd98760 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e3ffff640 0 client.16509 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 -- 172.21.15.27:0/2625956210 >> 172.21.15.27:0/2625956210 conn(0x55c54dd818a0 msgr2=0x55c54dd8c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 -- 172.21.15.27:0/2625956210 shutdown_connections 2024-04-25T01:29:18.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 1 -- 172.21.15.27:0/2625956210 wait complete. 2024-04-25T01:29:18.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.360+0000 7f4e69b81740 20 client.16509 trim_cache size 0 max 16384 2024-04-25T01:29:18.346 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:29:18.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 -- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4e500a2d80 msgr2=0x7f4e500a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4e500a2d80 0x7f4e500a5240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4e58005fd0 tx=0x7f4e580138e0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 -- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 msgr2=0x7f4e60100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:18.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 0x7f4e60100670 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4e5400d950 tx=0x7f4e5400de10 comp rx=0 tx=0).stop 2024-04-25T01:29:18.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 -- 172.21.15.27:0/1556740956 shutdown_connections 2024-04-25T01:29:18.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e60107730 0x7f4e601010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4e6006a780 0x7f4e60100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4e500a2d80 0x7f4e500a5240 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 --2- 172.21.15.27:0/1556740956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e60069dc0 0x7f4e60100670 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:18.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 -- 172.21.15.27:0/1556740956 >> 172.21.15.27:0/1556740956 conn(0x7f4e600776a0 msgr2=0x7f4e600fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:18.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 -- 172.21.15.27:0/1556740956 shutdown_connections 2024-04-25T01:29:18.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:18.361+0000 7f4e66d9d640 1 -- 172.21.15.27:0/1556740956 wait complete. 2024-04-25T01:29:18.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:18 smithi153 ceph-mon[32621]: pgmap v119: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:29:18.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:18 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4059080284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:18.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:18 smithi139 ceph-mon[32760]: pgmap v119: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:29:18.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:18 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4059080284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:18 smithi027 ceph-mon[29197]: pgmap v119: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:29:18.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:18 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4059080284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:19.351 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:29:19.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:29:19.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:29:19.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 -- 172.21.15.27:0/1605354148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd88406a780 msgr2=0x7fd88410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 --2- 172.21.15.27:0/1605354148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd88406a780 0x7fd88410e5f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009cd0 tx=0x7fd87c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 -- 172.21.15.27:0/1605354148 shutdown_connections 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 --2- 172.21.15.27:0/1605354148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd88406a780 0x7fd88410e5f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 --2- 172.21.15.27:0/1605354148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd884069dc0 0x7fd88406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 --2- 172.21.15.27:0/1605354148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd884107730 0x7fd884107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 -- 172.21.15.27:0/1605354148 >> 172.21.15.27:0/1605354148 conn(0x7fd8840776a0 msgr2=0x7fd884077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 -- 172.21.15.27:0/1605354148 shutdown_connections 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.491+0000 7fd88c653640 1 -- 172.21.15.27:0/1605354148 wait complete. 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88c653640 1 Processor -- start 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88c653640 1 -- start start 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88c653640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd884069dc0 0x7fd884197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88c653640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88406a780 0x7fd8841984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88c653640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd884107730 0x7fd88419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88c653640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd884111b40 con 0x7fd884107730 2024-04-25T01:29:19.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88c653640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841119c0 con 0x7fd884069dc0 2024-04-25T01:29:19.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88c653640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd884111cc0 con 0x7fd88406a780 2024-04-25T01:29:19.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88abf0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd884107730 0x7fd88419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88abf0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd884107730 0x7fd88419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58372/0 (socket says 172.21.15.27:58372) 2024-04-25T01:29:19.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88abf0640 1 -- 172.21.15.27:0/3560954275 learned_addr learned my addr 172.21.15.27:0/3560954275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd889bee640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88406a780 0x7fd8841984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.492+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd884069dc0 0x7fd884197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd88abf0640 1 -- 172.21.15.27:0/3560954275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88406a780 msgr2=0x7fd8841984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd88abf0640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88406a780 0x7fd8841984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd88abf0640 1 -- 172.21.15.27:0/3560954275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd884069dc0 msgr2=0x7fd884197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd88abf0640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd884069dc0 0x7fd884197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd88abf0640 1 -- 172.21.15.27:0/3560954275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88419cf50 con 0x7fd884107730 2024-04-25T01:29:19.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd88abf0640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd884107730 0x7fd88419c820 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd87c005cc0 tx=0x7fd87c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd87b7fe640 1 -- 172.21.15.27:0/3560954275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c004040 con 0x7fd884107730 2024-04-25T01:29:19.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd87b7fe640 1 -- 172.21.15.27:0/3560954275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd87c0046c0 con 0x7fd884107730 2024-04-25T01:29:19.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd88c653640 1 -- 172.21.15.27:0/3560954275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88419d1e0 con 0x7fd884107730 2024-04-25T01:29:19.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd87b7fe640 1 -- 172.21.15.27:0/3560954275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c046480 con 0x7fd884107730 2024-04-25T01:29:19.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.493+0000 7fd88c653640 1 -- 172.21.15.27:0/3560954275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8841a4ad0 con 0x7fd884107730 2024-04-25T01:29:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.495+0000 7fd87b7fe640 1 -- 172.21.15.27:0/3560954275 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd87c0305a0 con 0x7fd884107730 2024-04-25T01:29:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.496+0000 7fd88f433740 1 Processor -- start 2024-04-25T01:29:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.496+0000 7fd88f433740 1 -- start start 2024-04-25T01:29:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88f433740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88f433740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88f433740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 0x5621d49be400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88f433740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5621d49152c0 con 0x5621d49b2980 2024-04-25T01:29:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd87b7fe640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd8700a2d00 0x7fd8700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88f433740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5621d48bd910 con 0x5621d49bc010 2024-04-25T01:29:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88f433740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5621d4911280 con 0x5621d49b1290 2024-04-25T01:29:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd889bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd889bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58378/0 (socket says 172.21.15.27:58378) 2024-04-25T01:29:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd889bee640 1 -- 172.21.15.27:0/1110534258 learned_addr learned my addr 172.21.15.27:0/1110534258 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88abf0640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd8700a2d00 0x7fd8700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd889bee640 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 0x5621d49be400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd87b7fe640 1 -- 172.21.15.27:0/3560954275 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd87c02e030 con 0x7fd884107730 2024-04-25T01:29:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88a3ef640 1 -- 172.21.15.27:0/1110534258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 msgr2=0x5621d49be400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 0x5621d49be400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88a3ef640 1 -- 172.21.15.27:0/1110534258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 msgr2=0x5621d49b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd88a3ef640 1 -- 172.21.15.27:0/1110534258 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621d48cefb0 con 0x5621d49b1290 2024-04-25T01:29:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.497+0000 7fd889bee640 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.498+0000 7fd88abf0640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd8700a2d00 0x7fd8700a51c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0b9540 tx=0x7fd87c0a6040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.498+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b1710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd8800045b0 tx=0x7fd88000bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32654003e1074426 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.498+0000 7fd878bfe640 1 -- 172.21.15.27:0/1110534258 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880003bd0 con 0x5621d49b1290 2024-04-25T01:29:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.498+0000 7fd878bfe640 1 -- 172.21.15.27:0/1110534258 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd880015070 con 0x5621d49b1290 2024-04-25T01:29:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.498+0000 7fd878bfe640 1 -- 172.21.15.27:0/1110534258 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880004d00 con 0x5621d49b1290 2024-04-25T01:29:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- 172.21.15.27:0/1110534258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 msgr2=0x5621d49b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b1710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd8800045b0 tx=0x7fd88000bb20 comp rx=0 tx=0).stop 2024-04-25T01:29:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- 172.21.15.27:0/1110534258 shutdown_connections 2024-04-25T01:29:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 0x5621d49be400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b1710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 --2- 172.21.15.27:0/1110534258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- 172.21.15.27:0/1110534258 >> 172.21.15.27:0/1110534258 conn(0x5621d4979950 msgr2=0x5621d49ad7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- 172.21.15.27:0/1110534258 shutdown_connections 2024-04-25T01:29:19.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- 172.21.15.27:0/1110534258 wait complete. 2024-04-25T01:29:19.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- start start 2024-04-25T01:29:19.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 Processor -- start 2024-04-25T01:29:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b1290 0x5621d49b8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b2980 0x5621d49b9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5621d49beea0 con 0x5621d49bc010 2024-04-25T01:29:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5621d48d0e90 con 0x5621d49b1290 2024-04-25T01:29:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5621d49bf020 con 0x5621d49b2980 2024-04-25T01:29:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.499+0000 7fd88f433740 10 client.0 _ll_register_callbacks cb 0x5621d490c970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58388/0 (socket says 172.21.15.27:58388) 2024-04-25T01:29:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd88a3ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b2980 0x5621d49b9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 -- 172.21.15.27:0/2636018716 learned_addr learned my addr 172.21.15.27:0/2636018716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b1290 0x5621d49b8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 -- 172.21.15.27:0/2636018716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b2980 msgr2=0x5621d49b9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b2980 0x5621d49b9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 -- 172.21.15.27:0/2636018716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b1290 msgr2=0x5621d49b8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b1290 0x5621d49b8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 -- 172.21.15.27:0/2636018716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621d46f17e0 con 0x5621d49bc010 2024-04-25T01:29:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd889bee640 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b99c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd874012d40 tx=0x7fd87400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777212e366a5a307 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd86dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd86dffb640 1 -- 172.21.15.27:0/2636018716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd874012050 con 0x5621d49bc010 2024-04-25T01:29:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd86dffb640 1 -- 172.21.15.27:0/2636018716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd8740121f0 con 0x5621d49bc010 2024-04-25T01:29:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.500+0000 7fd88f433740 1 -- 172.21.15.27:0/2636018716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd880015070 con 0x5621d49bc010 2024-04-25T01:29:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd86dffb640 1 -- 172.21.15.27:0/2636018716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd874029050 con 0x5621d49bc010 2024-04-25T01:29:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88f433740 10 client.16521 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88f433740 1 -- 172.21.15.27:0/2636018716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5621d4751f70 con 0x5621d49bc010 2024-04-25T01:29:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd86dffb640 1 -- 172.21.15.27:0/2636018716 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd880015070 con 0x5621d49bc010 2024-04-25T01:29:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd86dffb640 1 -- 172.21.15.27:0/2636018716 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd874024030 con 0x5621d49bc010 2024-04-25T01:29:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88f433740 10 client.16521 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88f433740 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88f433740 20 client.16521 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88f433740 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5621d49cc190 0x5621d49ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88f433740 4 client.16521 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88f433740 1 -- 172.21.15.27:0/2636018716 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621d48e95a0 con 0x5621d49cc190 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.501+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5621d49cc190 0x5621d49ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.502+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5621d49cc190 0x5621d49ce580 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6464e60a26e34c1d server_cookie=5479f1c451f8cd8d in_seq=0 out_seq=0 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.502+0000 7fd86dffb640 10 client.16521 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.502+0000 7fd86dffb640 1 -- 172.21.15.27:0/2636018716 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621d48e95a0 con 0x5621d49cc190 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.502+0000 7fd86dffb640 10 client.16521 handle_command_reply: tid=0 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.502+0000 7fd88f433740 1 client.16521 shutdown 2024-04-25T01:29:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 -- 172.21.15.27:0/2636018716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 msgr2=0x5621d49b99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b99c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd874012d40 tx=0x7fd87400b390 comp rx=0 tx=0).stop 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 -- 172.21.15.27:0/2636018716 shutdown_connections 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5621d49cc190 0x5621d49ce580 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b99c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b2980 0x5621d49b9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd86dffb640 0 client.16521 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 --2- 172.21.15.27:0/2636018716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b1290 0x5621d49b8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 -- 172.21.15.27:0/2636018716 >> 172.21.15.27:0/2636018716 conn(0x5621d4979950 msgr2=0x5621d49b35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 -- 172.21.15.27:0/2636018716 shutdown_connections 2024-04-25T01:29:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 1 -- 172.21.15.27:0/2636018716 wait complete. 2024-04-25T01:29:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.503+0000 7fd88f433740 20 client.16521 trim_cache size 0 max 16384 2024-04-25T01:29:19.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88f433740 1 Processor -- start 2024-04-25T01:29:19.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88f433740 1 -- start start 2024-04-25T01:29:19.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88f433740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88f433740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88f433740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 0x5621d49ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88f433740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841b1380 con 0x5621d49b2980 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88f433740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841aec50 con 0x5621d49bc010 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88f433740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841acaa0 con 0x5621d49b1290 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58402/0 (socket says 172.21.15.27:58402) 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 -- 172.21.15.27:0/2417277684 learned_addr learned my addr 172.21.15.27:0/2417277684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 -- 172.21.15.27:0/2417277684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 msgr2=0x5621d49c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/2417277684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 -- 172.21.15.27:0/2417277684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 msgr2=0x5621d49ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/2417277684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 0x5621d49ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 -- 172.21.15.27:0/2417277684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621d45e9170 con 0x5621d49b2980 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/2417277684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b80c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd8800045b0 tx=0x7fd8800048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f300d1632965fb03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.521+0000 7fd86ffff640 1 -- 172.21.15.27:0/2417277684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880003dd0 con 0x5621d49b2980 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd86ffff640 1 -- 172.21.15.27:0/2417277684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd880015040 con 0x5621d49b2980 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd86ffff640 1 -- 172.21.15.27:0/2417277684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880011730 con 0x5621d49b2980 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 -- 172.21.15.27:0/2417277684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 msgr2=0x5621d49b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 --2- 172.21.15.27:0/2417277684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b80c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd8800045b0 tx=0x7fd8800048e0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 -- 172.21.15.27:0/2417277684 shutdown_connections 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 --2- 172.21.15.27:0/2417277684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49bc010 0x5621d49ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 --2- 172.21.15.27:0/2417277684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49b2980 0x5621d49b80c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 --2- 172.21.15.27:0/2417277684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 -- 172.21.15.27:0/2417277684 >> 172.21.15.27:0/2417277684 conn(0x5621d4979950 msgr2=0x5621d49ac0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 -- 172.21.15.27:0/2417277684 shutdown_connections 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 -- 172.21.15.27:0/2417277684 wait complete. 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.522+0000 7fd88f433740 1 -- start start 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88f433740 1 Processor -- start 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88f433740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88f433740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b2980 0x5621d49b60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88f433740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88f433740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841ad110 con 0x5621d49bc010 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88f433740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841ae580 con 0x5621d49b2980 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88f433740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841ad7e0 con 0x5621d49b1290 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88f433740 10 client.0 _ll_register_callbacks cb 0x7fd874012950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58408/0 (socket says 172.21.15.27:58408) 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 -- 172.21.15.27:0/3645633738 learned_addr learned my addr 172.21.15.27:0/3645633738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b2980 0x5621d49b60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 -- 172.21.15.27:0/3645633738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 msgr2=0x5621d49b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 -- 172.21.15.27:0/3645633738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b2980 msgr2=0x5621d49b60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b2980 0x5621d49b60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 -- 172.21.15.27:0/3645633738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621d4a316f0 con 0x5621d49bc010 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd889bee640 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b6630 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd874012530 tx=0x7fd87400b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c8bf8dee0dc79e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.523+0000 7fd86dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd86dffb640 1 -- 172.21.15.27:0/3645633738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8740035f0 con 0x5621d49bc010 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd86dffb640 1 -- 172.21.15.27:0/3645633738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd874003790 con 0x5621d49bc010 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 1 -- 172.21.15.27:0/3645633738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd880015040 con 0x5621d49bc010 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd86dffb640 1 -- 172.21.15.27:0/3645633738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd874005b90 con 0x5621d49bc010 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 10 client.16527 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 1 -- 172.21.15.27:0/3645633738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5621d4a32a10 con 0x5621d49bc010 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd86dffb640 1 -- 172.21.15.27:0/3645633738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd880015040 con 0x5621d49bc010 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd86dffb640 1 -- 172.21.15.27:0/3645633738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd874024040 con 0x5621d49bc010 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 10 client.16527 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 20 client.16527 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5621d4a33180 0x5621d4a355a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 4 client.16527 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88f433740 1 -- 172.21.15.27:0/3645633738 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5621d4922800 con 0x5621d4a33180 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.524+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5621d4a33180 0x5621d4a355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.525+0000 7fd88a3ef640 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5621d4a33180 0x5621d4a355a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1b487ed9fc18cce server_cookie=bfc1c48395952c0b in_seq=0 out_seq=0 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.525+0000 7fd86dffb640 10 client.16527 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd86dffb640 1 -- 172.21.15.27:0/3645633738 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5621d4922800 con 0x5621d4a33180 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd86dffb640 10 client.16527 handle_command_reply: tid=0 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 client.16527 shutdown 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 -- 172.21.15.27:0/3645633738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 msgr2=0x5621d49b6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b6630 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd874012530 tx=0x7fd87400b570 comp rx=0 tx=0).stop 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 -- 172.21.15.27:0/3645633738 shutdown_connections 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5621d4a33180 0x5621d4a355a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5621d49bc010 0x5621d49b6630 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5621d49b2980 0x5621d49b60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 --2- 172.21.15.27:0/3645633738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5621d49b1290 0x5621d49b5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 -- 172.21.15.27:0/3645633738 >> 172.21.15.27:0/3645633738 conn(0x5621d4979950 msgr2=0x5621d49ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd86dffb640 0 client.16527 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 -- 172.21.15.27:0/3645633738 shutdown_connections 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 1 -- 172.21.15.27:0/3645633738 wait complete. 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.526+0000 7fd88f433740 20 client.16527 trim_cache size 0 max 16384 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:29:19.516 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:29:19.517 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "fd03e8fd-66e3-489c-a4cd-0b2337b42f1e", 2024-04-25T01:29:19.517 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:29:19.517 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:29:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 -- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd8700a2d00 msgr2=0x7fd8700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd8700a2d00 0x7fd8700a51c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0b9540 tx=0x7fd87c0a6040 comp rx=0 tx=0).stop 2024-04-25T01:29:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 -- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd884107730 msgr2=0x7fd88419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd884107730 0x7fd88419c820 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd87c005cc0 tx=0x7fd87c003970 comp rx=0 tx=0).stop 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 -- 172.21.15.27:0/3560954275 shutdown_connections 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd8700a2d00 0x7fd8700a51c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd884107730 0x7fd88419c820 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88406a780 0x7fd8841984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 --2- 172.21.15.27:0/3560954275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd884069dc0 0x7fd884197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 -- 172.21.15.27:0/3560954275 >> 172.21.15.27:0/3560954275 conn(0x7fd8840776a0 msgr2=0x7fd8840fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 -- 172.21.15.27:0/3560954275 shutdown_connections 2024-04-25T01:29:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.527+0000 7fd88c653640 1 -- 172.21.15.27:0/3560954275 wait complete. 2024-04-25T01:29:19.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:29:19.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 -- 172.21.15.27:0/422059117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24100fe0 msgr2=0x7f6e2410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 --2- 172.21.15.27:0/422059117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24100fe0 0x7f6e2410d4f0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009930 tx=0x7f6e1402a420 comp rx=0 tx=0).stop 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 -- 172.21.15.27:0/422059117 shutdown_connections 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 --2- 172.21.15.27:0/422059117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24100fe0 0x7f6e2410d4f0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 --2- 172.21.15.27:0/422059117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e24100620 0x7f6e24100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 --2- 172.21.15.27:0/422059117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e24106630 0x7f6e24106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 -- 172.21.15.27:0/422059117 >> 172.21.15.27:0/422059117 conn(0x7f6e240776a0 msgr2=0x7f6e24077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 -- 172.21.15.27:0/422059117 shutdown_connections 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 -- 172.21.15.27:0/422059117 wait complete. 2024-04-25T01:29:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 Processor -- start 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.646+0000 7f6e2969a640 1 -- start start 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e2969a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e24100620 0x7f6e24198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e2969a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e24100fe0 0x7f6e24199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e2969a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24106630 0x7f6e241a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e2969a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e24110a00 con 0x7f6e24106630 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e2969a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6e24110880 con 0x7f6e24100620 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e2969a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e24110b80 con 0x7f6e24100fe0 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24106630 0x7f6e241a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24106630 0x7f6e241a3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58428/0 (socket says 172.21.15.27:58428) 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 -- 172.21.15.27:0/1186493934 learned_addr learned my addr 172.21.15.27:0/1186493934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e24100620 0x7f6e24198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 -- 172.21.15.27:0/1186493934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e24100fe0 msgr2=0x7f6e24199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e24100fe0 0x7f6e24199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 -- 172.21.15.27:0/1186493934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e24100620 msgr2=0x7f6e24198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e24100620 0x7f6e24198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 -- 172.21.15.27:0/1186493934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e241a43a0 con 0x7f6e24106630 2024-04-25T01:29:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e237fe640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24106630 0x7f6e241a3e60 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6e14005ab0 tx=0x7f6e140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e0bfff640 1 -- 172.21.15.27:0/1186493934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1403f030 con 0x7f6e24106630 2024-04-25T01:29:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e2969a640 1 -- 172.21.15.27:0/1186493934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e241a4570 con 0x7f6e24106630 2024-04-25T01:29:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e0bfff640 1 -- 172.21.15.27:0/1186493934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e14003ce0 con 0x7f6e24106630 2024-04-25T01:29:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.647+0000 7f6e0bfff640 1 -- 172.21.15.27:0/1186493934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14035070 con 0x7f6e24106630 2024-04-25T01:29:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.648+0000 7f6e2969a640 1 -- 172.21.15.27:0/1186493934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e241a49e0 con 0x7f6e24106630 2024-04-25T01:29:19.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e0bfff640 1 -- 172.21.15.27:0/1186493934 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6e14042030 con 0x7f6e24106630 2024-04-25T01:29:19.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e2c3a2740 1 Processor -- start 2024-04-25T01:29:19.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e2c3a2740 1 -- start start 2024-04-25T01:29:19.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 0x559238db7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 0x559238dc64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559238d1c6b0 con 0x559238dc6a30 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559238cc4880 con 0x559238db7530 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559238d18670 con 0x559238db7e50 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e227fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e227fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58434/0 (socket says 172.21.15.27:58434) 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e227fc640 1 -- 172.21.15.27:0/259215263 learned_addr learned my addr 172.21.15.27:0/259215263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.650+0000 7f6e0bfff640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6e0c0a2d00 0x7f6e0c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 0x559238dc64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e227fc640 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 0x559238db7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e0bfff640 1 -- 172.21.15.27:0/1186493934 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6e140f0040 con 0x7f6e24106630 2024-04-25T01:29:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e237fe640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6e0c0a2d00 0x7f6e0c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e227fc640 1 -- 172.21.15.27:0/259215263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 msgr2=0x559238dc64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e227fc640 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 0x559238dc64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e227fc640 1 -- 172.21.15.27:0/259215263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 msgr2=0x559238db7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e227fc640 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 0x559238db7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e227fc640 1 -- 172.21.15.27:0/259215263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559238afa7d0 con 0x559238dc6a30 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e227fc640 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc8e20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009f30 tx=0x7f6e180097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486fa32e10c70632 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e237fe640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6e0c0a2d00 0x7f6e0c0a51c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6e14006010 tx=0x7f6e14009260 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e097fa640 1 -- 172.21.15.27:0/259215263 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1800f7e0 con 0x559238dc6a30 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e097fa640 1 -- 172.21.15.27:0/259215263 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e18012450 con 0x559238dc6a30 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.651+0000 7f6e097fa640 1 -- 172.21.15.27:0/259215263 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1800fe60 con 0x559238dc6a30 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/259215263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 msgr2=0x559238dc8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc8e20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009f30 tx=0x7f6e180097f0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/259215263 shutdown_connections 2024-04-25T01:29:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc8e20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 0x559238dc64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/259215263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 0x559238db7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/259215263 >> 172.21.15.27:0/259215263 conn(0x559238db3950 msgr2=0x559238db3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/259215263 shutdown_connections 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/259215263 wait complete. 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 -- start start 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.652+0000 7f6e2c3a2740 1 Processor -- start 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 0x559238dc39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7e50 0x559238dc3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559238dc97f0 con 0x559238dc6a30 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559238cd7f50 con 0x559238db7e50 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559238dc9970 con 0x559238db7530 2024-04-25T01:29:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e2c3a2740 10 client.0 _ll_register_callbacks cb 0x559238d13000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58446/0 (socket says 172.21.15.27:58446) 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 -- 172.21.15.27:0/3781572800 learned_addr learned my addr 172.21.15.27:0/3781572800 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 0x559238dc39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7e50 0x559238dc3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 -- 172.21.15.27:0/3781572800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 msgr2=0x559238dc39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 0x559238dc39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 -- 172.21.15.27:0/3781572800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7e50 msgr2=0x559238dc3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7e50 0x559238dc3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 -- 172.21.15.27:0/3781572800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559238a303f0 con 0x559238dc6a30 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e227fc640 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc4450 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6e18014fd0 tx=0x7f6e18016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd14ea7c1c0cf0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e06ff5640 1 -- 172.21.15.27:0/3781572800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e180141e0 con 0x559238dc6a30 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e06ff5640 1 -- 172.21.15.27:0/3781572800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6e18014380 con 0x559238dc6a30 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.653+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/3781572800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e18012450 con 0x559238dc6a30 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e06ff5640 1 -- 172.21.15.27:0/3781572800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6e18004b20 con 0x559238dc6a30 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e2c3a2740 10 client.16539 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/3781572800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5592389f00f0 con 0x559238dc6a30 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e06ff5640 1 -- 172.21.15.27:0/3781572800 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e18012450 con 0x559238dc6a30 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e06ff5640 1 -- 172.21.15.27:0/3781572800 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6e1800f870 con 0x559238dc6a30 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e2c3a2740 10 client.16539 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e2c3a2740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e2c3a2740 20 client.16539 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559238dd6c10 0x559238dd9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e2c3a2740 4 client.16539 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/3781572800 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559238d29bf0 con 0x559238dd6c10 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.654+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559238dd6c10 0x559238dd9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.655+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559238dd6c10 0x559238dd9000 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30c1dbafcb754630 server_cookie=496774a528997896 in_seq=0 out_seq=0 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.655+0000 7f6e06ff5640 10 client.16539 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.655+0000 7f6e06ff5640 1 -- 172.21.15.27:0/3781572800 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559238d29bf0 con 0x559238dd6c10 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.655+0000 7f6e06ff5640 10 client.16539 handle_command_reply: tid=0 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.655+0000 7f6e2c3a2740 1 client.16539 shutdown 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/3781572800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 msgr2=0x559238dc4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc4450 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6e18014fd0 tx=0x7f6e18016040 comp rx=0 tx=0).stop 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/3781572800 shutdown_connections 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559238dd6c10 0x559238dd9000 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dc4450 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7e50 0x559238dc3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e06ff5640 0 client.16539 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/3781572800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 0x559238dc39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/3781572800 >> 172.21.15.27:0/3781572800 conn(0x559238db3950 msgr2=0x559238db9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/3781572800 shutdown_connections 2024-04-25T01:29:19.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/3781572800 wait complete. 2024-04-25T01:29:19.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.656+0000 7f6e2c3a2740 20 client.16539 trim_cache size 0 max 16384 2024-04-25T01:29:19.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.673+0000 7f6e2c3a2740 1 Processor -- start 2024-04-25T01:29:19.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.673+0000 7f6e2c3a2740 1 -- start start 2024-04-25T01:29:19.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 0x559238dc6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238db7e50 0x559238dd26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238dc6a30 0x559238dbecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e241af200 con 0x559238db7e50 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6e241acad0 con 0x559238dc6a30 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e241aa920 con 0x559238db7530 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238db7e50 0x559238dd26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238db7e50 0x559238dd26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58450/0 (socket says 172.21.15.27:58450) 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 -- 172.21.15.27:0/4038794151 learned_addr learned my addr 172.21.15.27:0/4038794151 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e227fc640 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 0x559238dc6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 -- 172.21.15.27:0/4038794151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 msgr2=0x559238dc6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 0x559238dc6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 -- 172.21.15.27:0/4038794151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238dc6a30 msgr2=0x559238dbecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e227fc640 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238dc6a30 0x559238dbecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238dc6a30 0x559238dbecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 -- 172.21.15.27:0/4038794151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559238db7c80 con 0x559238db7e50 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238db7e50 0x559238dd26f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c005920 tx=0x7f6e1c00a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395b60c1d6edfc36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e08ff9640 1 -- 172.21.15.27:0/4038794151 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c0202f0 con 0x559238db7e50 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e08ff9640 1 -- 172.21.15.27:0/4038794151 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e1c024070 con 0x559238db7e50 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.674+0000 7f6e08ff9640 1 -- 172.21.15.27:0/4038794151 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c01ed70 con 0x559238db7e50 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/4038794151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238db7e50 msgr2=0x559238dd26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238db7e50 0x559238dd26f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c005920 tx=0x7f6e1c00a2d0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/4038794151 shutdown_connections 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238dc6a30 0x559238dbecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238db7e50 0x559238dd26f0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/4038794151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7530 0x559238dc6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/4038794151 >> 172.21.15.27:0/4038794151 conn(0x559238db3950 msgr2=0x559238dc3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/4038794151 shutdown_connections 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/4038794151 wait complete. 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 -- start start 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 Processor -- start 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 0x559238dcee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 0x559238dca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.675+0000 7f6e2c3a2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dca720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6e241aaf90 con 0x559238dc6a30 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6e241ac400 con 0x559238db7530 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e2c3a2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e241ab660 con 0x559238db7e50 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e2c3a2740 10 client.0 _ll_register_callbacks cb 0x7f6e1c00a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dca720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58462/0 (socket says 172.21.15.27:58462) 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 -- 172.21.15.27:0/1505231639 learned_addr learned my addr 172.21.15.27:0/1505231639 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 0x559238dca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 0x559238dcee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 -- 172.21.15.27:0/1505231639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 msgr2=0x559238dca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 0x559238dca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 -- 172.21.15.27:0/1505231639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 msgr2=0x559238dcee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 0x559238dcee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 -- 172.21.15.27:0/1505231639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559238dcadc0 con 0x559238dc6a30 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e227fc640 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dca720 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009710 tx=0x7f6e18016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751229cea2a68185 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e06ff5640 1 -- 172.21.15.27:0/1505231639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1800fe80 con 0x559238dc6a30 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e06ff5640 1 -- 172.21.15.27:0/1505231639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6e180143b0 con 0x559238dc6a30 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/1505231639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e1c024070 con 0x559238dc6a30 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e06ff5640 1 -- 172.21.15.27:0/1505231639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6e180042d0 con 0x559238dc6a30 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.676+0000 7f6e2c3a2740 10 client.16545 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/1505231639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559238e3cef0 con 0x559238dc6a30 2024-04-25T01:29:19.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e06ff5640 1 -- 172.21.15.27:0/1505231639 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e1c024070 con 0x559238dc6a30 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e06ff5640 1 -- 172.21.15.27:0/1505231639 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6e18003f40 con 0x559238dc6a30 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e2c3a2740 10 client.16545 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e2c3a2740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e2c3a2740 20 client.16545 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559238e3d4f0 0x559238e3f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e2c3a2740 4 client.16545 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/1505231639 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559238d23f40 con 0x559238e3d4f0 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559238e3d4f0 0x559238e3f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e06ff5640 10 client.16545 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.677+0000 7f6e22ffd640 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559238e3d4f0 0x559238e3f8e0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84eab5d006e45202 server_cookie=35ea021354b54ebd in_seq=0 out_seq=0 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e06ff5640 1 -- 172.21.15.27:0/1505231639 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559238d23f40 con 0x559238e3d4f0 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e06ff5640 10 client.16545 handle_command_reply: tid=0 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 client.16545 shutdown 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/1505231639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 msgr2=0x559238dca720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dca720 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009710 tx=0x7f6e18016040 comp rx=0 tx=0).stop 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/1505231639 shutdown_connections 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x559238e3d4f0 0x559238e3f8e0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559238dc6a30 0x559238dca720 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e06ff5640 0 client.16545 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559238db7e50 0x559238dca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 --2- 172.21.15.27:0/1505231639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559238db7530 0x559238dcee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/1505231639 >> 172.21.15.27:0/1505231639 conn(0x559238db3950 msgr2=0x559238dc4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/1505231639 shutdown_connections 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 1 -- 172.21.15.27:0/1505231639 wait complete. 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2c3a2740 20 client.16545 trim_cache size 0 max 16384 2024-04-25T01:29:19.663 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: "fd03e8fd-66e3-489c-a4cd-0b2337b42f1e": { 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "fd03e8fd-66e3-489c-a4cd-0b2337b42f1e", 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:29:19.664 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:29:19.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 -- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6e0c0a2d00 msgr2=0x7f6e0c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6e0c0a2d00 0x7f6e0c0a51c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6e14006010 tx=0x7f6e14009260 comp rx=0 tx=0).stop 2024-04-25T01:29:19.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 -- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24106630 msgr2=0x7f6e241a3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:19.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24106630 0x7f6e241a3e60 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6e14005ab0 tx=0x7f6e140042f0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 -- 172.21.15.27:0/1186493934 shutdown_connections 2024-04-25T01:29:19.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6e0c0a2d00 0x7f6e0c0a51c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e24106630 0x7f6e241a3e60 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e24100fe0 0x7f6e24199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 --2- 172.21.15.27:0/1186493934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6e24100620 0x7f6e24198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:19.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.678+0000 7f6e2969a640 1 -- 172.21.15.27:0/1186493934 >> 172.21.15.27:0/1186493934 conn(0x7f6e240776a0 msgr2=0x7f6e24103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:19.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.679+0000 7f6e2969a640 1 -- 172.21.15.27:0/1186493934 shutdown_connections 2024-04-25T01:29:19.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:19.679+0000 7f6e2969a640 1 -- 172.21.15.27:0/1186493934 wait complete. 2024-04-25T01:29:19.668 INFO:tasks.cephfs.filesystem:scrub status for tag:fd03e8fd-66e3-489c-a4cd-0b2337b42f1e - {'path': '/', 'tag': 'fd03e8fd-66e3-489c-a4cd-0b2337b42f1e', 'options': 'recursive,force'} 2024-04-25T01:29:20.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:20 smithi153 ceph-mon[32621]: pgmap v120: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:20.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:20 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:29:20.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:20 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:20.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:20 smithi139 ceph-mon[32760]: pgmap v120: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:20.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:20 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:29:20.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:20 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:20.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:20 smithi027 ceph-mon[29197]: pgmap v120: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:20.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:20 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:29:20.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:20 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:21.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:21 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:29:21.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:21 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:21.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:21 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:29:21.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:21 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:21.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:21 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:29:21.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:21 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:21.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:21 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:29:21.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:21 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:21.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:21 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:29:21.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:21 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:21.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:21 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:29:21.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:21 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:22.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:22 smithi153 ceph-mon[32621]: pgmap v121: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:22.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:22 smithi139 ceph-mon[32760]: pgmap v121: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:22.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:22 smithi027 ceph-mon[29197]: pgmap v121: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:24.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:24 smithi153 ceph-mon[32621]: pgmap v122: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:29:24.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:24 smithi139 ceph-mon[32760]: pgmap v122: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:29:24.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:24 smithi027 ceph-mon[29197]: pgmap v122: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:29:26.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:26 smithi153 ceph-mon[32621]: pgmap v123: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:29:26.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:26 smithi139 ceph-mon[32760]: pgmap v123: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:29:26.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:26 smithi027 ceph-mon[29197]: pgmap v123: 129 pgs: 129 active+clean; 2.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:29:28.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:28 smithi153 ceph-mon[32621]: pgmap v124: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:28.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:28 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:29:28.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:28 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:29:28.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:28 smithi139 ceph-mon[32760]: pgmap v124: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:28.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:28 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:29:28.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:28 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:29:28.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:28 smithi027 ceph-mon[29197]: pgmap v124: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:28.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:28 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:29:28.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:28 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:29:30.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:30 smithi153 ceph-mon[32621]: pgmap v125: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:29:30.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:30 smithi139 ceph-mon[32760]: pgmap v125: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:29:30.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:30 smithi027 ceph-mon[29197]: pgmap v125: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:29:32.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:32 smithi153 ceph-mon[32621]: pgmap v126: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:32.542 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:32 smithi139 ceph-mon[32760]: pgmap v126: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:32.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:32 smithi027 ceph-mon[29197]: pgmap v126: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:33.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:33 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:33.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:33 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:33 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:34.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:34 smithi153 ceph-mon[32621]: pgmap v127: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:34.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:34 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:34.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:34 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:34.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:34 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:29:34.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:34 smithi139 ceph-mon[32760]: pgmap v127: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:34.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:34 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:34.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:34 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:34.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:34 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:29:34.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:34 smithi027 ceph-mon[29197]: pgmap v127: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:34.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:34 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:34.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:34 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:34.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:34 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:29:36.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:36 smithi153 ceph-mon[32621]: pgmap v128: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:36.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:36 smithi139 ceph-mon[32760]: pgmap v128: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:36.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:36 smithi027 ceph-mon[29197]: pgmap v128: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:38.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:38 smithi153 ceph-mon[32621]: pgmap v129: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:29:38.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:38 smithi139 ceph-mon[32760]: pgmap v129: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:29:38.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:38 smithi027 ceph-mon[29197]: pgmap v129: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:29:40.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:40 smithi153 ceph-mon[32621]: pgmap v130: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.3 KiB/s wr, 9 op/s 2024-04-25T01:29:40.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:40 smithi139 ceph-mon[32760]: pgmap v130: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.3 KiB/s wr, 9 op/s 2024-04-25T01:29:40.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:40 smithi027 ceph-mon[29197]: pgmap v130: 129 pgs: 129 active+clean; 2.3 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.3 KiB/s wr, 9 op/s 2024-04-25T01:29:42.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:42 smithi153 ceph-mon[32621]: pgmap v131: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:29:42.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:42 smithi139 ceph-mon[32760]: pgmap v131: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:29:42.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:42 smithi027 ceph-mon[29197]: pgmap v131: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:29:44.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:44 smithi153 ceph-mon[32621]: pgmap v132: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:44.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:44 smithi139 ceph-mon[32760]: pgmap v132: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:44.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:44 smithi027 ceph-mon[29197]: pgmap v132: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:46.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:46 smithi139 ceph-mon[32760]: pgmap v133: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:46.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:46 smithi027 ceph-mon[29197]: pgmap v133: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:46.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:46 smithi153 ceph-mon[32621]: pgmap v133: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:29:48.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:48 smithi139 ceph-mon[32760]: pgmap v134: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 14 op/s 2024-04-25T01:29:48.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:48 smithi027 ceph-mon[29197]: pgmap v134: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 14 op/s 2024-04-25T01:29:48.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:48 smithi153 ceph-mon[32621]: pgmap v134: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 14 op/s 2024-04-25T01:29:49.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:29:49.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:29:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 -- 172.21.15.27:0/903174230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 msgr2=0x7f5f7c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 --2- 172.21.15.27:0/903174230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 0x7f5f7c10b7c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5f70009900 tx=0x7f5f7002a510 comp rx=0 tx=0).stop 2024-04-25T01:29:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 -- 172.21.15.27:0/903174230 shutdown_connections 2024-04-25T01:29:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 --2- 172.21.15.27:0/903174230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f7c10bd00 0x7f5f7c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 --2- 172.21.15.27:0/903174230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 0x7f5f7c10b7c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 --2- 172.21.15.27:0/903174230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f7c1029c0 0x7f5f7c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 -- 172.21.15.27:0/903174230 >> 172.21.15.27:0/903174230 conn(0x7f5f7c077740 msgr2=0x7f5f7c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 -- 172.21.15.27:0/903174230 shutdown_connections 2024-04-25T01:29:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 -- 172.21.15.27:0/903174230 wait complete. 2024-04-25T01:29:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 Processor -- start 2024-04-25T01:29:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.807+0000 7f5f825a9640 1 -- start start 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f825a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f7c1029c0 0x7f5f7c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f825a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 0x7f5f7c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f825a9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f7c10bd00 0x7f5f7c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f825a9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c111b00 con 0x7f5f7c1032e0 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f825a9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c111980 con 0x7f5f7c10bd00 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f825a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c111c80 con 0x7f5f7c1029c0 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 0x7f5f7c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 0x7f5f7c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56146/0 (socket says 172.21.15.27:56146) 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7b7fe640 1 -- 172.21.15.27:0/1124785615 learned_addr learned my addr 172.21.15.27:0/1124785615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f80b46640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f7c10bd00 0x7f5f7c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f7c1029c0 0x7f5f7c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7b7fe640 1 -- 172.21.15.27:0/1124785615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f7c1029c0 msgr2=0x7f5f7c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f7c1029c0 0x7f5f7c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7b7fe640 1 -- 172.21.15.27:0/1124785615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f7c10bd00 msgr2=0x7f5f7c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f7c10bd00 0x7f5f7c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.808+0000 7f5f7b7fe640 1 -- 172.21.15.27:0/1124785615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c19cff0 con 0x7f5f7c1032e0 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.809+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 0x7f5f7c198540 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006010 tx=0x7f5f700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.809+0000 7f5f797fa640 1 -- 172.21.15.27:0/1124785615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7003f030 con 0x7f5f7c1032e0 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.809+0000 7f5f797fa640 1 -- 172.21.15.27:0/1124785615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f70003ce0 con 0x7f5f7c1032e0 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.809+0000 7f5f825a9640 1 -- 172.21.15.27:0/1124785615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7c19d280 con 0x7f5f7c1032e0 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.809+0000 7f5f797fa640 1 -- 172.21.15.27:0/1124785615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70035070 con 0x7f5f7c1032e0 2024-04-25T01:29:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.809+0000 7f5f825a9640 1 -- 172.21.15.27:0/1124785615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f7c1a4b70 con 0x7f5f7c1032e0 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.811+0000 7f5f797fa640 1 -- 172.21.15.27:0/1124785615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5f70042030 con 0x7f5f7c1032e0 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.811+0000 7f5f85396740 1 Processor -- start 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.811+0000 7f5f85396740 1 -- start start 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.811+0000 7f5f85396740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e0da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f85396740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e180e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f85396740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 0x560179e1aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f85396740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560179d726b0 con 0x560179e0d630 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f85396740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560179d1a880 con 0x560179e0df50 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f85396740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560179d6e670 con 0x560179e18620 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e0da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e0da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56156/0 (socket says 172.21.15.27:56156) 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 -- 172.21.15.27:0/2263695782 learned_addr learned my addr 172.21.15.27:0/2263695782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:49.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f797fa640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f640a2d00 0x7f5f640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 0x560179e1aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e180e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f640a2d00 0x7f5f640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f797fa640 1 -- 172.21.15.27:0/1124785615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5f700efc70 con 0x7f5f7c1032e0 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 -- 172.21.15.27:0/2263695782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 msgr2=0x560179e1aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 0x560179e1aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 -- 172.21.15.27:0/2263695782 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 msgr2=0x560179e180e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e180e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 -- 172.21.15.27:0/2263695782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560179b507d0 con 0x560179e0d630 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.812+0000 7f5f80b46640 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e0da10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5f74014a60 tx=0x7f5f74014f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155b0e5644a8c010 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f627fc640 1 -- 172.21.15.27:0/2263695782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7401ec50 con 0x560179e0d630 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f627fc640 1 -- 172.21.15.27:0/2263695782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f74023070 con 0x560179e0d630 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f627fc640 1 -- 172.21.15.27:0/2263695782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7401de00 con 0x560179e0d630 2024-04-25T01:29:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 -- 172.21.15.27:0/2263695782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 msgr2=0x560179e0da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e0da10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5f74014a60 tx=0x7f5f74014f20 comp rx=0 tx=0).stop 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f80b46640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f640a2d00 0x7f5f640a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5f74022000 tx=0x7f5f7401e000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 -- 172.21.15.27:0/2263695782 shutdown_connections 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 0x560179e1aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e180e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 --2- 172.21.15.27:0/2263695782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e0da10 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 -- 172.21.15.27:0/2263695782 >> 172.21.15.27:0/2263695782 conn(0x560179e09950 msgr2=0x560179e09d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 -- 172.21.15.27:0/2263695782 shutdown_connections 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.813+0000 7f5f85396740 1 -- 172.21.15.27:0/2263695782 wait complete. 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 1 -- start start 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 1 Processor -- start 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e155c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e15b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 0x560179e16040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560179e1b410 con 0x560179e0d630 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560179d2df50 con 0x560179e0df50 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560179e1b590 con 0x560179e18620 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f85396740 10 client.0 _ll_register_callbacks cb 0x560179d69000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e155c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e155c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56160/0 (socket says 172.21.15.27:56160) 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f7bfff640 1 -- 172.21.15.27:0/281692276 learned_addr learned my addr 172.21.15.27:0/281692276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f80b46640 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 0x560179e16040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.814+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e15b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f7bfff640 1 -- 172.21.15.27:0/281692276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 msgr2=0x560179e16040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 0x560179e16040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f7bfff640 1 -- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 msgr2=0x560179e15b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e15b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f7bfff640 1 -- 172.21.15.27:0/281692276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560179a863f0 con 0x560179e0d630 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e15b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e155c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00bad0 tx=0x7f5f6c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108811c5686d8633 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/281692276 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c003e00 con 0x560179e0d630 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/281692276 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5f6c005020 con 0x560179e0d630 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f85396740 1 -- 172.21.15.27:0/281692276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f74023070 con 0x560179e0d630 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/281692276 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5f6c005220 con 0x560179e0d630 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f85396740 10 client.16557 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f85396740 1 -- 172.21.15.27:0/281692276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560179a460f0 con 0x560179e0d630 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/281692276 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f74023070 con 0x560179e0d630 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.815+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/281692276 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5f6c01b630 con 0x560179e0d630 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f85396740 10 client.16557 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:49.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f85396740 10 client.16557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f85396740 20 client.16557 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f85396740 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560179e28960 0x560179e2add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f85396740 4 client.16557 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f85396740 1 -- 172.21.15.27:0/281692276 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560179d7fbf0 con 0x560179e28960 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560179e28960 0x560179e2add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560179e28960 0x560179e2add0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b5207f95fcb584c server_cookie=4225e84a84d48f7c in_seq=0 out_seq=0 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.816+0000 7f5f5b7fe640 10 client.16557 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/281692276 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560179d7fbf0 con 0x560179e28960 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f5b7fe640 10 client.16557 handle_command_reply: tid=0 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 client.16557 shutdown 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 -- 172.21.15.27:0/281692276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 msgr2=0x560179e155c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e155c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00bad0 tx=0x7f5f6c00bf90 comp rx=0 tx=0).stop 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 -- 172.21.15.27:0/281692276 shutdown_connections 2024-04-25T01:29:49.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560179e28960 0x560179e2add0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e18620 0x560179e16040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0df50 0x560179e15b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f5b7fe640 0 client.16557 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 --2- 172.21.15.27:0/281692276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e0d630 0x560179e155c0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 -- 172.21.15.27:0/281692276 >> 172.21.15.27:0/281692276 conn(0x560179e09950 msgr2=0x560179e0f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 -- 172.21.15.27:0/281692276 shutdown_connections 2024-04-25T01:29:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 1 -- 172.21.15.27:0/281692276 wait complete. 2024-04-25T01:29:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.817+0000 7f5f85396740 20 client.16557 trim_cache size 0 max 16384 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.835+0000 7f5f85396740 1 Processor -- start 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.835+0000 7f5f85396740 1 -- start start 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.835+0000 7f5f85396740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 0x560179e242d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.835+0000 7f5f85396740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 0x560179e08970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.835+0000 7f5f85396740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e306a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.835+0000 7f5f85396740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1b1560 con 0x560179e18620 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.835+0000 7f5f85396740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1aee30 con 0x560179e0d630 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.835+0000 7f5f85396740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1acc80 con 0x560179e0df50 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e306a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e306a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56172/0 (socket says 172.21.15.27:56172) 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 -- 172.21.15.27:0/3125923556 learned_addr learned my addr 172.21.15.27:0/3125923556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/3125923556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 0x560179e08970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 -- 172.21.15.27:0/3125923556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 msgr2=0x560179e08970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 --2- 172.21.15.27:0/3125923556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 0x560179e08970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 -- 172.21.15.27:0/3125923556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 msgr2=0x560179e242d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 --2- 172.21.15.27:0/3125923556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 0x560179e242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 -- 172.21.15.27:0/3125923556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560179e0cc40 con 0x560179e18620 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f80b46640 1 --2- 172.21.15.27:0/3125923556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e306a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5f74014720 tx=0x7f5f74000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338d7a25cc67792a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f61ffb640 1 -- 172.21.15.27:0/3125923556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7401ec50 con 0x560179e18620 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f61ffb640 1 -- 172.21.15.27:0/3125923556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f74036070 con 0x560179e18620 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.836+0000 7f5f61ffb640 1 -- 172.21.15.27:0/3125923556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7401ddd0 con 0x560179e18620 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 -- 172.21.15.27:0/3125923556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 msgr2=0x560179e306a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 --2- 172.21.15.27:0/3125923556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e306a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5f74014720 tx=0x7f5f74000f40 comp rx=0 tx=0).stop 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 -- 172.21.15.27:0/3125923556 shutdown_connections 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 --2- 172.21.15.27:0/3125923556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e306a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 --2- 172.21.15.27:0/3125923556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 0x560179e08970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 --2- 172.21.15.27:0/3125923556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 0x560179e242d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 -- 172.21.15.27:0/3125923556 >> 172.21.15.27:0/3125923556 conn(0x560179e09950 msgr2=0x560179e08260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 -- 172.21.15.27:0/3125923556 shutdown_connections 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 -- 172.21.15.27:0/3125923556 wait complete. 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 -- start start 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.837+0000 7f5f85396740 1 Processor -- start 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f85396740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 0x560179e12490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f85396740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 0x560179e129d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f85396740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e12f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f85396740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1ad2f0 con 0x560179e18620 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f85396740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1ae760 con 0x560179e0d630 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f85396740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1ad9c0 con 0x560179e0df50 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f85396740 10 client.0 _ll_register_callbacks cb 0x7f5f6c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e12f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e12f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56182/0 (socket says 172.21.15.27:56182) 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 -- 172.21.15.27:0/3936343808 learned_addr learned my addr 172.21.15.27:0/3936343808 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f7b7fe640 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 0x560179e129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 0x560179e12490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 -- 172.21.15.27:0/3936343808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 msgr2=0x560179e129d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 0x560179e129d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 -- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 msgr2=0x560179e12490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 0x560179e12490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 -- 172.21.15.27:0/3936343808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560179e8e110 con 0x560179e18620 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 0x560179e12490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f80b46640 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e12f10 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5f74014f60 tx=0x7f5f74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc059e85d9fbbb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/3936343808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7403e070 con 0x560179e18620 2024-04-25T01:29:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.838+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/3936343808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5f7401ec50 con 0x560179e18620 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 1 -- 172.21.15.27:0/3936343808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f74036070 con 0x560179e18620 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/3936343808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5f7401db50 con 0x560179e18620 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 10 client.16563 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 1 -- 172.21.15.27:0/3936343808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560179e8f400 con 0x560179e18620 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/3936343808 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f74036070 con 0x560179e18620 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/3936343808 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5f7400f040 con 0x560179e18620 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 10 client.16563 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 20 client.16563 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560179e8f8f0 0x560179e91d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 4 client.16563 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f85396740 1 -- 172.21.15.27:0/3936343808 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560179d79f40 con 0x560179e8f8f0 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.839+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560179e8f8f0 0x560179e91d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f5b7fe640 10 client.16563 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f7bfff640 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560179e8f8f0 0x560179e91d10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b04fdf015636fcfa server_cookie=bae02891b9913d15 in_seq=0 out_seq=0 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f5b7fe640 1 -- 172.21.15.27:0/3936343808 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560179d79f40 con 0x560179e8f8f0 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f5b7fe640 10 client.16563 handle_command_reply: tid=0 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 client.16563 shutdown 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 -- 172.21.15.27:0/3936343808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 msgr2=0x560179e12f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e12f10 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5f74014f60 tx=0x7f5f74009e30 comp rx=0 tx=0).stop 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 -- 172.21.15.27:0/3936343808 shutdown_connections 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560179e8f8f0 0x560179e91d10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560179e18620 0x560179e12f10 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560179e0df50 0x560179e129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 --2- 172.21.15.27:0/3936343808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560179e0d630 0x560179e12490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f5b7fe640 0 client.16563 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 -- 172.21.15.27:0/3936343808 >> 172.21.15.27:0/3936343808 conn(0x560179e09950 msgr2=0x560179e08af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:49.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 -- 172.21.15.27:0/3936343808 shutdown_connections 2024-04-25T01:29:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 1 -- 172.21.15.27:0/3936343808 wait complete. 2024-04-25T01:29:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.840+0000 7f5f85396740 20 client.16563 trim_cache size 0 max 16384 2024-04-25T01:29:49.826 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:29:49.826 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:29:49.826 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:29:49.826 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:29:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 -- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f640a2d00 msgr2=0x7f5f640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f640a2d00 0x7f5f640a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5f74022000 tx=0x7f5f7401e000 comp rx=0 tx=0).stop 2024-04-25T01:29:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 -- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 msgr2=0x7f5f7c198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 0x7f5f7c198540 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006010 tx=0x7f5f700042f0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 -- 172.21.15.27:0/1124785615 shutdown_connections 2024-04-25T01:29:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f7c10bd00 0x7f5f7c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f640a2d00 0x7f5f640a51c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f7c1032e0 0x7f5f7c198540 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 --2- 172.21.15.27:0/1124785615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f7c1029c0 0x7f5f7c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 -- 172.21.15.27:0/1124785615 >> 172.21.15.27:0/1124785615 conn(0x7f5f7c077740 msgr2=0x7f5f7c0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 -- 172.21.15.27:0/1124785615 shutdown_connections 2024-04-25T01:29:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.841+0000 7f5f825a9640 1 -- 172.21.15.27:0/1124785615 wait complete. 2024-04-25T01:29:49.830 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:29:49.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:29:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.955+0000 7f5faf6da640 1 -- 172.21.15.27:0/1702840980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 msgr2=0x7f5fa810e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.955+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1702840980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 0x7f5fa810e1d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009ca0 tx=0x7f5fa002a950 comp rx=0 tx=0).stop 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.955+0000 7f5faf6da640 1 -- 172.21.15.27:0/1702840980 shutdown_connections 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.955+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1702840980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 0x7f5fa810e1d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.955+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1702840980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fa8103340 0x7f5fa810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.955+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1702840980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fa8102a20 0x7f5fa8102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.955+0000 7f5faf6da640 1 -- 172.21.15.27:0/1702840980 >> 172.21.15.27:0/1702840980 conn(0x7f5fa80777a0 msgr2=0x7f5fa8077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 -- 172.21.15.27:0/1702840980 shutdown_connections 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 -- 172.21.15.27:0/1702840980 wait complete. 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 Processor -- start 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 -- start start 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fa8102a20 0x7f5fa8100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fa8103340 0x7f5fa8100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 0x7f5fa81010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa8111b30 con 0x7f5fa810bd60 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81119b0 con 0x7f5fa8102a20 2024-04-25T01:29:49.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5faf6da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa8111cb0 con 0x7f5fa8103340 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5fadc77640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 0x7f5fa81010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.956+0000 7f5fadc77640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 0x7f5fa81010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56206/0 (socket says 172.21.15.27:56206) 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fadc77640 1 -- 172.21.15.27:0/1779395262 learned_addr learned my addr 172.21.15.27:0/1779395262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5facc75640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fa8103340 0x7f5fa8100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fad476640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fa8102a20 0x7f5fa8100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fadc77640 1 -- 172.21.15.27:0/1779395262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fa8103340 msgr2=0x7f5fa8100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fadc77640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fa8103340 0x7f5fa8100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fadc77640 1 -- 172.21.15.27:0/1779395262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fa8102a20 msgr2=0x7f5fa8100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fadc77640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fa8102a20 0x7f5fa8100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fadc77640 1 -- 172.21.15.27:0/1779395262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa81a6450 con 0x7f5fa810bd60 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fad476640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fa8102a20 0x7f5fa8100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5fadc77640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 0x7f5fa81010f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0004af0 tx=0x7f5fa002f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5f9e7fc640 1 -- 172.21.15.27:0/1779395262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0034040 con 0x7f5fa810bd60 2024-04-25T01:29:49.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5f9e7fc640 1 -- 172.21.15.27:0/1779395262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fa0030920 con 0x7f5fa810bd60 2024-04-25T01:29:49.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa81a66e0 con 0x7f5fa810bd60 2024-04-25T01:29:49.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5f9e7fc640 1 -- 172.21.15.27:0/1779395262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0004670 con 0x7f5fa810bd60 2024-04-25T01:29:49.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.957+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa81a6bd0 con 0x7f5fa810bd60 2024-04-25T01:29:49.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.958+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa80689b0 con 0x7f5fa810bd60 2024-04-25T01:29:49.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.959+0000 7f5f9e7fc640 1 -- 172.21.15.27:0/1779395262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5fa0037020 con 0x7f5fa810bd60 2024-04-25T01:29:49.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.960+0000 7f5f9e7fc640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f940a2d30 0x7f5f940a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.960+0000 7f5fad476640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f940a2d30 0x7f5f940a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.960+0000 7f5f9e7fc640 1 -- 172.21.15.27:0/1779395262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5fa00f0450 con 0x7f5fa810bd60 2024-04-25T01:29:49.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.961+0000 7f5fad476640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f940a2d30 0x7f5f940a51f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4005d00 tx=0x7f5fa4009550 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:49.965+0000 7f5f9e7fc640 1 -- 172.21.15.27:0/1779395262 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5fa00bcdf0 con 0x7f5fa810bd60 2024-04-25T01:29:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.190+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fa802d170 con 0x7f5fa810bd60 2024-04-25T01:29:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.191+0000 7f5f9e7fc640 1 -- 172.21.15.27:0/1779395262 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7f5fa00bc6f0 con 0x7f5fa810bd60 2024-04-25T01:29:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:29:50.170 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:29:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f940a2d30 msgr2=0x7f5f940a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f940a2d30 0x7f5f940a51f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4005d00 tx=0x7f5fa4009550 comp rx=0 tx=0).stop 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 msgr2=0x7f5fa81010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 0x7f5fa81010f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0004af0 tx=0x7f5fa002f3a0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 shutdown_connections 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f940a2d30 0x7f5f940a51f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa810bd60 0x7f5fa81010f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fa8103340 0x7f5fa8100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 --2- 172.21.15.27:0/1779395262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5fa8102a20 0x7f5fa8100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.193+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 >> 172.21.15.27:0/1779395262 conn(0x7f5fa80777a0 msgr2=0x7f5fa80fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.194+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 shutdown_connections 2024-04-25T01:29:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.194+0000 7f5faf6da640 1 -- 172.21.15.27:0/1779395262 wait complete. 2024-04-25T01:29:50.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:29:50.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 -- 172.21.15.27:0/3177580182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 msgr2=0x7fadc4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3177580182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 0x7fadc4106a90 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009930 tx=0x7fadb402a420 comp rx=0 tx=0).stop 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 -- 172.21.15.27:0/3177580182 shutdown_connections 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3177580182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadc4100fd0 0x7fadc410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3177580182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadc4100610 0x7fadc4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3177580182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 0x7fadc4106a90 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 -- 172.21.15.27:0/3177580182 >> 172.21.15.27:0/3177580182 conn(0x7fadc4077740 msgr2=0x7fadc4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 -- 172.21.15.27:0/3177580182 shutdown_connections 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 -- 172.21.15.27:0/3177580182 wait complete. 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.309+0000 7fadc8a31640 1 Processor -- start 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc8a31640 1 -- start start 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc8a31640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadc4100610 0x7fadc4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc8a31640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadc4100fd0 0x7fadc4198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc8a31640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 0x7fadc419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc8a31640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fadc4110a30 con 0x7fadc41066b0 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc8a31640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41108b0 con 0x7fadc4100610 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc8a31640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fadc4110bb0 con 0x7fadc4100fd0 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 0x7fadc419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 0x7fadc419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56214/0 (socket says 172.21.15.27:56214) 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 -- 172.21.15.27:0/3482136514 learned_addr learned my addr 172.21.15.27:0/3482136514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc259c640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadc4100610 0x7fadc4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadbbfff640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadc4100fd0 0x7fadc4198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 -- 172.21.15.27:0/3482136514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadc4100fd0 msgr2=0x7fadc4198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadc4100fd0 0x7fadc4198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 -- 172.21.15.27:0/3482136514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadc4100610 msgr2=0x7fadc4198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadc4100610 0x7fadc4198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 -- 172.21.15.27:0/3482136514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc419cfe0 con 0x7fadc41066b0 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc259c640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadc4100610 0x7fadc4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadbbfff640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadc4100fd0 0x7fadc4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.310+0000 7fadc2d9d640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 0x7fadc419c8b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00eb10 tx=0x7fadbc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.311+0000 7fadbb7fe640 1 -- 172.21.15.27:0/3482136514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc004b20 con 0x7fadc41066b0 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.311+0000 7fadbb7fe640 1 -- 172.21.15.27:0/3482136514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadbc004cc0 con 0x7fadc41066b0 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.311+0000 7fadc8a31640 1 -- 172.21.15.27:0/3482136514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc419d2d0 con 0x7fadc41066b0 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.311+0000 7fadbb7fe640 1 -- 172.21.15.27:0/3482136514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc010760 con 0x7fadc41066b0 2024-04-25T01:29:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.311+0000 7fadc8a31640 1 -- 172.21.15.27:0/3482136514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc41a4b60 con 0x7fadc41066b0 2024-04-25T01:29:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadbb7fe640 1 -- 172.21.15.27:0/3482136514 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fadbc005020 con 0x7fadc41066b0 2024-04-25T01:29:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadcb819740 1 Processor -- start 2024-04-25T01:29:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadcb819740 1 -- start start 2024-04-25T01:29:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadcb819740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f1e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadcb819740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadcb819740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadcb819740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561317e83670 con 0x561317f2d980 2024-04-25T01:29:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadcb819740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561317e2b970 con 0x561317f1ec60 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadcb819740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561317e7f630 con 0x561317f1e2b0 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56222/0 (socket says 172.21.15.27:56222) 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadbbfff640 1 -- 172.21.15.27:0/2232165633 learned_addr learned my addr 172.21.15.27:0/2232165633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.314+0000 7fadbb7fe640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fadb00a2d00 0x7fadb00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadbbfff640 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f1e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadc259c640 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadc2d9d640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fadb00a2d00 0x7fadb00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadbbfff640 1 -- 172.21.15.27:0/2232165633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 msgr2=0x561317f1e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadbbfff640 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f1e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadbbfff640 1 -- 172.21.15.27:0/2232165633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 msgr2=0x561317f2d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadbbfff640 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadbbfff640 1 -- 172.21.15.27:0/2232165633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561317e4f400 con 0x561317f2d980 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadc259c640 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadbb7fe640 1 -- 172.21.15.27:0/3482136514 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fadbc0cfec0 con 0x7fadc41066b0 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadbbfff640 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2fd70 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fadac009fa0 tx=0x7fadac009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe03f150f1a40b1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadb8ff9640 1 -- 172.21.15.27:0/2232165633 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac0134c0 con 0x561317f2d980 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadb8ff9640 1 -- 172.21.15.27:0/2232165633 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadac01b070 con 0x561317f2d980 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.315+0000 7fadb8ff9640 1 -- 172.21.15.27:0/2232165633 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac0128b0 con 0x561317f2d980 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 -- 172.21.15.27:0/2232165633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 msgr2=0x561317f2fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadc2d9d640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fadb00a2d00 0x7fadb00a51c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0cd880 tx=0x7fadbc067070 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2fd70 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fadac009fa0 tx=0x7fadac009650 comp rx=0 tx=0).stop 2024-04-25T01:29:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 -- 172.21.15.27:0/2232165633 shutdown_connections 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2fd70 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 --2- 172.21.15.27:0/2232165633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f1e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 -- 172.21.15.27:0/2232165633 >> 172.21.15.27:0/2232165633 conn(0x561317f1a8a0 msgr2=0x561317f1ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 -- 172.21.15.27:0/2232165633 shutdown_connections 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 -- 172.21.15.27:0/2232165633 wait complete. 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 -- start start 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.316+0000 7fadcb819740 1 Processor -- start 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadcb819740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f2d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadcb819740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadcb819740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadcb819740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561317f30740 con 0x561317f2d980 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadcb819740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561317e3f050 con 0x561317f1ec60 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadcb819740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561317f308c0 con 0x561317f1e2b0 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadcb819740 10 client.0 _ll_register_callbacks cb 0x561317b96be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56236/0 (socket says 172.21.15.27:56236) 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 -- 172.21.15.27:0/2252343830 learned_addr learned my addr 172.21.15.27:0/2252343830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f2d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadc259c640 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 -- 172.21.15.27:0/2252343830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 msgr2=0x561317f2d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f2d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 -- 172.21.15.27:0/2252343830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 msgr2=0x561317f2a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 -- 172.21.15.27:0/2252343830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561317e10dc0 con 0x561317f2d980 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadbbfff640 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2ac50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fadac001eb0 tx=0x7fadac012490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffca91569907b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fada9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fada9ffb640 1 -- 172.21.15.27:0/2252343830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac013950 con 0x561317f2d980 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fadcb819740 1 -- 172.21.15.27:0/2252343830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadac01b070 con 0x561317f2d980 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.317+0000 7fada9ffb640 1 -- 172.21.15.27:0/2252343830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fadac013af0 con 0x561317f2d980 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fada9ffb640 1 -- 172.21.15.27:0/2252343830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fadac005020 con 0x561317f2d980 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadcb819740 10 client.16581 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadcb819740 1 -- 172.21.15.27:0/2252343830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561317b973f0 con 0x561317f2d980 2024-04-25T01:29:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fada9ffb640 1 -- 172.21.15.27:0/2252343830 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadac01b070 con 0x561317f2d980 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fada9ffb640 1 -- 172.21.15.27:0/2252343830 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fadac00c070 con 0x561317f2d980 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadcb819740 10 client.16581 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadcb819740 10 client.16581 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadcb819740 20 client.16581 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadcb819740 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561317f3d840 0x561317f3fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadcb819740 4 client.16581 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadcb819740 1 -- 172.21.15.27:0/2252343830 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561317e90bb0 con 0x561317f3d840 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.318+0000 7fadc259c640 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561317f3d840 0x561317f3fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.319+0000 7fadc259c640 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561317f3d840 0x561317f3fc30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17965cbe42eca46a server_cookie=e93f7f6aea925752 in_seq=0 out_seq=0 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.319+0000 7fada9ffb640 10 client.16581 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:29:50.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.319+0000 7fada9ffb640 1 -- 172.21.15.27:0/2252343830 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561317e90bb0 con 0x561317f3d840 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.319+0000 7fada9ffb640 10 client.16581 handle_command_reply: tid=0 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 client.16581 shutdown 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 -- 172.21.15.27:0/2252343830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 msgr2=0x561317f2ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2ac50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fadac001eb0 tx=0x7fadac012490 comp rx=0 tx=0).stop 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 -- 172.21.15.27:0/2252343830 shutdown_connections 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561317f3d840 0x561317f3fc30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f2d980 0x561317f2ac50 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1ec60 0x561317f2a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 --2- 172.21.15.27:0/2252343830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f2d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fada9ffb640 0 client.16581 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 -- 172.21.15.27:0/2252343830 >> 172.21.15.27:0/2252343830 conn(0x561317f1a8a0 msgr2=0x561317f206f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 -- 172.21.15.27:0/2252343830 shutdown_connections 2024-04-25T01:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 1 -- 172.21.15.27:0/2252343830 wait complete. 2024-04-25T01:29:50.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.320+0000 7fadcb819740 20 client.16581 trim_cache size 0 max 16384 2024-04-25T01:29:50.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadcb819740 1 Processor -- start 2024-04-25T01:29:50.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadcb819740 1 -- start start 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadcb819740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f39640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadcb819740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f1fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadcb819740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f2d980 0x561317f2ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadcb819740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41af360 con 0x561317f1ec60 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadcb819740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41acc30 con 0x561317f2d980 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadcb819740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41aaa80 con 0x561317f1e2b0 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadc259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f1fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadc259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f1fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56238/0 (socket says 172.21.15.27:56238) 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadc259c640 1 -- 172.21.15.27:0/202570376 learned_addr learned my addr 172.21.15.27:0/202570376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadbbfff640 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f2d980 0x561317f2ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadc259c640 1 -- 172.21.15.27:0/202570376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 msgr2=0x561317f39640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadc259c640 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f39640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.333+0000 7fadc259c640 1 -- 172.21.15.27:0/202570376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f2d980 msgr2=0x561317f2ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadc259c640 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f2d980 0x561317f2ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadc259c640 1 -- 172.21.15.27:0/202570376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561317c617d0 con 0x561317f1ec60 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadbbfff640 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f2d980 0x561317f2ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadc259c640 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f1fe60 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fadb4006010 tx=0x7fadb4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6150f7a890247c46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadabfff640 1 -- 172.21.15.27:0/202570376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb4035040 con 0x561317f1ec60 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadabfff640 1 -- 172.21.15.27:0/202570376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb4005520 con 0x561317f1ec60 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadabfff640 1 -- 172.21.15.27:0/202570376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb4043530 con 0x561317f1ec60 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 -- 172.21.15.27:0/202570376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 msgr2=0x561317f1fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f1fe60 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fadb4006010 tx=0x7fadb4005b50 comp rx=0 tx=0).stop 2024-04-25T01:29:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 -- 172.21.15.27:0/202570376 shutdown_connections 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f2d980 0x561317f2ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f1fe60 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 --2- 172.21.15.27:0/202570376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f1e2b0 0x561317f39640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 -- 172.21.15.27:0/202570376 >> 172.21.15.27:0/202570376 conn(0x561317f1a8a0 msgr2=0x561317f19b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 -- 172.21.15.27:0/202570376 shutdown_connections 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.334+0000 7fadcb819740 1 -- 172.21.15.27:0/202570376 wait complete. 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 1 -- start start 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 1 Processor -- start 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1e2b0 0x561317f35cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f31130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f2d980 0x561317f31670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41ab0f0 con 0x561317f1ec60 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41ac560 con 0x561317f1e2b0 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41ab7c0 con 0x561317f2d980 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadcb819740 10 client.0 _ll_register_callbacks cb 0x7fadac0154b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadc259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f31130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadc259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f31130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56244/0 (socket says 172.21.15.27:56244) 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadc259c640 1 -- 172.21.15.27:0/1483947922 learned_addr learned my addr 172.21.15.27:0/1483947922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadbbfff640 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1e2b0 0x561317f35cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadc259c640 1 -- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f2d980 msgr2=0x561317f31670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadbbfff640 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f2d980 0x561317f31670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadc259c640 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f2d980 0x561317f31670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadc259c640 1 -- 172.21.15.27:0/1483947922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1e2b0 msgr2=0x561317f35cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadc259c640 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1e2b0 0x561317f35cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadc259c640 1 -- 172.21.15.27:0/1483947922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561317f31c60 con 0x561317f1ec60 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.335+0000 7fadbbfff640 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f2d980 0x561317f31670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadc259c640 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f31130 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fadb40092a0 tx=0x7fadb4004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a712478025e1e390 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fada9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fada9ffb640 1 -- 172.21.15.27:0/1483947922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb4050070 con 0x561317f1ec60 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fada9ffb640 1 -- 172.21.15.27:0/1483947922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fadb40318c0 con 0x561317f1ec60 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 1 -- 172.21.15.27:0/1483947922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadb4005520 con 0x561317f1ec60 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fada9ffb640 1 -- 172.21.15.27:0/1483947922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fadb4035040 con 0x561317f1ec60 2024-04-25T01:29:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 10 client.16587 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 1 -- 172.21.15.27:0/1483947922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561317fa3910 con 0x561317f1ec60 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fada9ffb640 1 -- 172.21.15.27:0/1483947922 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb4005520 con 0x561317f1ec60 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fada9ffb640 1 -- 172.21.15.27:0/1483947922 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fadb403c440 con 0x561317f1ec60 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 10 client.16587 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 10 client.16587 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 20 client.16587 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561317fa4030 0x561317fa6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 4 client.16587 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.336+0000 7fadcb819740 1 -- 172.21.15.27:0/1483947922 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561317e8af00 con 0x561317fa4030 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadbbfff640 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561317fa4030 0x561317fa6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadbbfff640 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561317fa4030 0x561317fa6450 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ab5fbb2986920dc server_cookie=9a1c6fae1960fd91 in_seq=0 out_seq=0 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fada9ffb640 10 client.16587 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fada9ffb640 1 -- 172.21.15.27:0/1483947922 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561317e8af00 con 0x561317fa4030 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fada9ffb640 10 client.16587 handle_command_reply: tid=0 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 client.16587 shutdown 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 -- 172.21.15.27:0/1483947922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 msgr2=0x561317f31130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f31130 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fadb40092a0 tx=0x7fadb4004510 comp rx=0 tx=0).stop 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 -- 172.21.15.27:0/1483947922 shutdown_connections 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561317fa4030 0x561317fa6450 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561317f2d980 0x561317f31670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fada9ffb640 0 client.16587 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561317f1ec60 0x561317f31130 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 --2- 172.21.15.27:0/1483947922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561317f1e2b0 0x561317f35cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 -- 172.21.15.27:0/1483947922 >> 172.21.15.27:0/1483947922 conn(0x561317f1a8a0 msgr2=0x561317f38630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 -- 172.21.15.27:0/1483947922 shutdown_connections 2024-04-25T01:29:50.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 1 -- 172.21.15.27:0/1483947922 wait complete. 2024-04-25T01:29:50.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.337+0000 7fadcb819740 20 client.16587 trim_cache size 0 max 16384 2024-04-25T01:29:50.325 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:29:50.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 -- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fadb00a2d00 msgr2=0x7fadb00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fadb00a2d00 0x7fadb00a51c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0cd880 tx=0x7fadbc067070 comp rx=0 tx=0).stop 2024-04-25T01:29:50.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 -- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 msgr2=0x7fadc419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 0x7fadc419c8b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00eb10 tx=0x7fadbc00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 -- 172.21.15.27:0/3482136514 shutdown_connections 2024-04-25T01:29:50.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fadb00a2d00 0x7fadb00a51c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadc41066b0 0x7fadc419c8b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fadc4100fd0 0x7fadc4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 --2- 172.21.15.27:0/3482136514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fadc4100610 0x7fadc4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 -- 172.21.15.27:0/3482136514 >> 172.21.15.27:0/3482136514 conn(0x7fadc4077740 msgr2=0x7fadc40ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 -- 172.21.15.27:0/3482136514 shutdown_connections 2024-04-25T01:29:50.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.338+0000 7fadc8a31640 1 -- 172.21.15.27:0/3482136514 wait complete. 2024-04-25T01:29:50.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:29:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 -- 172.21.15.27:0/1949536616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c100920 msgr2=0x7ff71c100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 --2- 172.21.15.27:0/1949536616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c100920 0x7ff71c100da0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009930 tx=0x7ff70c02a420 comp rx=0 tx=0).stop 2024-04-25T01:29:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 -- 172.21.15.27:0/1949536616 shutdown_connections 2024-04-25T01:29:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 --2- 172.21.15.27:0/1949536616 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff71c1012e0 0x7ff71c10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 --2- 172.21.15.27:0/1949536616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c100920 0x7ff71c100da0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 --2- 172.21.15.27:0/1949536616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c106810 0x7ff71c106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 -- 172.21.15.27:0/1949536616 >> 172.21.15.27:0/1949536616 conn(0x7ff71c077620 msgr2=0x7ff71c077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 -- 172.21.15.27:0/1949536616 shutdown_connections 2024-04-25T01:29:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.453+0000 7ff72436a640 1 -- 172.21.15.27:0/1949536616 wait complete. 2024-04-25T01:29:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff72436a640 1 Processor -- start 2024-04-25T01:29:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff72436a640 1 -- start start 2024-04-25T01:29:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff72436a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c100920 0x7ff71c196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff72436a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1012e0 0x7ff71c1974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff72436a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff71c106810 0x7ff71c19b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff72436a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c110ae0 con 0x7ff71c1012e0 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff72436a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c110960 con 0x7ff71c106810 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff72436a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c110c60 con 0x7ff71c100920 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1012e0 0x7ff71c1974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1012e0 0x7ff71c1974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56272/0 (socket says 172.21.15.27:56272) 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 -- 172.21.15.27:0/1463397700 learned_addr learned my addr 172.21.15.27:0/1463397700 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff722106640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c100920 0x7ff71c196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff722907640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff71c106810 0x7ff71c19b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 -- 172.21.15.27:0/1463397700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c100920 msgr2=0x7ff71c196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c100920 0x7ff71c196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 -- 172.21.15.27:0/1463397700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff71c106810 msgr2=0x7ff71c19b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff71c106810 0x7ff71c19b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 -- 172.21.15.27:0/1463397700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c19bf90 con 0x7ff71c1012e0 2024-04-25T01:29:50.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff722907640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff71c106810 0x7ff71c19b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff722106640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c100920 0x7ff71c196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:50.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.454+0000 7ff721905640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1012e0 0x7ff71c1974e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009230 tx=0x7ff70c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.455+0000 7ff71b7fe640 1 -- 172.21.15.27:0/1463397700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c03f030 con 0x7ff71c1012e0 2024-04-25T01:29:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.455+0000 7ff71b7fe640 1 -- 172.21.15.27:0/1463397700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff70c003ad0 con 0x7ff71c1012e0 2024-04-25T01:29:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.455+0000 7ff72436a640 1 -- 172.21.15.27:0/1463397700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71c19c220 con 0x7ff71c1012e0 2024-04-25T01:29:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.455+0000 7ff71b7fe640 1 -- 172.21.15.27:0/1463397700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c035070 con 0x7ff71c1012e0 2024-04-25T01:29:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.455+0000 7ff72436a640 1 -- 172.21.15.27:0/1463397700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff71c069d60 con 0x7ff71c1012e0 2024-04-25T01:29:50.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.457+0000 7ff71b7fe640 1 -- 172.21.15.27:0/1463397700 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff70c042030 con 0x7ff71c1012e0 2024-04-25T01:29:50.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.457+0000 7ff72707f740 1 Processor -- start 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff72707f740 1 -- start start 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff72707f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04df960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff72707f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f04ea030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff72707f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 0x5565f04ec960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff72707f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5565f0444670 con 0x5565f04dfea0 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff72707f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5565f03ec970 con 0x5565f04ea570 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff72707f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5565f0440630 con 0x5565f04df580 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff71b7fe640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff7040a2d80 0x7ff7040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722106640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f04ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722106640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f04ea030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56282/0 (socket says 172.21.15.27:56282) 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722907640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff7040a2d80 0x7ff7040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722106640 1 -- 172.21.15.27:0/1365334751 learned_addr learned my addr 172.21.15.27:0/1365334751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff721905640 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 0x5565f04ec960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722907640 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff71b7fe640 1 -- 172.21.15.27:0/1463397700 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff70c0efd10 con 0x7ff71c1012e0 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722106640 1 -- 172.21.15.27:0/1365334751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 msgr2=0x5565f04df960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722106640 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04df960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722106640 1 -- 172.21.15.27:0/1365334751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 msgr2=0x5565f04ec960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.458+0000 7ff722106640 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 0x5565f04ec960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.459+0000 7ff722106640 1 -- 172.21.15.27:0/1365334751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f0410400 con 0x5565f04dfea0 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.459+0000 7ff722907640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff7040a2d80 0x7ff7040a5240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff714005fd0 tx=0x7ff714009df0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.459+0000 7ff722106640 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f04ea030 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff71000cc20 tx=0x7ff71000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb0507e3a2b71e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.459+0000 7ff718bfe640 1 -- 172.21.15.27:0/1365334751 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710005220 con 0x5565f04dfea0 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.459+0000 7ff718bfe640 1 -- 172.21.15.27:0/1365334751 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff710016070 con 0x5565f04dfea0 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.459+0000 7ff718bfe640 1 -- 172.21.15.27:0/1365334751 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710010630 con 0x5565f04dfea0 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- 172.21.15.27:0/1365334751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 msgr2=0x5565f04ea030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f04ea030 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff71000cc20 tx=0x7ff71000e720 comp rx=0 tx=0).stop 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- 172.21.15.27:0/1365334751 shutdown_connections 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 0x5565f04ec960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f04ea030 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 --2- 172.21.15.27:0/1365334751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04df960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- 172.21.15.27:0/1365334751 >> 172.21.15.27:0/1365334751 conn(0x5565f04db8a0 msgr2=0x5565f04dbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- 172.21.15.27:0/1365334751 shutdown_connections 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- 172.21.15.27:0/1365334751 wait complete. 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- start start 2024-04-25T01:29:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 Processor -- start 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04e9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 0x5565f04e7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04e7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5565f04ed360 con 0x5565f04ea570 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5565f0400050 con 0x5565f04dfea0 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.460+0000 7ff72707f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5565f04ed4e0 con 0x5565f04df580 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff72707f740 10 client.0 _ll_register_callbacks cb 0x5565f0157be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04e7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04e7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56298/0 (socket says 172.21.15.27:56298) 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722106640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04e9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 -- 172.21.15.27:0/3603102059 learned_addr learned my addr 172.21.15.27:0/3603102059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff721905640 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 0x5565f04e7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 -- 172.21.15.27:0/3603102059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 msgr2=0x5565f04e9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04e9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 -- 172.21.15.27:0/3603102059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 msgr2=0x5565f04e7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 0x5565f04e7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 -- 172.21.15.27:0/3603102059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f03d1dc0 con 0x5565f04ea570 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff722907640 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04e7660 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff714010ce0 tx=0x7ff714009440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14d32f821685655 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff701ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff701ffb640 1 -- 172.21.15.27:0/3603102059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714024070 con 0x5565f04ea570 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff701ffb640 1 -- 172.21.15.27:0/3603102059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff714003b70 con 0x5565f04ea570 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff72707f740 1 -- 172.21.15.27:0/3603102059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff710016070 con 0x5565f04ea570 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff701ffb640 1 -- 172.21.15.27:0/3603102059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff714003d70 con 0x5565f04ea570 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff72707f740 10 client.16599 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.461+0000 7ff72707f740 1 -- 172.21.15.27:0/3603102059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5565f01583f0 con 0x5565f04ea570 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff701ffb640 1 -- 172.21.15.27:0/3603102059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff710016070 con 0x5565f04ea570 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff701ffb640 1 -- 172.21.15.27:0/3603102059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff71401c040 con 0x5565f04ea570 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff72707f740 10 client.16599 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff72707f740 10 client.16599 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff72707f740 20 client.16599 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff72707f740 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5565f04fa570 0x5565f04fc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff72707f740 4 client.16599 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff72707f740 1 -- 172.21.15.27:0/3603102059 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565f0451bb0 con 0x5565f04fa570 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.462+0000 7ff722106640 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5565f04fa570 0x5565f04fc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff701ffb640 10 client.16599 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff722106640 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5565f04fa570 0x5565f04fc9c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb7e3716c8af7f29 server_cookie=5fcd6db2d085b79c in_seq=0 out_seq=0 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff701ffb640 1 -- 172.21.15.27:0/3603102059 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565f0451bb0 con 0x5565f04fa570 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff701ffb640 10 client.16599 handle_command_reply: tid=0 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 client.16599 shutdown 2024-04-25T01:29:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 -- 172.21.15.27:0/3603102059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 msgr2=0x5565f04e7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04e7660 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff714010ce0 tx=0x7ff714009440 comp rx=0 tx=0).stop 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 -- 172.21.15.27:0/3603102059 shutdown_connections 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5565f04fa570 0x5565f04fc9c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04e7660 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 0x5565f04e7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff701ffb640 0 client.16599 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 --2- 172.21.15.27:0/3603102059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04e9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.463+0000 7ff72707f740 1 -- 172.21.15.27:0/3603102059 >> 172.21.15.27:0/3603102059 conn(0x5565f04db8a0 msgr2=0x5565f04dab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.464+0000 7ff72707f740 1 -- 172.21.15.27:0/3603102059 shutdown_connections 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.464+0000 7ff72707f740 1 -- 172.21.15.27:0/3603102059 wait complete. 2024-04-25T01:29:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.464+0000 7ff72707f740 20 client.16599 trim_cache size 0 max 16384 2024-04-25T01:29:50.453 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:50 smithi027 ceph-mon[29197]: pgmap v135: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:29:50.454 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:50 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1779395262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff72707f740 1 Processor -- start 2024-04-25T01:29:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff72707f740 1 -- start start 2024-04-25T01:29:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff72707f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff72707f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f0502290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff72707f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 0x5565f05027d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff72707f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c1b14a0 con 0x5565f04dfea0 2024-04-25T01:29:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff72707f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c1aed70 con 0x5565f04ea570 2024-04-25T01:29:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff72707f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c1acbc0 con 0x5565f04df580 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff721905640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f0502290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff721905640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f0502290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56306/0 (socket says 172.21.15.27:56306) 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.477+0000 7ff721905640 1 -- 172.21.15.27:0/1987424512 learned_addr learned my addr 172.21.15.27:0/1987424512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff722106640 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff721905640 1 -- 172.21.15.27:0/1987424512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 msgr2=0x5565f04f6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff722907640 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 0x5565f05027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff721905640 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff721905640 1 -- 172.21.15.27:0/1987424512 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 msgr2=0x5565f05027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff721905640 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 0x5565f05027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff721905640 1 -- 172.21.15.27:0/1987424512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f02227d0 con 0x5565f04dfea0 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff722106640 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff721905640 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f0502290 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009a50 tx=0x7ff70c043f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b3909c4ae93e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff703fff640 1 -- 172.21.15.27:0/1987424512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c049040 con 0x5565f04dfea0 2024-04-25T01:29:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff703fff640 1 -- 172.21.15.27:0/1987424512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff70c0c4b10 con 0x5565f04dfea0 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.478+0000 7ff703fff640 1 -- 172.21.15.27:0/1987424512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c003c80 con 0x5565f04dfea0 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- 172.21.15.27:0/1987424512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 msgr2=0x5565f0502290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f0502290 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009a50 tx=0x7ff70c043f90 comp rx=0 tx=0).stop 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- 172.21.15.27:0/1987424512 shutdown_connections 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04ea570 0x5565f05027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04dfea0 0x5565f0502290 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 --2- 172.21.15.27:0/1987424512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- 172.21.15.27:0/1987424512 >> 172.21.15.27:0/1987424512 conn(0x5565f04db8a0 msgr2=0x5565f04dab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- 172.21.15.27:0/1987424512 shutdown_connections 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- 172.21.15.27:0/1987424512 wait complete. 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- start start 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 Processor -- start 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 0x5565f04f0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c1ad230 con 0x5565f04ea570 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c1ae6a0 con 0x5565f04dfea0 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c1ad900 con 0x5565f04df580 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff72707f740 10 client.0 _ll_register_callbacks cb 0x7ff714009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff722907640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff722907640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04f0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56308/0 (socket says 172.21.15.27:56308) 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff722907640 1 -- 172.21.15.27:0/3089862675 learned_addr learned my addr 172.21.15.27:0/3089862675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.479+0000 7ff722106640 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff722907640 1 -- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 msgr2=0x5565f04f0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff721905640 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 0x5565f04f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff722907640 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff722907640 1 -- 172.21.15.27:0/3089862675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 msgr2=0x5565f04f0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff722907640 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 0x5565f04f0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff722907640 1 -- 172.21.15.27:0/3089862675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f055fb10 con 0x5565f04ea570 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff722106640 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff722907640 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04f0bd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff71401e6a0 tx=0x7ff714003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f3f354180b0ab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff701ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff701ffb640 1 -- 172.21.15.27:0/3089862675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71403a030 con 0x5565f04ea570 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff701ffb640 1 -- 172.21.15.27:0/3089862675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff714011130 con 0x5565f04ea570 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff72707f740 1 -- 172.21.15.27:0/3089862675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff70c0c4b10 con 0x5565f04ea570 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff701ffb640 1 -- 172.21.15.27:0/3089862675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff714044050 con 0x5565f04ea570 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff72707f740 10 client.16605 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff72707f740 1 -- 172.21.15.27:0/3089862675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5565f0560d50 con 0x5565f04ea570 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff701ffb640 1 -- 172.21.15.27:0/3089862675 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff70c0c4b10 con 0x5565f04ea570 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.480+0000 7ff701ffb640 1 -- 172.21.15.27:0/3089862675 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff71401c050 con 0x5565f04ea570 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff72707f740 10 client.16605 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff72707f740 10 client.16605 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff72707f740 20 client.16605 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff72707f740 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5565f05614f0 0x5565f0563910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff72707f740 4 client.16605 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff72707f740 1 -- 172.21.15.27:0/3089862675 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565f044bf00 con 0x5565f05614f0 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff722106640 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5565f05614f0 0x5565f0563910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff722106640 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5565f05614f0 0x5565f0563910 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b8b29188c42beab server_cookie=4f9e25c3f10b8eec in_seq=0 out_seq=0 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff701ffb640 10 client.16605 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff701ffb640 1 -- 172.21.15.27:0/3089862675 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565f044bf00 con 0x5565f05614f0 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff701ffb640 10 client.16605 handle_command_reply: tid=0 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff72707f740 1 client.16605 shutdown 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.481+0000 7ff72707f740 1 -- 172.21.15.27:0/3089862675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 msgr2=0x5565f04f0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04f0bd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff71401e6a0 tx=0x7ff714003ba0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 -- 172.21.15.27:0/3089862675 shutdown_connections 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5565f05614f0 0x5565f0563910 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565f04ea570 0x5565f04f0bd0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565f04dfea0 0x5565f04f0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff701ffb640 0 client.16605 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 --2- 172.21.15.27:0/3089862675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565f04df580 0x5565f04f0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 -- 172.21.15.27:0/3089862675 >> 172.21.15.27:0/3089862675 conn(0x5565f04db8a0 msgr2=0x5565f04f5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 -- 172.21.15.27:0/3089862675 shutdown_connections 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 1 -- 172.21.15.27:0/3089862675 wait complete. 2024-04-25T01:29:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72707f740 20 client.16605 trim_cache size 0 max 16384 2024-04-25T01:29:50.470 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 -- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff7040a2d80 msgr2=0x7ff7040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff7040a2d80 0x7ff7040a5240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff714005fd0 tx=0x7ff714009df0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 -- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1012e0 msgr2=0x7ff71c1974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1012e0 0x7ff71c1974e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009230 tx=0x7ff70c004270 comp rx=0 tx=0).stop 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 -- 172.21.15.27:0/1463397700 shutdown_connections 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff71c106810 0x7ff71c19b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff7040a2d80 0x7ff7040a5240 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1012e0 0x7ff71c1974e0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 --2- 172.21.15.27:0/1463397700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c100920 0x7ff71c196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 -- 172.21.15.27:0/1463397700 >> 172.21.15.27:0/1463397700 conn(0x7ff71c077620 msgr2=0x7ff71c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 -- 172.21.15.27:0/1463397700 shutdown_connections 2024-04-25T01:29:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.482+0000 7ff72436a640 1 -- 172.21.15.27:0/1463397700 wait complete. 2024-04-25T01:29:50.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:29:50.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:50 smithi139 ceph-mon[32760]: pgmap v135: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:29:50.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:50 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1779395262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:50.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 -- 172.21.15.27:0/3820992733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c069dc0 msgr2=0x7f787c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 --2- 172.21.15.27:0/3820992733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c069dc0 0x7f787c06a240 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f786c009930 tx=0x7f786c02a420 comp rx=0 tx=0).stop 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 -- 172.21.15.27:0/3820992733 shutdown_connections 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 --2- 172.21.15.27:0/3820992733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f787c06a780 0x7f787c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 --2- 172.21.15.27:0/3820992733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c069dc0 0x7f787c06a240 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 --2- 172.21.15.27:0/3820992733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f787c107730 0x7f787c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 -- 172.21.15.27:0/3820992733 >> 172.21.15.27:0/3820992733 conn(0x7f787c0776a0 msgr2=0x7f787c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 -- 172.21.15.27:0/3820992733 shutdown_connections 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.597+0000 7f7881bd5640 1 -- 172.21.15.27:0/3820992733 wait complete. 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f7881bd5640 1 Processor -- start 2024-04-25T01:29:50.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f7881bd5640 1 -- start start 2024-04-25T01:29:50.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f7881bd5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f787c069dc0 0x7f787c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f7881bd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c06a780 0x7f787c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f7881bd5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f787c107730 0x7f787c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f7881bd5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f787c111b40 con 0x7f787c06a780 2024-04-25T01:29:50.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f7881bd5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f787c1119c0 con 0x7f787c069dc0 2024-04-25T01:29:50.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f7881bd5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f787c111cc0 con 0x7f787c107730 2024-04-25T01:29:50.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f787affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c06a780 0x7f787c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f787affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c06a780 0x7f787c198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56318/0 (socket says 172.21.15.27:56318) 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.598+0000 7f787affd640 1 -- 172.21.15.27:0/216733257 learned_addr learned my addr 172.21.15.27:0/216733257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f787bfff640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f787c107730 0x7f787c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f787b7fe640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f787c069dc0 0x7f787c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f787affd640 1 -- 172.21.15.27:0/216733257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f787c107730 msgr2=0x7f787c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f787affd640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f787c107730 0x7f787c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f787affd640 1 -- 172.21.15.27:0/216733257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f787c069dc0 msgr2=0x7f787c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f787affd640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f787c069dc0 0x7f787c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f787affd640 1 -- 172.21.15.27:0/216733257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f787c19cfd0 con 0x7f787c06a780 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f787affd640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c06a780 0x7f787c198520 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f786c009230 tx=0x7f786c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f7878ff9640 1 -- 172.21.15.27:0/216733257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c03f030 con 0x7f787c06a780 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f7881bd5640 1 -- 172.21.15.27:0/216733257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f787c19d260 con 0x7f787c06a780 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f7878ff9640 1 -- 172.21.15.27:0/216733257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786c003ad0 con 0x7f787c06a780 2024-04-25T01:29:50.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f7878ff9640 1 -- 172.21.15.27:0/216733257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c035070 con 0x7f787c06a780 2024-04-25T01:29:50.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.599+0000 7f7881bd5640 1 -- 172.21.15.27:0/216733257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f787c1a4b50 con 0x7f787c06a780 2024-04-25T01:29:50.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.601+0000 7f7878ff9640 1 -- 172.21.15.27:0/216733257 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f786c042030 con 0x7f787c06a780 2024-04-25T01:29:50.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.601+0000 7f78849c4740 1 Processor -- start 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f78849c4740 1 -- start start 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f7878ff9640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78680a2d00 0x7f78680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f78849c4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d23f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f78849c4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d23fd80 0x560a6d24e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f78849c4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d24e980 0x560a6d250d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f78849c4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560a6d1a4670 con 0x560a6d23f460 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f78849c4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560a6d14c970 con 0x560a6d24e980 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f78849c4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560a6d1a0630 con 0x560a6d23fd80 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787bfff640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78680a2d00 0x7f78680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d23f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f7878ff9640 1 -- 172.21.15.27:0/216733257 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f786c0efd90 con 0x7f787c06a780 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d23f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56322/0 (socket says 172.21.15.27:56322) 2024-04-25T01:29:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 -- 172.21.15.27:0/386736087 learned_addr learned my addr 172.21.15.27:0/386736087 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787affd640 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d23fd80 0x560a6d24e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787bfff640 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d24e980 0x560a6d250d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 -- 172.21.15.27:0/386736087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d23fd80 msgr2=0x560a6d24e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d23fd80 0x560a6d24e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 -- 172.21.15.27:0/386736087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d24e980 msgr2=0x560a6d250d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d24e980 0x560a6d250d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 -- 172.21.15.27:0/386736087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6d170400 con 0x560a6d23f460 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787bfff640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78680a2d00 0x7f78680a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7874005fd0 tx=0x7f7874014ab0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.602+0000 7f787b7fe640 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d23f840 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7864009b30 tx=0x7f786400e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9975c2771e8fc600 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f7871ffb640 1 -- 172.21.15.27:0/386736087 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864016070 con 0x560a6d23f460 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f7871ffb640 1 -- 172.21.15.27:0/386736087 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786400ed20 con 0x560a6d23f460 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f7871ffb640 1 -- 172.21.15.27:0/386736087 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864010750 con 0x560a6d23f460 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 -- 172.21.15.27:0/386736087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 msgr2=0x560a6d23f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d23f840 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7864009b30 tx=0x7f786400e2f0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 -- 172.21.15.27:0/386736087 shutdown_connections 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d24e980 0x560a6d250d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d23fd80 0x560a6d24e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 --2- 172.21.15.27:0/386736087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d23f840 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 -- 172.21.15.27:0/386736087 >> 172.21.15.27:0/386736087 conn(0x560a6d23b8a0 msgr2=0x560a6d23bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 -- 172.21.15.27:0/386736087 shutdown_connections 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.603+0000 7f78849c4740 1 -- 172.21.15.27:0/386736087 wait complete. 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 1 -- start start 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 1 Processor -- start 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d24b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d24ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560a6d2517c0 con 0x560a6d23fd80 2024-04-25T01:29:50.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560a6d160050 con 0x560a6d23f460 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560a6d251940 con 0x560a6d24e980 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f78849c4740 10 client.0 _ll_register_callbacks cb 0x560a6ceb7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f787affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d24b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f787affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d24b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56338/0 (socket says 172.21.15.27:56338) 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f787affd640 1 -- 172.21.15.27:0/1860743217 learned_addr learned my addr 172.21.15.27:0/1860743217 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.604+0000 7f787bfff640 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d24ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f787b7fe640 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f787affd640 1 -- 172.21.15.27:0/1860743217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 msgr2=0x560a6d24ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f787affd640 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d24ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f787affd640 1 -- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 msgr2=0x560a6d24e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f787affd640 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f787affd640 1 -- 172.21.15.27:0/1860743217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6d131dc0 con 0x560a6d23fd80 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f787b7fe640 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f787affd640 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d24b530 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f786c0096c0 tx=0x7f786c0bb9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db4c766984cbf7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f785affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f785affd640 1 -- 172.21.15.27:0/1860743217 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c049040 con 0x560a6d23fd80 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f785affd640 1 -- 172.21.15.27:0/1860743217 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f786c03b460 con 0x560a6d23fd80 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f78849c4740 1 -- 172.21.15.27:0/1860743217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f786400ed20 con 0x560a6d23fd80 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f785affd640 1 -- 172.21.15.27:0/1860743217 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f786c0bbf20 con 0x560a6d23fd80 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f78849c4740 10 client.16617 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f78849c4740 1 -- 172.21.15.27:0/1860743217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560a6ceb83f0 con 0x560a6d23fd80 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f785affd640 1 -- 172.21.15.27:0/1860743217 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786400ed20 con 0x560a6d23fd80 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f785affd640 1 -- 172.21.15.27:0/1860743217 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f786c0bf020 con 0x560a6d23fd80 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f78849c4740 10 client.16617 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.605+0000 7f78849c4740 10 client.16617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.606+0000 7f78849c4740 20 client.16617 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.606+0000 7f78849c4740 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a6d25ea50 0x560a6d260ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.606+0000 7f78849c4740 4 client.16617 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.606+0000 7f78849c4740 1 -- 172.21.15.27:0/1860743217 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a6d1b1bb0 con 0x560a6d25ea50 2024-04-25T01:29:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.606+0000 7f787b7fe640 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a6d25ea50 0x560a6d260ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.606+0000 7f785affd640 10 client.16617 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.606+0000 7f787b7fe640 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a6d25ea50 0x560a6d260ee0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=531fd9dbd3b1e975 server_cookie=93b576670e0d51d4 in_seq=0 out_seq=0 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f785affd640 1 -- 172.21.15.27:0/1860743217 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a6d1b1bb0 con 0x560a6d25ea50 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f785affd640 10 client.16617 handle_command_reply: tid=0 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 client.16617 shutdown 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 -- 172.21.15.27:0/1860743217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 msgr2=0x560a6d24b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d24b530 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f786c0096c0 tx=0x7f786c0bb9b0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 -- 172.21.15.27:0/1860743217 shutdown_connections 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a6d25ea50 0x560a6d260ee0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d24ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d24b530 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f785affd640 0 client.16617 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 --2- 172.21.15.27:0/1860743217 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 -- 172.21.15.27:0/1860743217 >> 172.21.15.27:0/1860743217 conn(0x560a6d23b8a0 msgr2=0x560a6d23ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 -- 172.21.15.27:0/1860743217 shutdown_connections 2024-04-25T01:29:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.607+0000 7f78849c4740 1 -- 172.21.15.27:0/1860743217 wait complete. 2024-04-25T01:29:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.608+0000 7f78849c4740 20 client.16617 trim_cache size 0 max 16384 2024-04-25T01:29:50.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78849c4740 1 Processor -- start 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78849c4740 1 -- start start 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78849c4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78849c4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d266760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78849c4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d266ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78849c4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f787c1af130 con 0x560a6d23fd80 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78849c4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f787c1aca00 con 0x560a6d23f460 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78849c4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f787c1aa850 con 0x560a6d24e980 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d266760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d266760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56344/0 (socket says 172.21.15.27:56344) 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 -- 172.21.15.27:0/2551556244 learned_addr learned my addr 172.21.15.27:0/2551556244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787b7fe640 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787bfff640 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d266ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 -- 172.21.15.27:0/2551556244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 msgr2=0x560a6d266ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d266ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 -- 172.21.15.27:0/2551556244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 msgr2=0x560a6d24dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 -- 172.21.15.27:0/2551556244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6cf827d0 con 0x560a6d23fd80 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787b7fe640 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787bfff640 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d266ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f787affd640 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d266760 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f786c009a50 tx=0x7f786c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2857c2bd9205eff5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.621+0000 7f78717fa640 1 -- 172.21.15.27:0/2551556244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c0c4b90 con 0x560a6d23fd80 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78717fa640 1 -- 172.21.15.27:0/2551556244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786c0ba220 con 0x560a6d23fd80 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78717fa640 1 -- 172.21.15.27:0/2551556244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c084e50 con 0x560a6d23fd80 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 -- 172.21.15.27:0/2551556244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 msgr2=0x560a6d266760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d266760 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f786c009a50 tx=0x7f786c02a420 comp rx=0 tx=0).stop 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 -- 172.21.15.27:0/2551556244 shutdown_connections 2024-04-25T01:29:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d266ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23fd80 0x560a6d266760 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 --2- 172.21.15.27:0/2551556244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23f460 0x560a6d24dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 -- 172.21.15.27:0/2551556244 >> 172.21.15.27:0/2551556244 conn(0x560a6d23b8a0 msgr2=0x560a6d240d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 -- 172.21.15.27:0/2551556244 shutdown_connections 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 -- 172.21.15.27:0/2551556244 wait complete. 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.622+0000 7f78849c4740 1 -- start start 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f78849c4740 1 Processor -- start 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f78849c4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d252550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f78849c4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23fd80 0x560a6d252a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f78849c4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d252fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f78849c4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f787c1aaec0 con 0x560a6d23f460 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f78849c4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f787c1ac330 con 0x560a6d23fd80 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f78849c4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f787c1ab590 con 0x560a6d24e980 2024-04-25T01:29:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f78849c4740 10 client.0 _ll_register_callbacks cb 0x7f786c0b82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d252550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d252550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56352/0 (socket says 172.21.15.27:56352) 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 -- 172.21.15.27:0/3343914571 learned_addr learned my addr 172.21.15.27:0/3343914571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787affd640 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23fd80 0x560a6d252a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 -- 172.21.15.27:0/3343914571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 msgr2=0x560a6d252fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d252fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 -- 172.21.15.27:0/3343914571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23fd80 msgr2=0x560a6d252a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23fd80 0x560a6d252a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 -- 172.21.15.27:0/3343914571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6d2c3fd0 con 0x560a6d23f460 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f787b7fe640 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d252550 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f786400e7f0 tx=0x7f78640094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d0aca876ad706b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f785affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.623+0000 7f785affd640 1 -- 172.21.15.27:0/3343914571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786402e060 con 0x560a6d23f460 2024-04-25T01:29:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f785affd640 1 -- 172.21.15.27:0/3343914571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f78640035f0 con 0x560a6d23f460 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 1 -- 172.21.15.27:0/3343914571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f786c0ba220 con 0x560a6d23f460 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f785affd640 1 -- 172.21.15.27:0/3343914571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7864035050 con 0x560a6d23f460 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 10 client.16623 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 1 -- 172.21.15.27:0/3343914571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560a6d2c5140 con 0x560a6d23f460 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f785affd640 1 -- 172.21.15.27:0/3343914571 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786c0ba220 con 0x560a6d23f460 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f785affd640 1 -- 172.21.15.27:0/3343914571 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f786400f040 con 0x560a6d23f460 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 10 client.16623 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 10 client.16623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 20 client.16623 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a6d2c5610 0x560a6d2c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 4 client.16623 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f78849c4740 1 -- 172.21.15.27:0/3343914571 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a6d1abf00 con 0x560a6d2c5610 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.624+0000 7f787affd640 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a6d2c5610 0x560a6d2c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f785affd640 10 client.16623 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f787affd640 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a6d2c5610 0x560a6d2c7a50 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6de5477917db4cf2 server_cookie=a61e7bf79e4233a1 in_seq=0 out_seq=0 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f785affd640 1 -- 172.21.15.27:0/3343914571 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a6d1abf00 con 0x560a6d2c5610 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f785affd640 10 client.16623 handle_command_reply: tid=0 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 client.16623 shutdown 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 -- 172.21.15.27:0/3343914571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 msgr2=0x560a6d252550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d252550 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f786400e7f0 tx=0x7f78640094f0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 -- 172.21.15.27:0/3343914571 shutdown_connections 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a6d2c5610 0x560a6d2c7a50 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a6d24e980 0x560a6d252fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f785affd640 0 client.16623 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a6d23fd80 0x560a6d252a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 --2- 172.21.15.27:0/3343914571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a6d23f460 0x560a6d252550 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 -- 172.21.15.27:0/3343914571 >> 172.21.15.27:0/3343914571 conn(0x560a6d23b8a0 msgr2=0x560a6d2415e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 -- 172.21.15.27:0/3343914571 shutdown_connections 2024-04-25T01:29:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 1 -- 172.21.15.27:0/3343914571 wait complete. 2024-04-25T01:29:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.625+0000 7f78849c4740 20 client.16623 trim_cache size 0 max 16384 2024-04-25T01:29:50.611 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 -- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78680a2d00 msgr2=0x7f78680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78680a2d00 0x7f78680a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7874005fd0 tx=0x7f7874014ab0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 -- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c06a780 msgr2=0x7f787c198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c06a780 0x7f787c198520 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f786c009230 tx=0x7f786c004270 comp rx=0 tx=0).stop 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 -- 172.21.15.27:0/216733257 shutdown_connections 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f787c107730 0x7f787c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78680a2d00 0x7f78680a51c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f787c06a780 0x7f787c198520 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 --2- 172.21.15.27:0/216733257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f787c069dc0 0x7f787c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 -- 172.21.15.27:0/216733257 >> 172.21.15.27:0/216733257 conn(0x7f787c0776a0 msgr2=0x7f787c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 -- 172.21.15.27:0/216733257 shutdown_connections 2024-04-25T01:29:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:50.626+0000 7f7881bd5640 1 -- 172.21.15.27:0/216733257 wait complete. 2024-04-25T01:29:50.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:50 smithi153 ceph-mon[32621]: pgmap v135: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 9 op/s 2024-04-25T01:29:50.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:50 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1779395262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:51.616 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:29:51.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:29:51.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:29:51.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 -- 172.21.15.27:0/4041215625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 msgr2=0x7f6554101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 --2- 172.21.15.27:0/4041215625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 0x7f6554101b90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f653c009930 tx=0x7f653c02a420 comp rx=0 tx=0).stop 2024-04-25T01:29:51.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 -- 172.21.15.27:0/4041215625 shutdown_connections 2024-04-25T01:29:51.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 --2- 172.21.15.27:0/4041215625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65541020d0 0x7f655410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 --2- 172.21.15.27:0/4041215625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 0x7f6554101b90 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 --2- 172.21.15.27:0/4041215625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6554107720 0x7f6554107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 -- 172.21.15.27:0/4041215625 >> 172.21.15.27:0/4041215625 conn(0x7f65540fd180 msgr2=0x7f65540ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 -- 172.21.15.27:0/4041215625 shutdown_connections 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.757+0000 7f655af29640 1 -- 172.21.15.27:0/4041215625 wait complete. 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f655af29640 1 Processor -- start 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f655af29640 1 -- start start 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f655af29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 0x7f6554198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f655af29640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65541020d0 0x7f65541985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f655af29640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6554107720 0x7f655419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f655af29640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6554111b40 con 0x7f6554101710 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f655af29640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f65541119c0 con 0x7f65541020d0 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f655af29640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6554111cc0 con 0x7f6554107720 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6558cc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 0x7f6554198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6558cc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 0x7f6554198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56378/0 (socket says 172.21.15.27:56378) 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6558cc5640 1 -- 172.21.15.27:0/784489277 learned_addr learned my addr 172.21.15.27:0/784489277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6553fff640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65541020d0 0x7f65541985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6558cc5640 1 -- 172.21.15.27:0/784489277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6554107720 msgr2=0x7f655419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f65594c6640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6554107720 0x7f655419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6558cc5640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6554107720 0x7f655419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6558cc5640 1 -- 172.21.15.27:0/784489277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65541020d0 msgr2=0x7f65541985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6558cc5640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65541020d0 0x7f65541985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.758+0000 7f6558cc5640 1 -- 172.21.15.27:0/784489277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655419d050 con 0x7f6554101710 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.759+0000 7f6558cc5640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 0x7f6554198060 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f654400d950 tx=0x7f654400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.759+0000 7f6551ffb640 1 -- 172.21.15.27:0/784489277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544017030 con 0x7f6554101710 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.759+0000 7f6551ffb640 1 -- 172.21.15.27:0/784489277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6544004020 con 0x7f6554101710 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.759+0000 7f655af29640 1 -- 172.21.15.27:0/784489277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655419d340 con 0x7f6554101710 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.759+0000 7f6551ffb640 1 -- 172.21.15.27:0/784489277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65440047a0 con 0x7f6554101710 2024-04-25T01:29:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.759+0000 7f655af29640 1 -- 172.21.15.27:0/784489277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65541a4bd0 con 0x7f6554101710 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.761+0000 7f6551ffb640 1 -- 172.21.15.27:0/784489277 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f65440041d0 con 0x7f6554101710 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f655dc37740 1 Processor -- start 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f655dc37740 1 -- start start 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f6551ffb640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f65400a2d80 0x7f65400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f655dc37740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 0x55d07eec1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f655dc37740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eec2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f65594c6640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f65400a2d80 0x7f65400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f655dc37740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eecd010 0x55d07eecf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f6551ffb640 1 -- 172.21.15.27:0/784489277 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f65440cf1c0 con 0x7f6554101710 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f655dc37740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d07ee262c0 con 0x55d07eecd010 2024-04-25T01:29:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f655dc37740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d07edce910 con 0x55d07eec2290 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f655dc37740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d07ee22280 con 0x55d07eec3980 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f65594c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eecd010 0x55d07eecf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f65594c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eecd010 0x55d07eecf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56388/0 (socket says 172.21.15.27:56388) 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f6553fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eec2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f65594c6640 1 -- 172.21.15.27:0/4130143154 learned_addr learned my addr 172.21.15.27:0/4130143154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.762+0000 7f6558cc5640 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 0x55d07eec1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f65594c6640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f65400a2d80 0x7f65400a5240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f654c013670 tx=0x7f654c013a70 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f65594c6640 1 -- 172.21.15.27:0/4130143154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 msgr2=0x55d07eec1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f65594c6640 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 0x55d07eec1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f65594c6640 1 -- 172.21.15.27:0/4130143154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 msgr2=0x55d07eec2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f65594c6640 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eec2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f65594c6640 1 -- 172.21.15.27:0/4130143154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d07eddffb0 con 0x55d07eecd010 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f6553fff640 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eec2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f65594c6640 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eecd010 0x55d07eecf400 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f654c003c90 tx=0x7f654c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad458fd4aeb80fd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f654affd640 1 -- 172.21.15.27:0/4130143154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c004d20 con 0x55d07eecd010 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f654affd640 1 -- 172.21.15.27:0/4130143154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f654c005500 con 0x55d07eecd010 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.763+0000 7f654affd640 1 -- 172.21.15.27:0/4130143154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c01f8c0 con 0x55d07eecd010 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 -- 172.21.15.27:0/4130143154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eecd010 msgr2=0x55d07eecf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eecd010 0x55d07eecf400 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f654c003c90 tx=0x7f654c01b000 comp rx=0 tx=0).stop 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 -- 172.21.15.27:0/4130143154 shutdown_connections 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eecd010 0x55d07eecf400 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eec2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 --2- 172.21.15.27:0/4130143154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 0x55d07eec1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 -- 172.21.15.27:0/4130143154 >> 172.21.15.27:0/4130143154 conn(0x55d07ee8a950 msgr2=0x55d07eebe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 -- 172.21.15.27:0/4130143154 shutdown_connections 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 -- 172.21.15.27:0/4130143154 wait complete. 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 -- start start 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.764+0000 7f655dc37740 1 Processor -- start 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f655dc37740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eec9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f655dc37740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec3980 0x55d07eeca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f655dc37740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 0x55d07eeca990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f655dc37740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d07eed4250 con 0x55d07eec2290 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f655dc37740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d07ede1e90 con 0x55d07eec3980 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f655dc37740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d07eed43d0 con 0x55d07eecd010 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f655dc37740 10 client.0 _ll_register_callbacks cb 0x55d07ee1d970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eec9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eec9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56396/0 (socket says 172.21.15.27:56396) 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 -- 172.21.15.27:0/2792895867 learned_addr learned my addr 172.21.15.27:0/2792895867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6553fff640 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec3980 0x55d07eeca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f65594c6640 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 0x55d07eeca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 -- 172.21.15.27:0/2792895867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 msgr2=0x55d07eeca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 0x55d07eeca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 -- 172.21.15.27:0/2792895867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec3980 msgr2=0x55d07eeca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec3980 0x55d07eeca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 -- 172.21.15.27:0/2792895867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d07ec027e0 con 0x55d07eec2290 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6558cc5640 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eec9f10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6544097750 tx=0x7f654409ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bedb35ab5a0e98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6533fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6533fff640 1 -- 172.21.15.27:0/2792895867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544095e60 con 0x55d07eec2290 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f6533fff640 1 -- 172.21.15.27:0/2792895867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f65440a3ad0 con 0x55d07eec2290 2024-04-25T01:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.765+0000 7f655dc37740 1 -- 172.21.15.27:0/2792895867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f654c005500 con 0x55d07eec2290 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f6533fff640 1 -- 172.21.15.27:0/2792895867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f654409f6a0 con 0x55d07eec2290 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f655dc37740 10 client.16635 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f655dc37740 1 -- 172.21.15.27:0/2792895867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d07ec62f70 con 0x55d07eec2290 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f6533fff640 1 -- 172.21.15.27:0/2792895867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f654c005500 con 0x55d07eec2290 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f6533fff640 1 -- 172.21.15.27:0/2792895867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f65440bb0c0 con 0x55d07eec2290 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f655dc37740 10 client.16635 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f655dc37740 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f655dc37740 20 client.16635 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f655dc37740 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d07eee1460 0x55d07eee38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f655dc37740 4 client.16635 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f655dc37740 1 -- 172.21.15.27:0/2792895867 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d07edfa5a0 con 0x55d07eee1460 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.766+0000 7f6553fff640 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d07eee1460 0x55d07eee38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.767+0000 7f6553fff640 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d07eee1460 0x55d07eee38b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a30630114a6e046 server_cookie=87bff5e478535857 in_seq=0 out_seq=0 2024-04-25T01:29:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.767+0000 7f6533fff640 10 client.16635 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.767+0000 7f6533fff640 1 -- 172.21.15.27:0/2792895867 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d07edfa5a0 con 0x55d07eee1460 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.767+0000 7f6533fff640 10 client.16635 handle_command_reply: tid=0 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 client.16635 shutdown 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 -- 172.21.15.27:0/2792895867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 msgr2=0x55d07eec9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eec9f10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6544097750 tx=0x7f654409ba60 comp rx=0 tx=0).stop 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 -- 172.21.15.27:0/2792895867 shutdown_connections 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d07eee1460 0x55d07eee38b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 0x55d07eeca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec3980 0x55d07eeca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 --2- 172.21.15.27:0/2792895867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eec9f10 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f6533fff640 0 client.16635 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 -- 172.21.15.27:0/2792895867 >> 172.21.15.27:0/2792895867 conn(0x55d07ee8a950 msgr2=0x55d07eec4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 -- 172.21.15.27:0/2792895867 shutdown_connections 2024-04-25T01:29:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 1 -- 172.21.15.27:0/2792895867 wait complete. 2024-04-25T01:29:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.768+0000 7f655dc37740 20 client.16635 trim_cache size 0 max 16384 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f655dc37740 1 Processor -- start 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f655dc37740 1 -- start start 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f655dc37740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eedd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f655dc37740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 0x55d07eee9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f655dc37740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eecd010 0x55d07eee9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f655dc37740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65541af2d0 con 0x55d07eec2290 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f655dc37740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f65541acba0 con 0x55d07eecd010 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f655dc37740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65541aa9f0 con 0x55d07eec3980 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f6558cc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eedd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f6558cc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eedd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56398/0 (socket says 172.21.15.27:56398) 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f6558cc5640 1 -- 172.21.15.27:0/1901937045 learned_addr learned my addr 172.21.15.27:0/1901937045 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f6558cc5640 1 -- 172.21.15.27:0/1901937045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 msgr2=0x55d07eee9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f6558cc5640 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 0x55d07eee9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f65594c6640 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eecd010 0x55d07eee9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f6558cc5640 1 -- 172.21.15.27:0/1901937045 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eecd010 msgr2=0x55d07eee9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f6558cc5640 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eecd010 0x55d07eee9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f6558cc5640 1 -- 172.21.15.27:0/1901937045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d07eafa170 con 0x55d07eec2290 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.786+0000 7f65594c6640 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eecd010 0x55d07eee9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f6558cc5640 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eedd180 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f654409b280 tx=0x7f6544004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df88812bf792507f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f654a7fc640 1 -- 172.21.15.27:0/1901937045 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544095e60 con 0x55d07eec2290 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f654a7fc640 1 -- 172.21.15.27:0/1901937045 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65440a3f60 con 0x55d07eec2290 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f654a7fc640 1 -- 172.21.15.27:0/1901937045 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65440a3d30 con 0x55d07eec2290 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 -- 172.21.15.27:0/1901937045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 msgr2=0x55d07eedd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eedd180 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f654409b280 tx=0x7f6544004730 comp rx=0 tx=0).stop 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 -- 172.21.15.27:0/1901937045 shutdown_connections 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eecd010 0x55d07eee9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eec3980 0x55d07eee9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 --2- 172.21.15.27:0/1901937045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec2290 0x55d07eedd180 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 -- 172.21.15.27:0/1901937045 >> 172.21.15.27:0/1901937045 conn(0x55d07ee8a950 msgr2=0x55d07eebd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 -- 172.21.15.27:0/1901937045 shutdown_connections 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.787+0000 7f655dc37740 1 -- 172.21.15.27:0/1901937045 wait complete. 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 1 -- start start 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 1 Processor -- start 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eed9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec3980 0x55d07eed5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 0x55d07eed64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ab060 con 0x55d07eec3980 2024-04-25T01:29:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ac4d0 con 0x55d07eec2290 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ab730 con 0x55d07eecd010 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f655dc37740 10 client.0 _ll_register_callbacks cb 0x7f654409acc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f6553fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec3980 0x55d07eed5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f6553fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec3980 0x55d07eed5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56414/0 (socket says 172.21.15.27:56414) 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f6553fff640 1 -- 172.21.15.27:0/2894409771 learned_addr learned my addr 172.21.15.27:0/2894409771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.788+0000 7f6558cc5640 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eed9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f65594c6640 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 0x55d07eed64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6553fff640 1 -- 172.21.15.27:0/2894409771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 msgr2=0x55d07eed64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6553fff640 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 0x55d07eed64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6553fff640 1 -- 172.21.15.27:0/2894409771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 msgr2=0x55d07eed9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6553fff640 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eed9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6553fff640 1 -- 172.21.15.27:0/2894409771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d07eed6cb0 con 0x55d07eec3980 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6553fff640 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec3980 0x55d07eed5ea0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f653c02a460 tx=0x7f653c0043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c7ef970768f05f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6533fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6533fff640 1 -- 172.21.15.27:0/2894409771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653c035040 con 0x55d07eec3980 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6533fff640 1 -- 172.21.15.27:0/2894409771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f653c045dc0 con 0x55d07eec3980 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f655dc37740 1 -- 172.21.15.27:0/2894409771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65440a3f60 con 0x55d07eec3980 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6533fff640 1 -- 172.21.15.27:0/2894409771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f653c052050 con 0x55d07eec3980 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f655dc37740 10 client.16641 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f655dc37740 1 -- 172.21.15.27:0/2894409771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d07ef48ac0 con 0x55d07eec3980 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6533fff640 1 -- 172.21.15.27:0/2894409771 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65440a3f60 con 0x55d07eec3980 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f6533fff640 1 -- 172.21.15.27:0/2894409771 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f653c04d030 con 0x55d07eec3980 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f655dc37740 10 client.16641 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f655dc37740 10 client.16641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.789+0000 7f655dc37740 20 client.16641 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.790+0000 7f655dc37740 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d07ef49290 0x55d07ef4b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.790+0000 7f655dc37740 4 client.16641 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.790+0000 7f655dc37740 1 -- 172.21.15.27:0/2894409771 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d07ee33800 con 0x55d07ef49290 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.790+0000 7f6558cc5640 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d07ef49290 0x55d07ef4b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.790+0000 7f6558cc5640 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d07ef49290 0x55d07ef4b6b0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74c07cff3f9ca05c server_cookie=fa5a15d0f7381459 in_seq=0 out_seq=0 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.790+0000 7f6533fff640 10 client.16641 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f6533fff640 1 -- 172.21.15.27:0/2894409771 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d07ee33800 con 0x55d07ef49290 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f6533fff640 10 client.16641 handle_command_reply: tid=0 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 client.16641 shutdown 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 -- 172.21.15.27:0/2894409771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec3980 msgr2=0x55d07eed5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec3980 0x55d07eed5ea0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f653c02a460 tx=0x7f653c0043d0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 -- 172.21.15.27:0/2894409771 shutdown_connections 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d07ef49290 0x55d07ef4b6b0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d07eecd010 0x55d07eed64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d07eec3980 0x55d07eed5ea0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 --2- 172.21.15.27:0/2894409771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d07eec2290 0x55d07eed9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f655dc37740 1 -- 172.21.15.27:0/2894409771 >> 172.21.15.27:0/2894409771 conn(0x55d07ee8a950 msgr2=0x55d07eec3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.791+0000 7f6533fff640 0 client.16641 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:51.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655dc37740 1 -- 172.21.15.27:0/2894409771 shutdown_connections 2024-04-25T01:29:51.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655dc37740 1 -- 172.21.15.27:0/2894409771 wait complete. 2024-04-25T01:29:51.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655dc37740 20 client.16641 trim_cache size 0 max 16384 2024-04-25T01:29:51.777 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:29:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:29:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "6e668d7a-60dd-4e66-82d5-4a8b51b0d87c", 2024-04-25T01:29:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:29:51.777 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:29:51.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 -- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f65400a2d80 msgr2=0x7f65400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f65400a2d80 0x7f65400a5240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f654c013670 tx=0x7f654c013a70 comp rx=0 tx=0).stop 2024-04-25T01:29:51.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 -- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 msgr2=0x7f6554198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 0x7f6554198060 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f654400d950 tx=0x7f654400de10 comp rx=0 tx=0).stop 2024-04-25T01:29:51.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 -- 172.21.15.27:0/784489277 shutdown_connections 2024-04-25T01:29:51.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6554107720 0x7f655419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65541020d0 0x7f65541985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f65400a2d80 0x7f65400a5240 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 --2- 172.21.15.27:0/784489277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6554101710 0x7f6554198060 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 -- 172.21.15.27:0/784489277 >> 172.21.15.27:0/784489277 conn(0x7f65540fd180 msgr2=0x7f65540fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 -- 172.21.15.27:0/784489277 shutdown_connections 2024-04-25T01:29:51.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.792+0000 7f655af29640 1 -- 172.21.15.27:0/784489277 wait complete. 2024-04-25T01:29:51.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:29:51.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 -- 172.21.15.27:0/2785817769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 msgr2=0x7fc52810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 --2- 172.21.15.27:0/2785817769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 0x7fc52810d4f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc520009cd0 tx=0x7fc52002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 -- 172.21.15.27:0/2785817769 shutdown_connections 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 --2- 172.21.15.27:0/2785817769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 0x7fc52810d4f0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 --2- 172.21.15.27:0/2785817769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc528100620 0x7fc528100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 --2- 172.21.15.27:0/2785817769 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc528106630 0x7fc528106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 -- 172.21.15.27:0/2785817769 >> 172.21.15.27:0/2785817769 conn(0x7fc5280776a0 msgr2=0x7fc528077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 -- 172.21.15.27:0/2785817769 shutdown_connections 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.907+0000 7fc5305b6640 1 -- 172.21.15.27:0/2785817769 wait complete. 2024-04-25T01:29:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc5305b6640 1 Processor -- start 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc5305b6640 1 -- start start 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc5305b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc528100620 0x7fc52819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc5305b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 0x7fc52819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc5305b6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc528106630 0x7fc52819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc5305b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc528110a00 con 0x7fc528100fe0 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc5305b6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc528110880 con 0x7fc528106630 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc5305b6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc528110b80 con 0x7fc528100620 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52db51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 0x7fc52819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52db51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 0x7fc52819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56432/0 (socket says 172.21.15.27:56432) 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52db51640 1 -- 172.21.15.27:0/3506102911 learned_addr learned my addr 172.21.15.27:0/3506102911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52e352640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc528100620 0x7fc52819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52eb53640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc528106630 0x7fc52819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52db51640 1 -- 172.21.15.27:0/3506102911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc528100620 msgr2=0x7fc52819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52db51640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc528100620 0x7fc52819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52db51640 1 -- 172.21.15.27:0/3506102911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc528106630 msgr2=0x7fc52819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52db51640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc528106630 0x7fc52819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.908+0000 7fc52db51640 1 -- 172.21.15.27:0/3506102911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52819f290 con 0x7fc528100fe0 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.909+0000 7fc52db51640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 0x7fc52819a7e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc51800d950 tx=0x7fc51800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.909+0000 7fc51f7fe640 1 -- 172.21.15.27:0/3506102911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518017070 con 0x7fc528100fe0 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.909+0000 7fc5305b6640 1 -- 172.21.15.27:0/3506102911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc52819f580 con 0x7fc528100fe0 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.909+0000 7fc51f7fe640 1 -- 172.21.15.27:0/3506102911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc518009c30 con 0x7fc528100fe0 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.909+0000 7fc51f7fe640 1 -- 172.21.15.27:0/3506102911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51800f6b0 con 0x7fc528100fe0 2024-04-25T01:29:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.909+0000 7fc5305b6640 1 -- 172.21.15.27:0/3506102911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5281a6e10 con 0x7fc528100fe0 2024-04-25T01:29:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc51f7fe640 1 -- 172.21.15.27:0/3506102911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc5180109d0 con 0x7fc528100fe0 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc53339c740 1 Processor -- start 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc53339c740 1 -- start start 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc53339c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996eba3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc53339c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3f50 0x55996ebae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc53339c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996ebae620 0x55996ebb0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc53339c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55996eb086b0 con 0x55996eba3630 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc53339c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55996eab0880 con 0x55996eba3f50 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc53339c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55996eb04670 con 0x55996ebae620 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc52eb53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996eba3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc52eb53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996eba3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56448/0 (socket says 172.21.15.27:56448) 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc52eb53640 1 -- 172.21.15.27:0/2406022010 learned_addr learned my addr 172.21.15.27:0/2406022010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.912+0000 7fc51f7fe640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc5140a2d00 0x7fc5140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52e352640 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3f50 0x55996ebae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52db51640 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996ebae620 0x55996ebb0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52eb53640 1 -- 172.21.15.27:0/2406022010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996ebae620 msgr2=0x55996ebb0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52eb53640 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996ebae620 0x55996ebb0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52eb53640 1 -- 172.21.15.27:0/2406022010 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3f50 msgr2=0x55996ebae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52eb53640 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3f50 0x55996ebae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52eb53640 1 -- 172.21.15.27:0/2406022010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55996e8e67d0 con 0x55996eba3630 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc51f7fe640 1 -- 172.21.15.27:0/3506102911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc5180cf430 con 0x7fc528100fe0 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52eb53640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc5140a2d00 0x7fc5140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc52eb53640 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996eba3a10 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc520005cc0 tx=0x7fc520031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0578d8bc6f7b4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc51cbfe640 1 -- 172.21.15.27:0/2406022010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520031c80 con 0x55996eba3630 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc51cbfe640 1 -- 172.21.15.27:0/2406022010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc520036040 con 0x55996eba3630 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.913+0000 7fc51cbfe640 1 -- 172.21.15.27:0/2406022010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5200307a0 con 0x55996eba3630 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 -- 172.21.15.27:0/2406022010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 msgr2=0x55996eba3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc52eb53640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc5140a2d00 0x7fc5140a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc52004b000 tx=0x7fc520048000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996eba3a10 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc520005cc0 tx=0x7fc520031630 comp rx=0 tx=0).stop 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 -- 172.21.15.27:0/2406022010 shutdown_connections 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996ebae620 0x55996ebb0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3f50 0x55996ebae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 --2- 172.21.15.27:0/2406022010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996eba3a10 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 -- 172.21.15.27:0/2406022010 >> 172.21.15.27:0/2406022010 conn(0x55996eb9f950 msgr2=0x55996eb9fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 -- 172.21.15.27:0/2406022010 shutdown_connections 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 -- 172.21.15.27:0/2406022010 wait complete. 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.914+0000 7fc53339c740 1 -- start start 2024-04-25T01:29:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc53339c740 1 Processor -- start 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc53339c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3630 0x55996ebab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc53339c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3f50 0x55996ebabb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc53339c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996ebac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc53339c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55996ebb1410 con 0x55996eba3f50 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc53339c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55996eac3f50 con 0x55996ebae620 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc53339c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55996ebb1590 con 0x55996eba3630 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc53339c740 10 client.0 _ll_register_callbacks cb 0x55996eaff000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3f50 0x55996ebabb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3f50 0x55996ebabb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56452/0 (socket says 172.21.15.27:56452) 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 -- 172.21.15.27:0/3212457183 learned_addr learned my addr 172.21.15.27:0/3212457183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52e352640 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3630 0x55996ebab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52eb53640 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996ebac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 -- 172.21.15.27:0/3212457183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3630 msgr2=0x55996ebab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3630 0x55996ebab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 -- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 msgr2=0x55996ebac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996ebac040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 -- 172.21.15.27:0/3212457183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55996e81c3f0 con 0x55996eba3f50 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52eb53640 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996ebac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc52db51640 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3f50 0x55996ebabb00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc5180cc850 tx=0x7fc518000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27eb81390d68821 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.915+0000 7fc511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc511ffb640 1 -- 172.21.15.27:0/3212457183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518020040 con 0x55996eba3f50 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc511ffb640 1 -- 172.21.15.27:0/3212457183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc5180cca70 con 0x55996eba3f50 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 1 -- 172.21.15.27:0/3212457183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc520036040 con 0x55996eba3f50 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc511ffb640 1 -- 172.21.15.27:0/3212457183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc5180ccc70 con 0x55996eba3f50 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 10 client.16653 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 1 -- 172.21.15.27:0/3212457183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55996e7dc0f0 con 0x55996eba3f50 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc511ffb640 1 -- 172.21.15.27:0/3212457183 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc520036040 con 0x55996eba3f50 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc511ffb640 1 -- 172.21.15.27:0/3212457183 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc51809aa10 con 0x55996eba3f50 2024-04-25T01:29:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 10 client.16653 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 20 client.16653 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55996ebbe8a0 0x55996ebc0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 4 client.16653 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc53339c740 1 -- 172.21.15.27:0/3212457183 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55996eb15bf0 con 0x55996ebbe8a0 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.916+0000 7fc52e352640 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55996ebbe8a0 0x55996ebc0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.917+0000 7fc52e352640 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55996ebbe8a0 0x55996ebc0d00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=111b35425e1ebfec server_cookie=8688f00cf66e604d in_seq=0 out_seq=0 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.917+0000 7fc511ffb640 10 client.16653 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.917+0000 7fc511ffb640 1 -- 172.21.15.27:0/3212457183 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55996eb15bf0 con 0x55996ebbe8a0 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.917+0000 7fc511ffb640 10 client.16653 handle_command_reply: tid=0 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.917+0000 7fc53339c740 1 client.16653 shutdown 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 -- 172.21.15.27:0/3212457183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3f50 msgr2=0x55996ebabb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3f50 0x55996ebabb00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc5180cc850 tx=0x7fc518000bc0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 -- 172.21.15.27:0/3212457183 shutdown_connections 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55996ebbe8a0 0x55996ebc0d00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996ebac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc511ffb640 0 client.16653 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3f50 0x55996ebabb00 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 --2- 172.21.15.27:0/3212457183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3630 0x55996ebab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 -- 172.21.15.27:0/3212457183 >> 172.21.15.27:0/3212457183 conn(0x55996eb9f950 msgr2=0x55996eba5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 -- 172.21.15.27:0/3212457183 shutdown_connections 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 1 -- 172.21.15.27:0/3212457183 wait complete. 2024-04-25T01:29:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.918+0000 7fc53339c740 20 client.16653 trim_cache size 0 max 16384 2024-04-25T01:29:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc53339c740 1 Processor -- start 2024-04-25T01:29:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc53339c740 1 -- start start 2024-04-25T01:29:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc53339c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996ebba2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc53339c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 0x55996eb9a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc53339c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996eba5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc53339c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc5281b35f0 con 0x55996eba3630 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc53339c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5281b0ec0 con 0x55996ebae620 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc53339c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5281aed10 con 0x55996eba3f50 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc52e352640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996ebba2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc52e352640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996ebba2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56458/0 (socket says 172.21.15.27:56458) 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.935+0000 7fc52e352640 1 -- 172.21.15.27:0/3289188728 learned_addr learned my addr 172.21.15.27:0/3289188728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52db51640 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 0x55996eb9a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52eb53640 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996eba5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52e352640 1 -- 172.21.15.27:0/3289188728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 msgr2=0x55996eb9a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52e352640 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 0x55996eb9a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52e352640 1 -- 172.21.15.27:0/3289188728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 msgr2=0x55996eba5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52e352640 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996eba5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52e352640 1 -- 172.21.15.27:0/3289188728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55996ebaa570 con 0x55996eba3630 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52eb53640 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996eba5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc52e352640 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996ebba2d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc524004700 tx=0x7fc524013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06a17412893ad86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc513fff640 1 -- 172.21.15.27:0/3289188728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524013110 con 0x55996eba3630 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc513fff640 1 -- 172.21.15.27:0/3289188728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5240132b0 con 0x55996eba3630 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.936+0000 7fc513fff640 1 -- 172.21.15.27:0/3289188728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524004a90 con 0x55996eba3630 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- 172.21.15.27:0/3289188728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 msgr2=0x55996ebba2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996ebba2d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc524004700 tx=0x7fc524013830 comp rx=0 tx=0).stop 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- 172.21.15.27:0/3289188728 shutdown_connections 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996ebae620 0x55996eba5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 0x55996eb9a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 --2- 172.21.15.27:0/3289188728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996eba3630 0x55996ebba2d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- 172.21.15.27:0/3289188728 >> 172.21.15.27:0/3289188728 conn(0x55996eb9f950 msgr2=0x55996ebad570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- 172.21.15.27:0/3289188728 shutdown_connections 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- 172.21.15.27:0/3289188728 wait complete. 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- start start 2024-04-25T01:29:51.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 Processor -- start 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3630 0x55996eba8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 0x55996eba8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996ebae620 0x55996ec23df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc5281af380 con 0x55996ebae620 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc5281b07f0 con 0x55996eba3630 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5281afa50 con 0x55996eba3f50 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc53339c740 10 client.0 _ll_register_callbacks cb 0x7fc51800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc52eb53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996ebae620 0x55996ec23df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc52eb53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996ebae620 0x55996ec23df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56474/0 (socket says 172.21.15.27:56474) 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.937+0000 7fc52eb53640 1 -- 172.21.15.27:0/2677121151 learned_addr learned my addr 172.21.15.27:0/2677121151 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52db51640 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 0x55996eba8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52e352640 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3630 0x55996eba8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52eb53640 1 -- 172.21.15.27:0/2677121151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 msgr2=0x55996eba8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52eb53640 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 0x55996eba8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52eb53640 1 -- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3630 msgr2=0x55996eba8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52eb53640 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3630 0x55996eba8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52eb53640 1 -- 172.21.15.27:0/2677121151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55996ec24330 con 0x55996ebae620 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52e352640 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3630 0x55996eba8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc52eb53640 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996ebae620 0x55996ec23df0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc52002ae00 tx=0x7fc520009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95dd41d28aa6239d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc511ffb640 1 -- 172.21.15.27:0/2677121151 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5200318c0 con 0x55996ebae620 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc511ffb640 1 -- 172.21.15.27:0/2677121151 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc52004d050 con 0x55996ebae620 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc53339c740 1 -- 172.21.15.27:0/2677121151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5240132b0 con 0x55996ebae620 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc511ffb640 1 -- 172.21.15.27:0/2677121151 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc520060050 con 0x55996ebae620 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc53339c740 10 client.16659 fetch_fsmap learned FSMap version 27 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc53339c740 1 -- 172.21.15.27:0/2677121151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55996ec25560 con 0x55996ebae620 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.938+0000 7fc511ffb640 1 -- 172.21.15.27:0/2677121151 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5240132b0 con 0x55996ebae620 2024-04-25T01:29:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc511ffb640 1 -- 172.21.15.27:0/2677121151 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc520030070 con 0x55996ebae620 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc53339c740 10 client.16659 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc53339c740 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc53339c740 20 client.16659 populate_metadata read hostname 'smithi027' 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc53339c740 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55996ec25840 0x55996ec27c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc53339c740 4 client.16659 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc53339c740 1 -- 172.21.15.27:0/2677121151 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55996eb0ff40 con 0x55996ec25840 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc52e352640 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55996ec25840 0x55996ec27c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc511ffb640 10 client.16659 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.939+0000 7fc52e352640 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55996ec25840 0x55996ec27c30 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd18d737df52ad58 server_cookie=125a77a8996224af in_seq=0 out_seq=0 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc511ffb640 1 -- 172.21.15.27:0/2677121151 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55996eb0ff40 con 0x55996ec25840 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc511ffb640 10 client.16659 handle_command_reply: tid=0 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 client.16659 shutdown 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 -- 172.21.15.27:0/2677121151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996ebae620 msgr2=0x55996ec23df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996ebae620 0x55996ec23df0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc52002ae00 tx=0x7fc520009560 comp rx=0 tx=0).stop 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 -- 172.21.15.27:0/2677121151 shutdown_connections 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55996ec25840 0x55996ec27c30 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55996ebae620 0x55996ec23df0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55996eba3f50 0x55996eba8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 --2- 172.21.15.27:0/2677121151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55996eba3630 0x55996eba8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 -- 172.21.15.27:0/2677121151 >> 172.21.15.27:0/2677121151 conn(0x55996eb9f950 msgr2=0x55996ebacbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc511ffb640 0 client.16659 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 -- 172.21.15.27:0/2677121151 shutdown_connections 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 1 -- 172.21.15.27:0/2677121151 wait complete. 2024-04-25T01:29:51.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc53339c740 20 client.16659 trim_cache size 0 max 16384 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "6e668d7a-60dd-4e66-82d5-4a8b51b0d87c": { 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "6e668d7a-60dd-4e66-82d5-4a8b51b0d87c", 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:29:51.927 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 -- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc5140a2d00 msgr2=0x7fc5140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc5140a2d00 0x7fc5140a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc52004b000 tx=0x7fc520048000 comp rx=0 tx=0).stop 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 -- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 msgr2=0x7fc52819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 0x7fc52819a7e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc51800d950 tx=0x7fc51800de10 comp rx=0 tx=0).stop 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 -- 172.21.15.27:0/3506102911 shutdown_connections 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc528106630 0x7fc52819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc5140a2d00 0x7fc5140a51c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc528100fe0 0x7fc52819a7e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 --2- 172.21.15.27:0/3506102911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc528100620 0x7fc52819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 -- 172.21.15.27:0/3506102911 >> 172.21.15.27:0/3506102911 conn(0x7fc5280776a0 msgr2=0x7fc52810bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.940+0000 7fc5305b6640 1 -- 172.21.15.27:0/3506102911 shutdown_connections 2024-04-25T01:29:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:29:51.941+0000 7fc5305b6640 1 -- 172.21.15.27:0/3506102911 wait complete. 2024-04-25T01:29:51.930 INFO:tasks.cephfs.filesystem:scrub status for tag:6e668d7a-60dd-4e66-82d5-4a8b51b0d87c - {'path': '/', 'tag': '6e668d7a-60dd-4e66-82d5-4a8b51b0d87c', 'options': 'recursive,force'} 2024-04-25T01:29:52.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:52 smithi139 ceph-mon[32760]: pgmap v136: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:52.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:52 smithi027 ceph-mon[29197]: pgmap v136: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:52.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:52 smithi153 ceph-mon[32621]: pgmap v136: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:53.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:53 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:29:53.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:53 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:53.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:53 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:29:53.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:53 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:29:53.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:53 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:53.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:53 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:53.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:53 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:29:53.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:53 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:53.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:53 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:29:53.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:53 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:29:53.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:53 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:53.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:53 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:53.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:53 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:29:53.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:53 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:53.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:53 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:29:53.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:53 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:29:53.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:53 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:53.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:53 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:29:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:54 smithi027 ceph-mon[29197]: pgmap v137: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:29:54.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:54 smithi153 ceph-mon[32621]: pgmap v137: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:29:54.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:54 smithi139 ceph-mon[32760]: pgmap v137: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:29:55.058 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.058 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.058 INFO:tasks.workunit.client.0.smithi027.stdout:127/600: open fd = 3 2024-04-25T01:29:55.058 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.058 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.058 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.058 INFO:tasks.workunit.client.0.smithi027.stdout:128/600: open fd = 3 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:129/600: open fd = 3 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:130/600: open fd = 3 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:131/600: open fd = 3 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.059 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:132/600: open fd = 3 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:133/600: open fd = 3 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:134/600: open fd = 3 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:135/600: open fd = 3 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.060 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:136/600: open fd = 3 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:137/600: open fd = 3 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:138/600: open fd = 3 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:139/600: open fd = 3 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.061 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:140/600: open fd = 3 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:141/600: open fd = 3 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:142/600: open fd = 3 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.062 INFO:tasks.workunit.client.0.smithi027.stdout:143/600: open fd = 3 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:144/600: open fd = 3 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:145/600: open fd = 3 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:146/600: open fd = 3 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.063 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:147/600: open fd = 3 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:148/600: open fd = 3 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:149/600: open fd = 3 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:150/600: open fd = 3 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.064 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:151/600: open fd = 3 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:152/600: open fd = 3 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:153/600: open fd = 3 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:154/600: open fd = 3 2024-04-25T01:29:55.065 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:155/600: open fd = 3 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:156/600: open fd = 3 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:157/600: open fd = 3 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.066 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:158/600: open fd = 3 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:159/600: open fd = 3 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:160/600: open fd = 3 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:161/600: open fd = 3 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.067 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:162/600: open fd = 3 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:163/600: open fd = 3 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:164/600: open fd = 3 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.068 INFO:tasks.workunit.client.0.smithi027.stdout:165/600: open fd = 3 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:166/600: open fd = 3 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:167/600: open fd = 3 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:168/600: open fd = 3 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.069 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:169/600: open fd = 3 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:170/600: open fd = 3 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:171/600: open fd = 3 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:172/600: open fd = 3 2024-04-25T01:29:55.070 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:173/600: open fd = 3 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:174/600: open fd = 3 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:175/600: open fd = 3 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.071 INFO:tasks.workunit.client.0.smithi027.stdout:176/600: open fd = 3 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:177/600: open fd = 3 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:178/600: open fd = 3 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:179/600: open fd = 3 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.072 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:180/600: open fd = 3 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:181/600: open fd = 3 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:182/600: open fd = 3 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:183/600: open fd = 3 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.073 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:184/600: open fd = 3 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:185/600: open fd = 3 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:186/600: open fd = 3 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:187/600: open fd = 3 2024-04-25T01:29:55.074 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.075 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.075 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:29:55.075 INFO:tasks.workunit.client.0.smithi027.stdout:188/600: open fd = 3 2024-04-25T01:29:55.075 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:55.075 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:29:56.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:56 smithi027 ceph-mon[29197]: pgmap v138: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:29:56.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:56 smithi153 ceph-mon[32621]: pgmap v138: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:29:56.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:56 smithi139 ceph-mon[32760]: pgmap v138: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2024-04-25T01:29:58.576 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:58 smithi153 ceph-mon[32621]: pgmap v139: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:58.577 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:58 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:29:58.577 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:29:58 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:29:58.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:58 smithi027 ceph-mon[29197]: pgmap v139: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:58.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:58 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:29:58.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:29:58 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:29:58.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:58 smithi139 ceph-mon[32760]: pgmap v139: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:29:58.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:58 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:29:58.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:29:58 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:30:00.592 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:00 smithi139 ceph-mon[32760]: pgmap v140: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.4 KiB/s wr, 9 op/s 2024-04-25T01:30:00.592 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:00 smithi139 ceph-mon[32760]: overall HEALTH_OK 2024-04-25T01:30:00.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:00 smithi027 ceph-mon[29197]: pgmap v140: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.4 KiB/s wr, 9 op/s 2024-04-25T01:30:00.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:00 smithi027 ceph-mon[29197]: overall HEALTH_OK 2024-04-25T01:30:00.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:00 smithi153 ceph-mon[32621]: pgmap v140: 129 pgs: 129 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.4 KiB/s wr, 9 op/s 2024-04-25T01:30:00.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:00 smithi153 ceph-mon[32621]: overall HEALTH_OK 2024-04-25T01:30:02.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:02 smithi027 ceph-mon[29197]: pgmap v141: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:30:02.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:02 smithi153 ceph-mon[32621]: pgmap v141: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:30:02.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:02 smithi139 ceph-mon[32760]: pgmap v141: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:30:04.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:04 smithi027 ceph-mon[29197]: pgmap v142: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 24 op/s 2024-04-25T01:30:04.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:04 smithi153 ceph-mon[32621]: pgmap v142: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 24 op/s 2024-04-25T01:30:04.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:04 smithi139 ceph-mon[32760]: pgmap v142: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 24 op/s 2024-04-25T01:30:06.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:06 smithi027 ceph-mon[29197]: pgmap v143: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 24 op/s 2024-04-25T01:30:06.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:06 smithi153 ceph-mon[32621]: pgmap v143: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 24 op/s 2024-04-25T01:30:06.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:06 smithi139 ceph-mon[32760]: pgmap v143: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 24 op/s 2024-04-25T01:30:08.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:08 smithi027 ceph-mon[29197]: pgmap v144: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:30:08.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:08 smithi153 ceph-mon[32621]: pgmap v144: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:30:08.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:08 smithi139 ceph-mon[32760]: pgmap v144: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:30:10.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:10 smithi027 ceph-mon[29197]: pgmap v145: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-25T01:30:10.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:10 smithi153 ceph-mon[32621]: pgmap v145: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-25T01:30:10.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:10 smithi139 ceph-mon[32760]: pgmap v145: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-25T01:30:12.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:12 smithi027 ceph-mon[29197]: pgmap v146: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:30:12.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:12 smithi153 ceph-mon[32621]: pgmap v146: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:30:12.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:12 smithi139 ceph-mon[32760]: pgmap v146: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:30:14.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:14 smithi027 ceph-mon[29197]: pgmap v147: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-25T01:30:14.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:14 smithi153 ceph-mon[32621]: pgmap v147: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-25T01:30:14.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:14 smithi139 ceph-mon[32760]: pgmap v147: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-25T01:30:16.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:16 smithi027 ceph-mon[29197]: pgmap v148: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-25T01:30:16.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:16 smithi153 ceph-mon[32621]: pgmap v148: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-25T01:30:16.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:16 smithi139 ceph-mon[32760]: pgmap v148: 129 pgs: 129 active+clean; 2.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-25T01:30:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:18 smithi027 ceph-mon[29197]: pgmap v149: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:30:18.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:18 smithi153 ceph-mon[32621]: pgmap v149: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:30:18.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:18 smithi139 ceph-mon[32760]: pgmap v149: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:30:20.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:20 smithi027 ceph-mon[29197]: pgmap v150: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:20.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:20 smithi153 ceph-mon[32621]: pgmap v150: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:20.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:20 smithi139 ceph-mon[32760]: pgmap v150: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:21.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:30:21.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.072+0000 7f811c922640 1 -- 172.21.15.27:0/1327166249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 msgr2=0x7f8118103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.072+0000 7f811c922640 1 --2- 172.21.15.27:0/1327166249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 0x7f8118103e00 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f810c009930 tx=0x7f810c02a420 comp rx=0 tx=0).stop 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 -- 172.21.15.27:0/1327166249 shutdown_connections 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 --2- 172.21.15.27:0/1327166249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8118104340 0x7f8118110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 --2- 172.21.15.27:0/1327166249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 0x7f8118103e00 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 --2- 172.21.15.27:0/1327166249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8118109990 0x7f8118109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 -- 172.21.15.27:0/1327166249 >> 172.21.15.27:0/1327166249 conn(0x7f81180ff4d0 msgr2=0x7f81181018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 -- 172.21.15.27:0/1327166249 shutdown_connections 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 -- 172.21.15.27:0/1327166249 wait complete. 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 Processor -- start 2024-04-25T01:30:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.073+0000 7f811c922640 1 -- start start 2024-04-25T01:30:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811c922640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 0x7f811819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811c922640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8118104340 0x7f811819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811c922640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8118109990 0x7f811819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811c922640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8118113d70 con 0x7f8118103980 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811c922640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8118113bf0 con 0x7f8118104340 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811c922640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8118113ef0 con 0x7f8118109990 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 0x7f811819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 0x7f811819a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56106/0 (socket says 172.21.15.27:56106) 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 -- 172.21.15.27:0/442298919 learned_addr learned my addr 172.21.15.27:0/442298919 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f8115d9b640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8118104340 0x7f811819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f8116d9d640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8118109990 0x7f811819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 -- 172.21.15.27:0/442298919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8118109990 msgr2=0x7f811819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8118109990 0x7f811819ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 -- 172.21.15.27:0/442298919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8118104340 msgr2=0x7f811819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8118104340 0x7f811819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 -- 172.21.15.27:0/442298919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811819f2f0 con 0x7f8118103980 2024-04-25T01:30:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f811659c640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 0x7f811819a300 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f810800d950 tx=0x7f810800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.074+0000 7f81077fe640 1 -- 172.21.15.27:0/442298919 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108017030 con 0x7f8118103980 2024-04-25T01:30:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.075+0000 7f81077fe640 1 -- 172.21.15.27:0/442298919 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8108004020 con 0x7f8118103980 2024-04-25T01:30:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.075+0000 7f811c922640 1 -- 172.21.15.27:0/442298919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811819f5e0 con 0x7f8118103980 2024-04-25T01:30:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.075+0000 7f81077fe640 1 -- 172.21.15.27:0/442298919 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81080047a0 con 0x7f8118103980 2024-04-25T01:30:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.075+0000 7f811c922640 1 -- 172.21.15.27:0/442298919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81181a6e70 con 0x7f8118103980 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.077+0000 7f81077fe640 1 -- 172.21.15.27:0/442298919 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f81080041d0 con 0x7f8118103980 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.077+0000 7f811f632740 1 Processor -- start 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811f632740 1 -- start start 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f81077fe640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f81000a2d00 0x7f81000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811f632740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44630 0x557882f44a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811f632740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811f632740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f4f620 0x557882f51a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811f632740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557882ea96b0 con 0x557882f44630 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811f632740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557882e51880 con 0x557882f4f620 2024-04-25T01:30:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811f632740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557882ea5670 con 0x557882f44f50 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f8116d9d640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f81000a2d00 0x7f81000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f81077fe640 1 -- 172.21.15.27:0/442298919 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f81080cf130 con 0x7f8118103980 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44630 0x557882f44a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44630 0x557882f44a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56120/0 (socket says 172.21.15.27:56120) 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 -- 172.21.15.27:0/959638720 learned_addr learned my addr 172.21.15.27:0/959638720 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f8116d9d640 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f4f620 0x557882f51a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f8115d9b640 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 -- 172.21.15.27:0/959638720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 msgr2=0x557882f4f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 -- 172.21.15.27:0/959638720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f4f620 msgr2=0x557882f51a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f4f620 0x557882f51a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 -- 172.21.15.27:0/959638720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557882c877d0 con 0x557882f44630 2024-04-25T01:30:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f8116d9d640 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f4f620 0x557882f51a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f8116d9d640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f81000a2d00 0x7f81000a51c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8110005fd0 tx=0x7f81100139c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.078+0000 7f811659c640 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44630 0x557882f44a10 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f81080a3a00 tx=0x7f810809aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25974d0f4a2b9d31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f8104ff9640 1 -- 172.21.15.27:0/959638720 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810809b2c0 con 0x557882f44630 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f8104ff9640 1 -- 172.21.15.27:0/959638720 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f810809f670 con 0x557882f44630 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f8104ff9640 1 -- 172.21.15.27:0/959638720 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108097930 con 0x557882f44630 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f811f632740 1 -- 172.21.15.27:0/959638720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44630 msgr2=0x557882f44a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f811f632740 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44630 0x557882f44a10 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f81080a3a00 tx=0x7f810809aec0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f811f632740 1 -- 172.21.15.27:0/959638720 shutdown_connections 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f811f632740 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f4f620 0x557882f51a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f811f632740 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f811f632740 1 --2- 172.21.15.27:0/959638720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44630 0x557882f44a10 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.079+0000 7f811f632740 1 -- 172.21.15.27:0/959638720 >> 172.21.15.27:0/959638720 conn(0x557882f40950 msgr2=0x557882f40d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 -- 172.21.15.27:0/959638720 shutdown_connections 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 -- 172.21.15.27:0/959638720 wait complete. 2024-04-25T01:30:22.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 -- start start 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 Processor -- start 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f4c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f4f620 0x557882f4d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557882f52410 con 0x557882f4f620 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557882e64f50 con 0x557882f44630 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557882f52590 con 0x557882f44f50 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f811f632740 10 client.0 _ll_register_callbacks cb 0x557882ea0000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f8116d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f4f620 0x557882f4d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f8116d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f4f620 0x557882f4d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56124/0 (socket says 172.21.15.27:56124) 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.080+0000 7f8116d9d640 1 -- 172.21.15.27:0/1466296132 learned_addr learned my addr 172.21.15.27:0/1466296132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f8115d9b640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f811659c640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f4c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f8116d9d640 1 -- 172.21.15.27:0/1466296132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 msgr2=0x557882f4cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f8116d9d640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f8116d9d640 1 -- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 msgr2=0x557882f4c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f8116d9d640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f4c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f8116d9d640 1 -- 172.21.15.27:0/1466296132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557882bbd3f0 con 0x557882f4f620 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f8115d9b640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f811659c640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f4c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f8116d9d640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f4f620 0x557882f4d070 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8110011810 tx=0x7f8110004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd1dc8722838652 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f80fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f80fdffb640 1 -- 172.21.15.27:0/1466296132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110024030 con 0x557882f4f620 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f80fdffb640 1 -- 172.21.15.27:0/1466296132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8110004b20 con 0x557882f4f620 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f811f632740 1 -- 172.21.15.27:0/1466296132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f810809f670 con 0x557882f4f620 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f80fdffb640 1 -- 172.21.15.27:0/1466296132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8110030050 con 0x557882f4f620 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f811f632740 10 client.16671 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f811f632740 1 -- 172.21.15.27:0/1466296132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557882b7d0f0 con 0x557882f4f620 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.081+0000 7f80fdffb640 1 -- 172.21.15.27:0/1466296132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f810809f670 con 0x557882f4f620 2024-04-25T01:30:22.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f80fdffb640 1 -- 172.21.15.27:0/1466296132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8110005470 con 0x557882f4f620 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f811f632740 10 client.16671 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f811f632740 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f811f632740 20 client.16671 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f811f632740 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557882f5f8e0 0x557882f61d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f811f632740 4 client.16671 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f811f632740 1 -- 172.21.15.27:0/1466296132 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557882eb6bf0 con 0x557882f5f8e0 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f811659c640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557882f5f8e0 0x557882f61d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f811659c640 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557882f5f8e0 0x557882f61d60 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7de7d6a0080486f server_cookie=8ec22a1b1fae5ac8 in_seq=0 out_seq=0 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.082+0000 7f80fdffb640 10 client.16671 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f80fdffb640 1 -- 172.21.15.27:0/1466296132 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557882eb6bf0 con 0x557882f5f8e0 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f80fdffb640 10 client.16671 handle_command_reply: tid=0 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 client.16671 shutdown 2024-04-25T01:30:22.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 -- 172.21.15.27:0/1466296132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f4f620 msgr2=0x557882f4d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f4f620 0x557882f4d070 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8110011810 tx=0x7f8110004ab0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 -- 172.21.15.27:0/1466296132 shutdown_connections 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557882f5f8e0 0x557882f61d60 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f4f620 0x557882f4d070 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44f50 0x557882f4cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 --2- 172.21.15.27:0/1466296132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f4c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f80fdffb640 0 client.16671 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.083+0000 7f811f632740 1 -- 172.21.15.27:0/1466296132 >> 172.21.15.27:0/1466296132 conn(0x557882f40950 msgr2=0x557882f46650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.084+0000 7f811f632740 1 -- 172.21.15.27:0/1466296132 shutdown_connections 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.084+0000 7f811f632740 1 -- 172.21.15.27:0/1466296132 wait complete. 2024-04-25T01:30:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.084+0000 7f811f632740 20 client.16671 trim_cache size 0 max 16384 2024-04-25T01:30:22.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.101+0000 7f811f632740 1 Processor -- start 2024-04-25T01:30:22.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.101+0000 7f811f632740 1 -- start start 2024-04-25T01:30:22.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.101+0000 7f811f632740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f40950 0x557882f5b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.101+0000 7f811f632740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f46d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.101+0000 7f811f632740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f67630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.101+0000 7f811f632740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f81181b37c0 con 0x557882f44f50 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.101+0000 7f811f632740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f81181b1090 con 0x557882f44630 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.101+0000 7f811f632740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81181aeee0 con 0x557882f40950 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f67630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f67630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56140/0 (socket says 172.21.15.27:56140) 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 -- 172.21.15.27:0/4027579181 learned_addr learned my addr 172.21.15.27:0/4027579181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f811659c640 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f40950 0x557882f5b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8115d9b640 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f46d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 -- 172.21.15.27:0/4027579181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f40950 msgr2=0x557882f5b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f40950 0x557882f5b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 -- 172.21.15.27:0/4027579181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 msgr2=0x557882f46d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f46d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 -- 172.21.15.27:0/4027579181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557882f32090 con 0x557882f44f50 2024-04-25T01:30:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f8116d9d640 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f67630 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f81100097e0 tx=0x7f81100093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbc9420a45129a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f80fffff640 1 -- 172.21.15.27:0/4027579181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811002d070 con 0x557882f44f50 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f80fffff640 1 -- 172.21.15.27:0/4027579181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f811000ee00 con 0x557882f44f50 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.102+0000 7f80fffff640 1 -- 172.21.15.27:0/4027579181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110024040 con 0x557882f44f50 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 -- 172.21.15.27:0/4027579181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 msgr2=0x557882f67630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f67630 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f81100097e0 tx=0x7f81100093c0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 -- 172.21.15.27:0/4027579181 shutdown_connections 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f67630 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f44630 0x557882f46d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 --2- 172.21.15.27:0/4027579181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f40950 0x557882f5b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 -- 172.21.15.27:0/4027579181 >> 172.21.15.27:0/4027579181 conn(0x557882f4b230 msgr2=0x557882f3faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 -- 172.21.15.27:0/4027579181 shutdown_connections 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 -- 172.21.15.27:0/4027579181 wait complete. 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 -- start start 2024-04-25T01:30:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.103+0000 7f811f632740 1 Processor -- start 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811f632740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f40950 0x557882f53370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811f632740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44630 0x557882f538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811f632740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f53df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811f632740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f81181af550 con 0x557882f44f50 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811f632740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f81181b09c0 con 0x557882f40950 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811f632740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81181afc20 con 0x557882f44630 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811f632740 10 client.0 _ll_register_callbacks cb 0x7f8110009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f53df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f53df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56148/0 (socket says 172.21.15.27:56148) 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 -- 172.21.15.27:0/2409123093 learned_addr learned my addr 172.21.15.27:0/2409123093 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811659c640 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f40950 0x557882f53370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8115d9b640 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44630 0x557882f538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 -- 172.21.15.27:0/2409123093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44630 msgr2=0x557882f538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44630 0x557882f538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 -- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f40950 msgr2=0x557882f53370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f40950 0x557882f53370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 -- 172.21.15.27:0/2409123093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557882fc5050 con 0x557882f44f50 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811659c640 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f40950 0x557882f53370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f8116d9d640 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f53df0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f811000efd0 tx=0x7f8110013400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373f31774e952f03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f80fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f80fdffb640 1 -- 172.21.15.27:0/2409123093 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110041070 con 0x557882f44f50 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f80fdffb640 1 -- 172.21.15.27:0/2409123093 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f811000e920 con 0x557882f44f50 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.104+0000 7f811f632740 1 -- 172.21.15.27:0/2409123093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f811000ee00 con 0x557882f44f50 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f80fdffb640 1 -- 172.21.15.27:0/2409123093 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f811004d050 con 0x557882f44f50 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811f632740 10 client.16677 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811f632740 1 -- 172.21.15.27:0/2409123093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557882fc6320 con 0x557882f44f50 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f80fdffb640 1 -- 172.21.15.27:0/2409123093 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f811000ee00 con 0x557882f44f50 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f80fdffb640 1 -- 172.21.15.27:0/2409123093 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8110046ca0 con 0x557882f44f50 2024-04-25T01:30:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811f632740 10 client.16677 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811f632740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811f632740 20 client.16677 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811f632740 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557882f4f620 0x557882fc8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811f632740 4 client.16677 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811f632740 1 -- 172.21.15.27:0/2409123093 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557882eb0f40 con 0x557882f4f620 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.105+0000 7f811659c640 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557882f4f620 0x557882fc8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811659c640 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557882f4f620 0x557882fc8970 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d782e12c451ccd8 server_cookie=6e8e2bce3584cd31 in_seq=0 out_seq=0 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f80fdffb640 10 client.16677 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f80fdffb640 1 -- 172.21.15.27:0/2409123093 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557882eb0f40 con 0x557882f4f620 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f80fdffb640 10 client.16677 handle_command_reply: tid=0 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 client.16677 shutdown 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 -- 172.21.15.27:0/2409123093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 msgr2=0x557882f53df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f53df0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f811000efd0 tx=0x7f8110013400 comp rx=0 tx=0).stop 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 -- 172.21.15.27:0/2409123093 shutdown_connections 2024-04-25T01:30:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x557882f4f620 0x557882fc8970 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557882f44f50 0x557882f53df0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f80fdffb640 0 client.16677 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557882f44630 0x557882f538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 --2- 172.21.15.27:0/2409123093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557882f40950 0x557882f53370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 -- 172.21.15.27:0/2409123093 >> 172.21.15.27:0/2409123093 conn(0x557882f4b230 msgr2=0x557882f4c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 -- 172.21.15.27:0/2409123093 shutdown_connections 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 1 -- 172.21.15.27:0/2409123093 wait complete. 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.106+0000 7f811f632740 20 client.16677 trim_cache size 0 max 16384 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:30:22.090 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:30:22.091 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:30:22.091 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:30:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 -- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f81000a2d00 msgr2=0x7f81000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f81000a2d00 0x7f81000a51c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8110005fd0 tx=0x7f81100139c0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 -- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 msgr2=0x7f811819a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 0x7f811819a300 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f810800d950 tx=0x7f810800de10 comp rx=0 tx=0).stop 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 -- 172.21.15.27:0/442298919 shutdown_connections 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8118109990 0x7f811819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8118104340 0x7f811819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f81000a2d00 0x7f81000a51c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 --2- 172.21.15.27:0/442298919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8118103980 0x7f811819a300 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 -- 172.21.15.27:0/442298919 >> 172.21.15.27:0/442298919 conn(0x7f81180ff4d0 msgr2=0x7f81180ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 -- 172.21.15.27:0/442298919 shutdown_connections 2024-04-25T01:30:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.107+0000 7f811c922640 1 -- 172.21.15.27:0/442298919 wait complete. 2024-04-25T01:30:22.095 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:30:22.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 -- 172.21.15.27:0/1531204647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 msgr2=0x7f5e1810e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1531204647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 0x7f5e1810e5a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009ca0 tx=0x7f5e1002a950 comp rx=0 tx=0).stop 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 -- 172.21.15.27:0/1531204647 shutdown_connections 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1531204647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 0x7f5e1810e5a0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1531204647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e181017c0 0x7f5e18101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1531204647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e18107750 0x7f5e18107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 -- 172.21.15.27:0/1531204647 >> 172.21.15.27:0/1531204647 conn(0x7f5e180788d0 msgr2=0x7f5e18078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 -- 172.21.15.27:0/1531204647 shutdown_connections 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 -- 172.21.15.27:0/1531204647 wait complete. 2024-04-25T01:30:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 Processor -- start 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.220+0000 7f5e1f561640 1 -- start start 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1f561640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e181017c0 0x7f5e18197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1f561640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 0x7f5e18198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1f561640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e18107750 0x7f5e1819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1f561640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18111b10 con 0x7f5e18102160 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1f561640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18111990 con 0x7f5e18107750 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1f561640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18111c90 con 0x7f5e181017c0 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 0x7f5e18198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 0x7f5e18198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56168/0 (socket says 172.21.15.27:56168) 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 -- 172.21.15.27:0/1463907408 learned_addr learned my addr 172.21.15.27:0/1463907408 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1d2fd640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e181017c0 0x7f5e18197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1dafe640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e18107750 0x7f5e1819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 -- 172.21.15.27:0/1463907408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e181017c0 msgr2=0x7f5e18197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e181017c0 0x7f5e18197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 -- 172.21.15.27:0/1463907408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e18107750 msgr2=0x7f5e1819c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e18107750 0x7f5e1819c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 -- 172.21.15.27:0/1463907408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1819cec0 con 0x7f5e18102160 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1cafc640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 0x7f5e18198410 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800c920 tx=0x7f5e0800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e0e7fc640 1 -- 172.21.15.27:0/1463907408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08007c70 con 0x7f5e18102160 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e0e7fc640 1 -- 172.21.15.27:0/1463907408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e08007e10 con 0x7f5e18102160 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1819d1b0 con 0x7f5e18102160 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.221+0000 7f5e0e7fc640 1 -- 172.21.15.27:0/1463907408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08018890 con 0x7f5e18102160 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.222+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e181a4a40 con 0x7f5e18102160 2024-04-25T01:30:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.222+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e180689b0 con 0x7f5e18102160 2024-04-25T01:30:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.223+0000 7f5e0e7fc640 1 -- 172.21.15.27:0/1463907408 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5e08018a50 con 0x7f5e18102160 2024-04-25T01:30:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.224+0000 7f5e0e7fc640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5e040a2d80 0x7f5e040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.224+0000 7f5e1d2fd640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5e040a2d80 0x7f5e040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.224+0000 7f5e0e7fc640 1 -- 172.21.15.27:0/1463907408 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5e080cf940 con 0x7f5e18102160 2024-04-25T01:30:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.225+0000 7f5e1d2fd640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5e040a2d80 0x7f5e040a5240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009ae0 tx=0x7f5e14005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.229+0000 7f5e0e7fc640 1 -- 172.21.15.27:0/1463907408 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5e08014030 con 0x7f5e18102160 2024-04-25T01:30:22.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.455+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e181a4e00 con 0x7f5e18102160 2024-04-25T01:30:22.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.456+0000 7f5e0e7fc640 1 -- 172.21.15.27:0/1463907408 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7f5e0800de20 con 0x7f5e18102160 2024-04-25T01:30:22.434 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:30:22.434 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:30:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5e040a2d80 msgr2=0x7f5e040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5e040a2d80 0x7f5e040a5240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009ae0 tx=0x7f5e14005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 msgr2=0x7f5e18198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 0x7f5e18198410 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800c920 tx=0x7f5e0800cde0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 shutdown_connections 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e18107750 0x7f5e1819c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5e040a2d80 0x7f5e040a5240 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e18102160 0x7f5e18198410 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 --2- 172.21.15.27:0/1463907408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e181017c0 0x7f5e18197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 >> 172.21.15.27:0/1463907408 conn(0x7f5e180788d0 msgr2=0x7f5e18100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 shutdown_connections 2024-04-25T01:30:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.458+0000 7f5e1f561640 1 -- 172.21.15.27:0/1463907408 wait complete. 2024-04-25T01:30:22.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:30:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 -- 172.21.15.27:0/1973217839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c106aa0 msgr2=0x7f0b8c10e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 --2- 172.21.15.27:0/1973217839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c106aa0 0x7f0b8c10e6b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009ca0 tx=0x7f0b8402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 -- 172.21.15.27:0/1973217839 shutdown_connections 2024-04-25T01:30:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 --2- 172.21.15.27:0/1973217839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c106aa0 0x7f0b8c10e6b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 --2- 172.21.15.27:0/1973217839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b8c100c90 0x7f0b8c1063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 --2- 172.21.15.27:0/1973217839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b8c1002e0 0x7f0b8c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 -- 172.21.15.27:0/1973217839 >> 172.21.15.27:0/1973217839 conn(0x7f0b8c0758d0 msgr2=0x7f0b8c075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 -- 172.21.15.27:0/1973217839 shutdown_connections 2024-04-25T01:30:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.573+0000 7f0b91990640 1 -- 172.21.15.27:0/1973217839 wait complete. 2024-04-25T01:30:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b91990640 1 Processor -- start 2024-04-25T01:30:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b91990640 1 -- start start 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b91990640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b8c1002e0 0x7f0b8c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b91990640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b8c106aa0 0x7f0b8c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b91990640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c19c8b0 0x7f0b8c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b91990640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c111b90 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b91990640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c111a10 con 0x7f0b8c1002e0 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b91990640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c111d10 con 0x7f0b8c106aa0 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b8b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c19c8b0 0x7f0b8c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b8b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c19c8b0 0x7f0b8c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56200/0 (socket says 172.21.15.27:56200) 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.574+0000 7f0b8b7fe640 1 -- 172.21.15.27:0/998473106 learned_addr learned my addr 172.21.15.27:0/998473106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b8c106aa0 0x7f0b8c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8affd640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b8c1002e0 0x7f0b8c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8b7fe640 1 -- 172.21.15.27:0/998473106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b8c106aa0 msgr2=0x7f0b8c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8b7fe640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b8c106aa0 0x7f0b8c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8b7fe640 1 -- 172.21.15.27:0/998473106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b8c1002e0 msgr2=0x7f0b8c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8b7fe640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b8c1002e0 0x7f0b8c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8b7fe640 1 -- 172.21.15.27:0/998473106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8c19d3a0 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b8c106aa0 0x7f0b8c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b8b7fe640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c19c8b0 0x7f0b8c19cd60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0b84031a60 tx=0x7f0b84031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b9098e640 1 -- 172.21.15.27:0/998473106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b84003c30 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b91990640 1 -- 172.21.15.27:0/998473106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8c19d630 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b9098e640 1 -- 172.21.15.27:0/998473106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b84003dd0 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b9098e640 1 -- 172.21.15.27:0/998473106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b840047e0 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.575+0000 7f0b91990640 1 -- 172.21.15.27:0/998473106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b8c1a4e80 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b9098e640 1 -- 172.21.15.27:0/998473106 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0b8402f790 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b94779740 1 Processor -- start 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b94779740 1 -- start start 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b94779740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83480 0x55c78cf83860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b94779740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 0x55c78cf92440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b94779740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 0x55c78cf94d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b94779740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c78cee8670 con 0x55c78cf83480 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b94779740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c78ce90970 con 0x55c78cf83da0 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b94779740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c78cee4630 con 0x55c78cf92980 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.578+0000 7f0b9098e640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0b740a2d00 0x7f0b740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83480 0x55c78cf83860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8b7fe640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0b740a2d00 0x7f0b740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83480 0x55c78cf83860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56202/0 (socket says 172.21.15.27:56202) 2024-04-25T01:30:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/1258800956 learned_addr learned my addr 172.21.15.27:0/1258800956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 0x55c78cf94d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b9098e640 1 -- 172.21.15.27:0/998473106 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0b840f0bb0 con 0x7f0b8c19c8b0 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8affd640 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 0x55c78cf92440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/1258800956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 msgr2=0x55c78cf94d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 0x55c78cf94d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/1258800956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 msgr2=0x55c78cf92440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 0x55c78cf92440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/1258800956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c78ceb4400 con 0x55c78cf83480 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8affd640 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 0x55c78cf92440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83480 0x55c78cf83860 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0b80012470 tx=0x7f0b8000c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3498cdc6e57909c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b8b7fe640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0b740a2d00 0x7f0b740a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0b840ee130 tx=0x7f0b8408ee10 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b71ffb640 1 -- 172.21.15.27:0/1258800956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80018030 con 0x55c78cf83480 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b71ffb640 1 -- 172.21.15.27:0/1258800956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b80005500 con 0x55c78cf83480 2024-04-25T01:30:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.579+0000 7f0b71ffb640 1 -- 172.21.15.27:0/1258800956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80010880 con 0x55c78cf83480 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 -- 172.21.15.27:0/1258800956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83480 msgr2=0x55c78cf83860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83480 0x55c78cf83860 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0b80012470 tx=0x7f0b8000c5c0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 -- 172.21.15.27:0/1258800956 shutdown_connections 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 0x55c78cf94d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 0x55c78cf92440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 --2- 172.21.15.27:0/1258800956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83480 0x55c78cf83860 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 -- 172.21.15.27:0/1258800956 >> 172.21.15.27:0/1258800956 conn(0x55c78cf7f8a0 msgr2=0x55c78cf7fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 -- 172.21.15.27:0/1258800956 shutdown_connections 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 -- 172.21.15.27:0/1258800956 wait complete. 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.580+0000 7f0b94779740 1 -- start start 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b94779740 1 Processor -- start 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b94779740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 0x55c78cf92240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b94779740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83da0 0x55c78cf8f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b94779740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cf8fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b94779740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c78cf95740 con 0x55c78cf92980 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b94779740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c78cea4050 con 0x55c78cf83480 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b94779740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c78cf958c0 con 0x55c78cf83da0 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b94779740 10 client.0 _ll_register_callbacks cb 0x55c78cbfbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cf8fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cf8fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56208/0 (socket says 172.21.15.27:56208) 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/3492103268 learned_addr learned my addr 172.21.15.27:0/3492103268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8affd640 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83da0 0x55c78cf8f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 0x55c78cf92240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/3492103268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83da0 msgr2=0x55c78cf8f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83da0 0x55c78cf8f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/3492103268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 msgr2=0x55c78cf92240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 0x55c78cf92240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/3492103268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c78ce75dc0 con 0x55c78cf92980 2024-04-25T01:30:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cf8fa70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0b800054d0 tx=0x7f0b80004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d1525061a2bcd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.581+0000 7f0b6affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b6affd640 1 -- 172.21.15.27:0/3492103268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80018040 con 0x55c78cf92980 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 1 -- 172.21.15.27:0/3492103268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b80005500 con 0x55c78cf92980 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b6affd640 1 -- 172.21.15.27:0/3492103268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0b80004970 con 0x55c78cf92980 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b6affd640 1 -- 172.21.15.27:0/3492103268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b80033050 con 0x55c78cf92980 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 10 client.16695 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 1 -- 172.21.15.27:0/3492103268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c78cbfc3f0 con 0x55c78cf92980 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b6affd640 1 -- 172.21.15.27:0/3492103268 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b80005500 con 0x55c78cf92980 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b6affd640 1 -- 172.21.15.27:0/3492103268 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0b8001d6c0 con 0x55c78cf92980 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 10 client.16695 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 10 client.16695 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 20 client.16695 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c78cfa27d0 0x55c78cfa4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 4 client.16695 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b94779740 1 -- 172.21.15.27:0/3492103268 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c78cef5bb0 con 0x55c78cfa27d0 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.582+0000 7f0b8affd640 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c78cfa27d0 0x55c78cfa4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.583+0000 7f0b8affd640 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c78cfa27d0 0x55c78cfa4bc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8500b1264cdbfaf4 server_cookie=49d6be693b8417b0 in_seq=0 out_seq=0 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.583+0000 7f0b6affd640 10 client.16695 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.583+0000 7f0b6affd640 1 -- 172.21.15.27:0/3492103268 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c78cef5bb0 con 0x55c78cfa27d0 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b6affd640 10 client.16695 handle_command_reply: tid=0 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 client.16695 shutdown 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 -- 172.21.15.27:0/3492103268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 msgr2=0x55c78cf8fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cf8fa70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0b800054d0 tx=0x7f0b80004d90 comp rx=0 tx=0).stop 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 -- 172.21.15.27:0/3492103268 shutdown_connections 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c78cfa27d0 0x55c78cfa4bc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cf8fa70 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b6affd640 0 client.16695 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:30:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83da0 0x55c78cf8f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 --2- 172.21.15.27:0/3492103268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 0x55c78cf92240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 -- 172.21.15.27:0/3492103268 >> 172.21.15.27:0/3492103268 conn(0x55c78cf7f8a0 msgr2=0x55c78cf7eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 -- 172.21.15.27:0/3492103268 shutdown_connections 2024-04-25T01:30:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 1 -- 172.21.15.27:0/3492103268 wait complete. 2024-04-25T01:30:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.584+0000 7f0b94779740 20 client.16695 trim_cache size 0 max 16384 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.597+0000 7f0b94779740 1 Processor -- start 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.597+0000 7f0b94779740 1 -- start start 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.597+0000 7f0b94779740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83480 0x55c78cf8e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.597+0000 7f0b94779740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 0x55c78cf91d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.597+0000 7f0b94779740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cfaa280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.597+0000 7f0b94779740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c1af230 con 0x55c78cf92980 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.597+0000 7f0b94779740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c1acb00 con 0x55c78cf83da0 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.597+0000 7f0b94779740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c1aa950 con 0x55c78cf83480 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cfaa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cfaa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56216/0 (socket says 172.21.15.27:56216) 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/2407323455 learned_addr learned my addr 172.21.15.27:0/2407323455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/2407323455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83480 0x55c78cf8e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/2407323455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83480 msgr2=0x55c78cf8e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/2407323455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83480 0x55c78cf8e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/2407323455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 msgr2=0x55c78cf91d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/2407323455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 0x55c78cf91d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 -- 172.21.15.27:0/2407323455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c78cf8b150 con 0x55c78cf92980 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/2407323455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cfaa280 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0b80012470 tx=0x7f0b80005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679b599bf1c46d9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b717fa640 1 -- 172.21.15.27:0/2407323455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80018020 con 0x55c78cf92980 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b717fa640 1 -- 172.21.15.27:0/2407323455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b80005500 con 0x55c78cf92980 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.598+0000 7f0b717fa640 1 -- 172.21.15.27:0/2407323455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80021070 con 0x55c78cf92980 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- 172.21.15.27:0/2407323455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 msgr2=0x55c78cfaa280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 --2- 172.21.15.27:0/2407323455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cfaa280 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0b80012470 tx=0x7f0b80005a50 comp rx=0 tx=0).stop 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- 172.21.15.27:0/2407323455 shutdown_connections 2024-04-25T01:30:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 --2- 172.21.15.27:0/2407323455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf92980 0x55c78cfaa280 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 --2- 172.21.15.27:0/2407323455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83da0 0x55c78cf91d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 --2- 172.21.15.27:0/2407323455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf83480 0x55c78cf8e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- 172.21.15.27:0/2407323455 >> 172.21.15.27:0/2407323455 conn(0x55c78cf7f8a0 msgr2=0x55c78cf84d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- 172.21.15.27:0/2407323455 shutdown_connections 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- 172.21.15.27:0/2407323455 wait complete. 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- start start 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 Processor -- start 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 0x55c78cf9ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83da0 0x55c78cf8bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 0x55c78cf8c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c1aafc0 con 0x55c78cf83da0 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c1ac430 con 0x55c78cf83480 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.599+0000 7f0b94779740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8c1ab690 con 0x55c78cf92980 2024-04-25T01:30:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b94779740 10 client.0 _ll_register_callbacks cb 0x7f0b8000cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83da0 0x55c78cf8bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83da0 0x55c78cf8bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56224/0 (socket says 172.21.15.27:56224) 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 -- 172.21.15.27:0/1189058084 learned_addr learned my addr 172.21.15.27:0/1189058084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 0x55c78cf8c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 0x55c78cf9ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 -- 172.21.15.27:0/1189058084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 msgr2=0x55c78cf8c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 0x55c78cf8c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 -- 172.21.15.27:0/1189058084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 msgr2=0x55c78cf9ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 0x55c78cf9ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 -- 172.21.15.27:0/1189058084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c78ccc67d0 con 0x55c78cf83da0 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b8affd640 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83da0 0x55c78cf8bd80 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00cb90 tx=0x7f0b7c00e700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4254315241948256 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b6affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b6affd640 1 -- 172.21.15.27:0/1189058084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c01c630 con 0x55c78cf83da0 2024-04-25T01:30:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b6affd640 1 -- 172.21.15.27:0/1189058084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0b7c01c7d0 con 0x55c78cf83da0 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b94779740 1 -- 172.21.15.27:0/1189058084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b80005500 con 0x55c78cf83da0 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b6affd640 1 -- 172.21.15.27:0/1189058084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b7c02c050 con 0x55c78cf83da0 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b94779740 10 client.16701 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b94779740 1 -- 172.21.15.27:0/1189058084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c78d008810 con 0x55c78cf83da0 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.600+0000 7f0b6affd640 1 -- 172.21.15.27:0/1189058084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b80005500 con 0x55c78cf83da0 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b6affd640 1 -- 172.21.15.27:0/1189058084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0b7c01b3b0 con 0x55c78cf83da0 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b94779740 10 client.16701 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b94779740 10 client.16701 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b94779740 20 client.16701 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b94779740 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c78d008f70 0x55c78d00b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b94779740 4 client.16701 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b94779740 1 -- 172.21.15.27:0/1189058084 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c78ceeff00 con 0x55c78d008f70 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c78d008f70 0x55c78d00b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b6affd640 10 client.16701 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:30:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.601+0000 7f0b8a7fc640 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c78d008f70 0x55c78d00b390 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=817c2490749a91fc server_cookie=553f503b0a5cc018 in_seq=0 out_seq=0 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b6affd640 1 -- 172.21.15.27:0/1189058084 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c78ceeff00 con 0x55c78d008f70 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b6affd640 10 client.16701 handle_command_reply: tid=0 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 client.16701 shutdown 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 -- 172.21.15.27:0/1189058084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83da0 msgr2=0x55c78cf8bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83da0 0x55c78cf8bd80 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00cb90 tx=0x7f0b7c00e700 comp rx=0 tx=0).stop 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 -- 172.21.15.27:0/1189058084 shutdown_connections 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c78d008f70 0x55c78d00b390 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c78cf92980 0x55c78cf8c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c78cf83da0 0x55c78cf8bd80 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b6affd640 0 client.16701 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 --2- 172.21.15.27:0/1189058084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c78cf83480 0x55c78cf9ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 -- 172.21.15.27:0/1189058084 >> 172.21.15.27:0/1189058084 conn(0x55c78cf7f8a0 msgr2=0x55c78cf855d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 -- 172.21.15.27:0/1189058084 shutdown_connections 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 1 -- 172.21.15.27:0/1189058084 wait complete. 2024-04-25T01:30:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b94779740 20 client.16701 trim_cache size 0 max 16384 2024-04-25T01:30:22.586 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:30:22.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 -- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0b740a2d00 msgr2=0x7f0b740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0b740a2d00 0x7f0b740a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0b840ee130 tx=0x7f0b8408ee10 comp rx=0 tx=0).stop 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 -- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c19c8b0 msgr2=0x7f0b8c19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c19c8b0 0x7f0b8c19cd60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0b84031a60 tx=0x7f0b84031a90 comp rx=0 tx=0).stop 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 -- 172.21.15.27:0/998473106 shutdown_connections 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0b740a2d00 0x7f0b740a51c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c19c8b0 0x7f0b8c19cd60 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b8c106aa0 0x7f0b8c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 --2- 172.21.15.27:0/998473106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0b8c1002e0 0x7f0b8c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.602+0000 7f0b91990640 1 -- 172.21.15.27:0/998473106 >> 172.21.15.27:0/998473106 conn(0x7f0b8c0758d0 msgr2=0x7f0b8c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.603+0000 7f0b91990640 1 -- 172.21.15.27:0/998473106 shutdown_connections 2024-04-25T01:30:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.603+0000 7f0b91990640 1 -- 172.21.15.27:0/998473106 wait complete. 2024-04-25T01:30:22.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:30:22.611 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:22 smithi027 ceph-mon[29197]: pgmap v151: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:22.665 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:22 smithi139 ceph-mon[32760]: pgmap v151: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 -- 172.21.15.27:0/3383634749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4581020d0 msgr2=0x7fa45810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 --2- 172.21.15.27:0/3383634749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4581020d0 0x7fa45810e5e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa450009cd0 tx=0x7fa45002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 -- 172.21.15.27:0/3383634749 shutdown_connections 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 --2- 172.21.15.27:0/3383634749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4581020d0 0x7fa45810e5e0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 --2- 172.21.15.27:0/3383634749 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa458101710 0x7fa458101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 --2- 172.21.15.27:0/3383634749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa458107720 0x7fa458107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 -- 172.21.15.27:0/3383634749 >> 172.21.15.27:0/3383634749 conn(0x7fa4580fd180 msgr2=0x7fa4580ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 -- 172.21.15.27:0/3383634749 shutdown_connections 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.717+0000 7fa45d548640 1 -- 172.21.15.27:0/3383634749 wait complete. 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa45d548640 1 Processor -- start 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa45d548640 1 -- start start 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa45d548640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa458101710 0x7fa458073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa45d548640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4581020d0 0x7fa458073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa45d548640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458107720 0x7fa4580744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa45d548640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa458111b30 con 0x7fa458107720 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa45d548640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa4581119b0 con 0x7fa4581020d0 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa45d548640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa458111cb0 con 0x7fa458101710 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa4577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458107720 0x7fa4580744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa4577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458107720 0x7fa4580744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56240/0 (socket says 172.21.15.27:56240) 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa4577fe640 1 -- 172.21.15.27:0/900264079 learned_addr learned my addr 172.21.15.27:0/900264079 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa44ffff640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4581020d0 0x7fa458073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa456ffd640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa458101710 0x7fa458073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa4577fe640 1 -- 172.21.15.27:0/900264079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa458101710 msgr2=0x7fa458073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa4577fe640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa458101710 0x7fa458073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa4577fe640 1 -- 172.21.15.27:0/900264079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4581020d0 msgr2=0x7fa458073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa4577fe640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4581020d0 0x7fa458073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa4577fe640 1 -- 172.21.15.27:0/900264079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4581a6510 con 0x7fa458107720 2024-04-25T01:30:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.718+0000 7fa44ffff640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4581020d0 0x7fa458073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.719+0000 7fa4577fe640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458107720 0x7fa4580744c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa450005cc0 tx=0x7fa450032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.719+0000 7fa454ff9640 1 -- 172.21.15.27:0/900264079 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450004040 con 0x7fa458107720 2024-04-25T01:30:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.719+0000 7fa454ff9640 1 -- 172.21.15.27:0/900264079 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4500046c0 con 0x7fa458107720 2024-04-25T01:30:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.719+0000 7fa45d548640 1 -- 172.21.15.27:0/900264079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4581a67a0 con 0x7fa458107720 2024-04-25T01:30:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.719+0000 7fa454ff9640 1 -- 172.21.15.27:0/900264079 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450030e00 con 0x7fa458107720 2024-04-25T01:30:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.719+0000 7fa45d548640 1 -- 172.21.15.27:0/900264079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4581a6cf0 con 0x7fa458107720 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.721+0000 7fa454ff9640 1 -- 172.21.15.27:0/900264079 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa4500305a0 con 0x7fa458107720 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.721+0000 7fa46024b740 1 Processor -- start 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.721+0000 7fa46024b740 1 -- start start 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa46024b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc50b580 0x5570bc50b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa46024b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc516030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa46024b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc516570 0x5570bc518960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa46024b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5570bc470670 con 0x5570bc50bea0 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa46024b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5570bc418970 con 0x5570bc50b580 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa46024b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5570bc46c630 con 0x5570bc516570 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc516030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc516030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56242/0 (socket says 172.21.15.27:56242) 2024-04-25T01:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 -- 172.21.15.27:0/444542239 learned_addr learned my addr 172.21.15.27:0/444542239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa454ff9640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa4440a2d00 0x7fa4440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa44ffff640 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc516570 0x5570bc518960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa44ffff640 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc50b580 0x5570bc50b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa4577fe640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa4440a2d00 0x7fa4440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 -- 172.21.15.27:0/444542239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc516570 msgr2=0x5570bc518960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc516570 0x5570bc518960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 -- 172.21.15.27:0/444542239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc50b580 msgr2=0x5570bc50b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc50b580 0x5570bc50b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 -- 172.21.15.27:0/444542239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570bc43c400 con 0x5570bc50bea0 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa44ffff640 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc516570 0x5570bc518960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa454ff9640 1 -- 172.21.15.27:0/900264079 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa45002e030 con 0x7fa458107720 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.722+0000 7fa456ffd640 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc516030 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa448009c10 tx=0x7fa4480117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885815befc995d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa4577fe640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa4440a2d00 0x7fa4440a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa4500ee540 tx=0x7fa45008f220 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa44d7fa640 1 -- 172.21.15.27:0/444542239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa448012450 con 0x5570bc50bea0 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa44d7fa640 1 -- 172.21.15.27:0/444542239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa448012ad0 con 0x5570bc50bea0 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa44d7fa640 1 -- 172.21.15.27:0/444542239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44800fb70 con 0x5570bc50bea0 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa46024b740 1 -- 172.21.15.27:0/444542239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 msgr2=0x5570bc516030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa46024b740 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc516030 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa448009c10 tx=0x7fa4480117b0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa46024b740 1 -- 172.21.15.27:0/444542239 shutdown_connections 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa46024b740 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc516570 0x5570bc518960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa46024b740 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc516030 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa46024b740 1 --2- 172.21.15.27:0/444542239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc50b580 0x5570bc50b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa46024b740 1 -- 172.21.15.27:0/444542239 >> 172.21.15.27:0/444542239 conn(0x5570bc5078a0 msgr2=0x5570bc507c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.723+0000 7fa46024b740 1 -- 172.21.15.27:0/444542239 shutdown_connections 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 -- 172.21.15.27:0/444542239 wait complete. 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 -- start start 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 Processor -- start 2024-04-25T01:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc515ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc5131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc513710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5570bc519360 con 0x5570bc50b580 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5570bc42c050 con 0x5570bc516570 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5570bc5194e0 con 0x5570bc50bea0 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa46024b740 10 client.0 _ll_register_callbacks cb 0x5570bc183be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa44ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc515ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa44ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc515ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56256/0 (socket says 172.21.15.27:56256) 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa44ffff640 1 -- 172.21.15.27:0/371178044 learned_addr learned my addr 172.21.15.27:0/371178044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa44ffff640 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc513710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa456ffd640 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc5131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.724+0000 7fa44ffff640 1 -- 172.21.15.27:0/371178044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 msgr2=0x5570bc5131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa44ffff640 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc5131b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa44ffff640 1 -- 172.21.15.27:0/371178044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 msgr2=0x5570bc513710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa44ffff640 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc513710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa44ffff640 1 -- 172.21.15.27:0/371178044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570bc3fddc0 con 0x5570bc50b580 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa44ffff640 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc515ec0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa44000be60 tx=0x7fa44000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720d349db9bf8b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa43e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa43e7fc640 1 -- 172.21.15.27:0/371178044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440004220 con 0x5570bc50b580 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa43e7fc640 1 -- 172.21.15.27:0/371178044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa4400043c0 con 0x5570bc50b580 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa46024b740 1 -- 172.21.15.27:0/371178044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa448012ad0 con 0x5570bc50b580 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa43e7fc640 1 -- 172.21.15.27:0/371178044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa440025050 con 0x5570bc50b580 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa46024b740 10 client.16713 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa46024b740 1 -- 172.21.15.27:0/371178044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5570bc1843f0 con 0x5570bc50b580 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa43e7fc640 1 -- 172.21.15.27:0/371178044 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa448012ad0 con 0x5570bc50b580 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa43e7fc640 1 -- 172.21.15.27:0/371178044 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa44000f040 con 0x5570bc50b580 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa46024b740 10 client.16713 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa46024b740 10 client.16713 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.725+0000 7fa46024b740 20 client.16713 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:22.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.726+0000 7fa46024b740 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5570bc5266f0 0x5570bc528b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.726+0000 7fa46024b740 4 client.16713 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.726+0000 7fa46024b740 1 -- 172.21.15.27:0/371178044 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570bc47dbb0 con 0x5570bc5266f0 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.726+0000 7fa456ffd640 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5570bc5266f0 0x5570bc528b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.726+0000 7fa456ffd640 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5570bc5266f0 0x5570bc528b00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d12756ff16e709bd server_cookie=3b21e8c6f1ebde7a in_seq=0 out_seq=0 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.726+0000 7fa43e7fc640 10 client.16713 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa43e7fc640 1 -- 172.21.15.27:0/371178044 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570bc47dbb0 con 0x5570bc5266f0 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa43e7fc640 10 client.16713 handle_command_reply: tid=0 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 client.16713 shutdown 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 -- 172.21.15.27:0/371178044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 msgr2=0x5570bc515ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc515ec0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa44000be60 tx=0x7fa44000b960 comp rx=0 tx=0).stop 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 -- 172.21.15.27:0/371178044 shutdown_connections 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5570bc5266f0 0x5570bc528b00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc513710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc5131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa43e7fc640 0 client.16713 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:30:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 --2- 172.21.15.27:0/371178044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc515ec0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 -- 172.21.15.27:0/371178044 >> 172.21.15.27:0/371178044 conn(0x5570bc5078a0 msgr2=0x5570bc506b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 -- 172.21.15.27:0/371178044 shutdown_connections 2024-04-25T01:30:22.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 1 -- 172.21.15.27:0/371178044 wait complete. 2024-04-25T01:30:22.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.727+0000 7fa46024b740 20 client.16713 trim_cache size 0 max 16384 2024-04-25T01:30:22.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.740+0000 7fa46024b740 1 Processor -- start 2024-04-25T01:30:22.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.740+0000 7fa46024b740 1 -- start start 2024-04-25T01:30:22.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa46024b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50b580 0x5570bc522250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa46024b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc5159f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa46024b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc52e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa46024b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa4581b3610 con 0x5570bc50bea0 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa46024b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa4581b0ee0 con 0x5570bc516570 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa46024b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4581aed30 con 0x5570bc50b580 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc5159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc5159f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56266/0 (socket says 172.21.15.27:56266) 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 -- 172.21.15.27:0/3758774157 learned_addr learned my addr 172.21.15.27:0/3758774157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa44ffff640 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50b580 0x5570bc522250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa44ffff640 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc52e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 -- 172.21.15.27:0/3758774157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50b580 msgr2=0x5570bc522250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50b580 0x5570bc522250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 -- 172.21.15.27:0/3758774157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 msgr2=0x5570bc52e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc52e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 -- 172.21.15.27:0/3758774157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570bc24e7d0 con 0x5570bc50bea0 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa44ffff640 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50b580 0x5570bc522250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.741+0000 7fa456ffd640 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc5159f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x5570bc50cb40 tx=0x7fa44800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a829205ba9f84f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa44cff9640 1 -- 172.21.15.27:0/3758774157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44800f440 con 0x5570bc50bea0 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa44cff9640 1 -- 172.21.15.27:0/3758774157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa44802a070 con 0x5570bc50bea0 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa44cff9640 1 -- 172.21.15.27:0/3758774157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa448005680 con 0x5570bc50bea0 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 -- 172.21.15.27:0/3758774157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 msgr2=0x5570bc5159f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc5159f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x5570bc50cb40 tx=0x7fa44800f770 comp rx=0 tx=0).stop 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 -- 172.21.15.27:0/3758774157 shutdown_connections 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc52e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50bea0 0x5570bc5159f0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 --2- 172.21.15.27:0/3758774157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50b580 0x5570bc522250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 -- 172.21.15.27:0/3758774157 >> 172.21.15.27:0/3758774157 conn(0x5570bc5078a0 msgr2=0x5570bc50e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 -- 172.21.15.27:0/3758774157 shutdown_connections 2024-04-25T01:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.742+0000 7fa46024b740 1 -- 172.21.15.27:0/3758774157 wait complete. 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 1 -- start start 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 1 Processor -- start 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc51e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc50fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc50ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa4581af3a0 con 0x5570bc50b580 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa4581b0810 con 0x5570bc516570 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4581afa70 con 0x5570bc50bea0 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa46024b740 10 client.0 _ll_register_callbacks cb 0x7fa44000b7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc51e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc51e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56284/0 (socket says 172.21.15.27:56284) 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 -- 172.21.15.27:0/3011408139 learned_addr learned my addr 172.21.15.27:0/3011408139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 -- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 msgr2=0x5570bc50fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa456ffd640 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc50fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc50fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 -- 172.21.15.27:0/3011408139 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 msgr2=0x5570bc50ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc50ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 -- 172.21.15.27:0/3011408139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570bc5105b0 con 0x5570bc50b580 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa456ffd640 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc50fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.743+0000 7fa44ffff640 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc51e9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa44000be60 tx=0x7fa44000b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2dda5ba5fe3c032 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa43e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa43e7fc640 1 -- 172.21.15.27:0/3011408139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44002f060 con 0x5570bc50b580 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 1 -- 172.21.15.27:0/3011408139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa44802a070 con 0x5570bc50b580 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa43e7fc640 1 -- 172.21.15.27:0/3011408139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fa440011a60 con 0x5570bc50b580 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa43e7fc640 1 -- 172.21.15.27:0/3011408139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa440011c60 con 0x5570bc50b580 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 10 client.16719 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 1 -- 172.21.15.27:0/3011408139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5570bc58cb00 con 0x5570bc50b580 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa43e7fc640 1 -- 172.21.15.27:0/3011408139 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa44802a070 con 0x5570bc50b580 2024-04-25T01:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa43e7fc640 1 -- 172.21.15.27:0/3011408139 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa440028040 con 0x5570bc50b580 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 10 client.16719 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 10 client.16719 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 20 client.16719 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5570bc58d2e0 0x5570bc58f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 4 client.16719 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa46024b740 1 -- 172.21.15.27:0/3011408139 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5570bc477f00 con 0x5570bc58d2e0 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.744+0000 7fa456ffd640 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5570bc58d2e0 0x5570bc58f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa456ffd640 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5570bc58d2e0 0x5570bc58f700 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=999405c523c024ab server_cookie=126828fb9f14dfbb in_seq=0 out_seq=0 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa43e7fc640 10 client.16719 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa43e7fc640 1 -- 172.21.15.27:0/3011408139 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5570bc477f00 con 0x5570bc58d2e0 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa43e7fc640 10 client.16719 handle_command_reply: tid=0 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 client.16719 shutdown 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 -- 172.21.15.27:0/3011408139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 msgr2=0x5570bc51e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc51e9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa44000be60 tx=0x7fa44000b9d0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 -- 172.21.15.27:0/3011408139 shutdown_connections 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5570bc58d2e0 0x5570bc58f700 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5570bc516570 0x5570bc50ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5570bc50bea0 0x5570bc50fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa43e7fc640 0 client.16719 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 --2- 172.21.15.27:0/3011408139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5570bc50b580 0x5570bc51e9c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 -- 172.21.15.27:0/3011408139 >> 172.21.15.27:0/3011408139 conn(0x5570bc5078a0 msgr2=0x5570bc50ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 -- 172.21.15.27:0/3011408139 shutdown_connections 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 1 -- 172.21.15.27:0/3011408139 wait complete. 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.745+0000 7fa46024b740 20 client.16719 trim_cache size 0 max 16384 2024-04-25T01:30:22.729 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 -- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa4440a2d00 msgr2=0x7fa4440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa4440a2d00 0x7fa4440a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa4500ee540 tx=0x7fa45008f220 comp rx=0 tx=0).stop 2024-04-25T01:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 -- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458107720 msgr2=0x7fa4580744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458107720 0x7fa4580744c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa450005cc0 tx=0x7fa450032f40 comp rx=0 tx=0).stop 2024-04-25T01:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 -- 172.21.15.27:0/900264079 shutdown_connections 2024-04-25T01:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa4440a2d00 0x7fa4440a51c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458107720 0x7fa4580744c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa4581020d0 0x7fa458073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 --2- 172.21.15.27:0/900264079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa458101710 0x7fa458073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 -- 172.21.15.27:0/900264079 >> 172.21.15.27:0/900264079 conn(0x7fa4580fd180 msgr2=0x7fa4580fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 -- 172.21.15.27:0/900264079 shutdown_connections 2024-04-25T01:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.746+0000 7fa45d548640 1 -- 172.21.15.27:0/900264079 wait complete. 2024-04-25T01:30:22.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:30:22.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:22 smithi153 ceph-mon[32621]: pgmap v151: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1785702843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f41002e0 msgr2=0x7fd2f41006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1785702843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f41002e0 0x7fd2f41006c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009900 tx=0x7fd2f002a410 comp rx=0 tx=0).stop 2024-04-25T01:30:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1785702843 shutdown_connections 2024-04-25T01:30:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1785702843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2f4106aa0 0x7fd2f410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1785702843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f4100c90 0x7fd2f41063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1785702843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f41002e0 0x7fd2f41006c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1785702843 >> 172.21.15.27:0/1785702843 conn(0x7fd2f40758d0 msgr2=0x7fd2f4075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1785702843 shutdown_connections 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1785702843 wait complete. 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 Processor -- start 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.901+0000 7fd2fc2a5640 1 -- start start 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fc2a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f41002e0 0x7fd2f4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fc2a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2f4106aa0 0x7fd2f41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fc2a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f419c950 0x7fd2f419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fc2a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f4111b90 con 0x7fd2f419c950 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fc2a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f4111a10 con 0x7fd2f41002e0 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fc2a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f4111d10 con 0x7fd2f4106aa0 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f419c950 0x7fd2f419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f419c950 0x7fd2f419ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56310/0 (socket says 172.21.15.27:56310) 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 -- 172.21.15.27:0/1113236617 learned_addr learned my addr 172.21.15.27:0/1113236617 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2f9840640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2f4106aa0 0x7fd2f41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa041640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f41002e0 0x7fd2f4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 -- 172.21.15.27:0/1113236617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2f4106aa0 msgr2=0x7fd2f41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2f4106aa0 0x7fd2f41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 -- 172.21.15.27:0/1113236617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f41002e0 msgr2=0x7fd2f4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f41002e0 0x7fd2f4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 -- 172.21.15.27:0/1113236617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f419d440 con 0x7fd2f419c950 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fa842640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f419c950 0x7fd2f419ce00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00eaa0 tx=0x7fd2ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2eb7fe640 1 -- 172.21.15.27:0/1113236617 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec00cdb0 con 0x7fd2f419c950 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2eb7fe640 1 -- 172.21.15.27:0/1113236617 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2ec005020 con 0x7fd2f419c950 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1113236617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f419d730 con 0x7fd2f419c950 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2eb7fe640 1 -- 172.21.15.27:0/1113236617 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0116b0 con 0x7fd2f419c950 2024-04-25T01:30:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.902+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1113236617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f41a4f20 con 0x7fd2f419c950 2024-04-25T01:30:22.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2eb7fe640 1 -- 172.21.15.27:0/1113236617 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd2ec005560 con 0x7fd2f419c950 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2fefb9740 1 Processor -- start 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2fefb9740 1 -- start start 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 0x55f051bc1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bc3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f051b1b670 con 0x55f051bb6580 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f051ac3970 con 0x55f051bc1570 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f051b17630 con 0x55f051bb6ea0 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2f9840640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.905+0000 7fd2eb7fe640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd2e00a2d00 0x7fd2e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56320/0 (socket says 172.21.15.27:56320) 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 -- 172.21.15.27:0/1484726225 learned_addr learned my addr 172.21.15.27:0/1484726225 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2fa041640 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 0x55f051bc1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2fa842640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd2e00a2d00 0x7fd2e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bc3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2eb7fe640 1 -- 172.21.15.27:0/1113236617 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd2ec0cf9c0 con 0x7fd2f419c950 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 -- 172.21.15.27:0/1484726225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 msgr2=0x55f051bc1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 0x55f051bc1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 -- 172.21.15.27:0/1484726225 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 msgr2=0x55f051bc3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bc3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 -- 172.21.15.27:0/1484726225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f051ae7400 con 0x55f051bb6580 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2f9840640 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb6960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40058a0 tx=0x7fd2e400b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb79d93f8b9baa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2fa842640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd2e00a2d00 0x7fd2e00a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0cd380 tx=0x7fd2ec06db10 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2e8bfe640 1 -- 172.21.15.27:0/1484726225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4004030 con 0x55f051bb6580 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2e8bfe640 1 -- 172.21.15.27:0/1484726225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e4005500 con 0x55f051bb6580 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.906+0000 7fd2e8bfe640 1 -- 172.21.15.27:0/1484726225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4004be0 con 0x55f051bb6580 2024-04-25T01:30:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 -- 172.21.15.27:0/1484726225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 msgr2=0x55f051bb6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb6960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40058a0 tx=0x7fd2e400b820 comp rx=0 tx=0).stop 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 -- 172.21.15.27:0/1484726225 shutdown_connections 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bc3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 0x55f051bc1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/1484726225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb6960 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 -- 172.21.15.27:0/1484726225 >> 172.21.15.27:0/1484726225 conn(0x55f051bb28a0 msgr2=0x55f051bb2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 -- 172.21.15.27:0/1484726225 shutdown_connections 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 -- 172.21.15.27:0/1484726225 wait complete. 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.907+0000 7fd2fefb9740 1 -- start start 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fefb9740 1 Processor -- start 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6580 0x55f051bc0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6ea0 0x55f051bbe170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bbe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f051bc4360 con 0x55f051bb6ea0 2024-04-25T01:30:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55f051ad7050 con 0x55f051bc1570 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f051bc44e0 con 0x55f051bb6580 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fefb9740 10 client.0 _ll_register_callbacks cb 0x55f05182ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6ea0 0x55f051bbe170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6ea0 0x55f051bbe170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56330/0 (socket says 172.21.15.27:56330) 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 -- 172.21.15.27:0/462010102 learned_addr learned my addr 172.21.15.27:0/462010102 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2f9840640 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bbe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 -- 172.21.15.27:0/462010102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6580 msgr2=0x55f051bc0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6580 0x55f051bc0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 -- 172.21.15.27:0/462010102 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 msgr2=0x55f051bbe6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bbe6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 -- 172.21.15.27:0/462010102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f051aa8dc0 con 0x55f051bb6ea0 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2fa041640 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6ea0 0x55f051bbe170 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009900 tx=0x7fd2f0031850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b63fe9676664c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.908+0000 7fd2ddffb640 1 -- 172.21.15.27:0/462010102 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0035040 con 0x55f051bb6ea0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2ddffb640 1 -- 172.21.15.27:0/462010102 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd2f0004240 con 0x55f051bb6ea0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 1 -- 172.21.15.27:0/462010102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e4005500 con 0x55f051bb6ea0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2ddffb640 1 -- 172.21.15.27:0/462010102 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2f0004440 con 0x55f051bb6ea0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 10 client.16731 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 1 -- 172.21.15.27:0/462010102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f05182f3f0 con 0x55f051bb6ea0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2ddffb640 1 -- 172.21.15.27:0/462010102 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e4005500 con 0x55f051bb6ea0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2ddffb640 1 -- 172.21.15.27:0/462010102 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd2f0004b10 con 0x55f051bb6ea0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 10 client.16731 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 10 client.16731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 20 client.16731 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55f051bd15c0 0x55f051bd39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 4 client.16731 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2fefb9740 1 -- 172.21.15.27:0/462010102 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f051b28bb0 con 0x55f051bd15c0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.909+0000 7fd2f9840640 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55f051bd15c0 0x55f051bd39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.910+0000 7fd2f9840640 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55f051bd15c0 0x55f051bd39e0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8aeb62171b3e79b0 server_cookie=1187ff2628e0869e in_seq=0 out_seq=0 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.910+0000 7fd2ddffb640 10 client.16731 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2ddffb640 1 -- 172.21.15.27:0/462010102 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f051b28bb0 con 0x55f051bd15c0 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2ddffb640 10 client.16731 handle_command_reply: tid=0 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 client.16731 shutdown 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 -- 172.21.15.27:0/462010102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6ea0 msgr2=0x55f051bbe170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6ea0 0x55f051bbe170 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009900 tx=0x7fd2f0031850 comp rx=0 tx=0).stop 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 -- 172.21.15.27:0/462010102 shutdown_connections 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55f051bd15c0 0x55f051bd39e0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bbe6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6ea0 0x55f051bbe170 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/462010102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6580 0x55f051bc0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2ddffb640 0 client.16731 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 -- 172.21.15.27:0/462010102 >> 172.21.15.27:0/462010102 conn(0x55f051bb28a0 msgr2=0x55f051bb86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 -- 172.21.15.27:0/462010102 shutdown_connections 2024-04-25T01:30:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 1 -- 172.21.15.27:0/462010102 wait complete. 2024-04-25T01:30:22.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.911+0000 7fd2fefb9740 20 client.16731 trim_cache size 0 max 16384 2024-04-25T01:30:22.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.924+0000 7fd2fefb9740 1 Processor -- start 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fefb9740 1 -- start start 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bb6ea0 0x55f051bc09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bc1570 0x55f051bd9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f41b14e0 con 0x55f051bb6580 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f41aedb0 con 0x55f051bb6ea0 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f41acc00 con 0x55f051bc1570 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56346/0 (socket says 172.21.15.27:56346) 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 -- 172.21.15.27:0/383588399 learned_addr learned my addr 172.21.15.27:0/383588399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fa041640 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bb6ea0 0x55f051bc09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 -- 172.21.15.27:0/383588399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bc1570 msgr2=0x55f051bd9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bc1570 0x55f051bd9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 -- 172.21.15.27:0/383588399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bb6ea0 msgr2=0x55f051bc09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bb6ea0 0x55f051bc09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 -- 172.21.15.27:0/383588399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0518f97d0 con 0x55f051bb6580 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2fa041640 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bb6ea0 0x55f051bc09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2f9840640 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb88b0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400b3d0 tx=0x7fd2e400b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca79239e044185d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2dffff640 1 -- 172.21.15.27:0/383588399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e40120b0 con 0x55f051bb6580 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2dffff640 1 -- 172.21.15.27:0/383588399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e402a560 con 0x55f051bb6580 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.925+0000 7fd2dffff640 1 -- 172.21.15.27:0/383588399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e40107c0 con 0x55f051bb6580 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 -- 172.21.15.27:0/383588399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 msgr2=0x55f051bb88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb88b0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400b3d0 tx=0x7fd2e400b770 comp rx=0 tx=0).stop 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 -- 172.21.15.27:0/383588399 shutdown_connections 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bc1570 0x55f051bd9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bb6ea0 0x55f051bc09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/383588399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bb88b0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 -- 172.21.15.27:0/383588399 >> 172.21.15.27:0/383588399 conn(0x55f051bb28a0 msgr2=0x55f051bb1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 -- 172.21.15.27:0/383588399 shutdown_connections 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 -- 172.21.15.27:0/383588399 wait complete. 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 -- start start 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.926+0000 7fd2fefb9740 1 Processor -- start 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bc9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 0x55f051bc9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bc6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f41ad270 con 0x55f051bb6580 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f41ae6e0 con 0x55f051bc1570 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f41ad940 con 0x55f051bb6ea0 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 10 client.0 _ll_register_callbacks cb 0x7fd2f00314d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bc9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bc9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56354/0 (socket says 172.21.15.27:56354) 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 -- 172.21.15.27:0/2230473909 learned_addr learned my addr 172.21.15.27:0/2230473909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 -- 172.21.15.27:0/2230473909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 msgr2=0x55f051bc9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 0x55f051bc9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 -- 172.21.15.27:0/2230473909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 msgr2=0x55f051bc6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bc6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 -- 172.21.15.27:0/2230473909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f051bc6f10 con 0x55f051bb6580 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2f9840640 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bc9730 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4014040 tx=0x7fd2e40121b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b4c731ea3185e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2ddffb640 1 -- 172.21.15.27:0/2230473909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4030070 con 0x55f051bb6580 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2ddffb640 1 -- 172.21.15.27:0/2230473909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd2e402a090 con 0x55f051bb6580 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 1 -- 172.21.15.27:0/2230473909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e402a560 con 0x55f051bb6580 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2ddffb640 1 -- 172.21.15.27:0/2230473909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2e403c050 con 0x55f051bb6580 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 10 client.16737 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.927+0000 7fd2fefb9740 1 -- 172.21.15.27:0/2230473909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f051c38580 con 0x55f051bb6580 2024-04-25T01:30:22.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2ddffb640 1 -- 172.21.15.27:0/2230473909 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e402a560 con 0x55f051bb6580 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2ddffb640 1 -- 172.21.15.27:0/2230473909 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd2e401c020 con 0x55f051bb6580 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2fefb9740 10 client.16737 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2fefb9740 10 client.16737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2fefb9740 20 client.16737 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55f051c389b0 0x55f051c3ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2fefb9740 4 client.16737 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2fefb9740 1 -- 172.21.15.27:0/2230473909 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f051b22f00 con 0x55f051c389b0 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2fa041640 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55f051c389b0 0x55f051c3ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2fa041640 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55f051c389b0 0x55f051c3ada0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cad98513f202b8a server_cookie=1d79cd19d929f767 in_seq=0 out_seq=0 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.928+0000 7fd2ddffb640 10 client.16737 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2ddffb640 1 -- 172.21.15.27:0/2230473909 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f051b22f00 con 0x55f051c389b0 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2ddffb640 10 client.16737 handle_command_reply: tid=0 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 client.16737 shutdown 2024-04-25T01:30:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 -- 172.21.15.27:0/2230473909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 msgr2=0x55f051bc9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bc9730 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4014040 tx=0x7fd2e40121b0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 -- 172.21.15.27:0/2230473909 shutdown_connections 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55f051c389b0 0x55f051c3ada0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f051bc1570 0x55f051bc6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f051bb6ea0 0x55f051bc9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2ddffb640 0 client.16737 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 --2- 172.21.15.27:0/2230473909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f051bb6580 0x55f051bc9730 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 -- 172.21.15.27:0/2230473909 >> 172.21.15.27:0/2230473909 conn(0x55f051bb28a0 msgr2=0x55f051bb1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 -- 172.21.15.27:0/2230473909 shutdown_connections 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 1 -- 172.21.15.27:0/2230473909 wait complete. 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fefb9740 20 client.16737 trim_cache size 0 max 16384 2024-04-25T01:30:22.913 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:30:22.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd2e00a2d00 msgr2=0x7fd2e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd2e00a2d00 0x7fd2e00a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0cd380 tx=0x7fd2ec06db10 comp rx=0 tx=0).stop 2024-04-25T01:30:22.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f419c950 msgr2=0x7fd2f419ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.929+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f419c950 0x7fd2f419ce00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00eaa0 tx=0x7fd2ec00ef60 comp rx=0 tx=0).stop 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.930+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1113236617 shutdown_connections 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.930+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd2e00a2d00 0x7fd2e00a51c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.930+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f419c950 0x7fd2f419ce00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.930+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2f4106aa0 0x7fd2f41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.930+0000 7fd2fc2a5640 1 --2- 172.21.15.27:0/1113236617 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f41002e0 0x7fd2f4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.930+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1113236617 >> 172.21.15.27:0/1113236617 conn(0x7fd2f40758d0 msgr2=0x7fd2f40fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.930+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1113236617 shutdown_connections 2024-04-25T01:30:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:22.930+0000 7fd2fc2a5640 1 -- 172.21.15.27:0/1113236617 wait complete. 2024-04-25T01:30:23.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:23 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1463907408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:23.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:23 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1463907408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:23.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:23 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1463907408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:23.918 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:30:23.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:23.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:30:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 -- 172.21.15.27:0/4055979964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098107730 msgr2=0x7f6098107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 --2- 172.21.15.27:0/4055979964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098107730 0x7f6098107b10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f608c009930 tx=0x7f608c02a420 comp rx=0 tx=0).stop 2024-04-25T01:30:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 -- 172.21.15.27:0/4055979964 shutdown_connections 2024-04-25T01:30:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 --2- 172.21.15.27:0/4055979964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f609806a780 0x7f609810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 --2- 172.21.15.27:0/4055979964 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6098069dc0 0x7f609806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 --2- 172.21.15.27:0/4055979964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098107730 0x7f6098107b10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 -- 172.21.15.27:0/4055979964 >> 172.21.15.27:0/4055979964 conn(0x7f60980776a0 msgr2=0x7f6098077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 -- 172.21.15.27:0/4055979964 shutdown_connections 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.060+0000 7f60a0512640 1 -- 172.21.15.27:0/4055979964 wait complete. 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f60a0512640 1 Processor -- start 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f60a0512640 1 -- start start 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f60a0512640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098069dc0 0x7f6098100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f60a0512640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f609806a780 0x7f6098100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f60a0512640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6098107730 0x7f60981010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f60a0512640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6098111bc0 con 0x7f6098069dc0 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f60a0512640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6098111a40 con 0x7f6098107730 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f60a0512640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6098111d40 con 0x7f609806a780 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609e2ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098069dc0 0x7f6098100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609e2ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098069dc0 0x7f6098100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56380/0 (socket says 172.21.15.27:56380) 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609e2ae640 1 -- 172.21.15.27:0/2583767994 learned_addr learned my addr 172.21.15.27:0/2583767994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609eaaf640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6098107730 0x7f60981010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609daad640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f609806a780 0x7f6098100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609e2ae640 1 -- 172.21.15.27:0/2583767994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f609806a780 msgr2=0x7f6098100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609e2ae640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f609806a780 0x7f6098100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609e2ae640 1 -- 172.21.15.27:0/2583767994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6098107730 msgr2=0x7f60981010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609e2ae640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6098107730 0x7f60981010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.061+0000 7f609e2ae640 1 -- 172.21.15.27:0/2583767994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60981a66a0 con 0x7f6098069dc0 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.062+0000 7f609e2ae640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098069dc0 0x7f6098100670 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f608c009a50 tx=0x7f608c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.062+0000 7f608b7fe640 1 -- 172.21.15.27:0/2583767994 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c03f030 con 0x7f6098069dc0 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.062+0000 7f608b7fe640 1 -- 172.21.15.27:0/2583767994 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f608c003ce0 con 0x7f6098069dc0 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.062+0000 7f60a0512640 1 -- 172.21.15.27:0/2583767994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60981a6930 con 0x7f6098069dc0 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.062+0000 7f608b7fe640 1 -- 172.21.15.27:0/2583767994 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c035070 con 0x7f6098069dc0 2024-04-25T01:30:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.062+0000 7f60a0512640 1 -- 172.21.15.27:0/2583767994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60981a6e70 con 0x7f6098069dc0 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.064+0000 7f608b7fe640 1 -- 172.21.15.27:0/2583767994 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f608c042030 con 0x7f6098069dc0 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f60a3217740 1 Processor -- start 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f60a3217740 1 -- start start 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f60a3217740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c0310cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f60a3217740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0311710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f60a3217740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c031e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f60a3217740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5565c02752c0 con 0x5565c0312980 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f60a3217740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5565c021d910 con 0x5565c0311290 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f60a3217740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5565c0271280 con 0x5565c031c010 2024-04-25T01:30:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f608b7fe640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60840a2d80 0x7f60840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f609eaaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c0310cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f609eaaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c0310cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56392/0 (socket says 172.21.15.27:56392) 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f609eaaf640 1 -- 172.21.15.27:0/2537799044 learned_addr learned my addr 172.21.15.27:0/2537799044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f609eaaf640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60840a2d80 0x7f60840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f609e2ae640 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0311710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f608b7fe640 1 -- 172.21.15.27:0/2583767994 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f608c0efcd0 con 0x7f6098069dc0 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.065+0000 7f609daad640 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c031e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f609eaaf640 1 -- 172.21.15.27:0/2537799044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 msgr2=0x5565c031e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f609eaaf640 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c031e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f609eaaf640 1 -- 172.21.15.27:0/2537799044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 msgr2=0x5565c0311710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f609eaaf640 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0311710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f609eaaf640 1 -- 172.21.15.27:0/2537799044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c022efb0 con 0x5565c0312980 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f609e2ae640 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0311710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f609eaaf640 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c0310cc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6090008340 tx=0x7f6090008800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ef794c5a4ca0ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f609eaaf640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60840a2d80 0x7f60840a5240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6090009900 tx=0x7f609001b000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f6088bfe640 1 -- 172.21.15.27:0/2537799044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6090027070 con 0x5565c0312980 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f6088bfe640 1 -- 172.21.15.27:0/2537799044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6090019d30 con 0x5565c0312980 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.066+0000 7f6088bfe640 1 -- 172.21.15.27:0/2537799044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6090014040 con 0x5565c0312980 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 -- 172.21.15.27:0/2537799044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 msgr2=0x5565c0310cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c0310cc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6090008340 tx=0x7f6090008800 comp rx=0 tx=0).stop 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 -- 172.21.15.27:0/2537799044 shutdown_connections 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c031e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0311710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 --2- 172.21.15.27:0/2537799044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c0310cc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 -- 172.21.15.27:0/2537799044 >> 172.21.15.27:0/2537799044 conn(0x5565c02d9950 msgr2=0x5565c030d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 -- 172.21.15.27:0/2537799044 shutdown_connections 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 -- 172.21.15.27:0/2537799044 wait complete. 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.067+0000 7f60a3217740 1 -- start start 2024-04-25T01:30:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f60a3217740 1 Processor -- start 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f60a3217740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 0x5565c0318f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f60a3217740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0312980 0x5565c0319480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f60a3217740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c031c010 0x5565c03199c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f60a3217740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5565c031ee40 con 0x5565c031c010 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f60a3217740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5565c0230e90 con 0x5565c0312980 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f60a3217740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5565c031efc0 con 0x5565c0311290 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f60a3217740 10 client.0 _ll_register_callbacks cb 0x5565c026c970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c031c010 0x5565c03199c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c031c010 0x5565c03199c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56400/0 (socket says 172.21.15.27:56400) 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 -- 172.21.15.27:0/1614508456 learned_addr learned my addr 172.21.15.27:0/1614508456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609daad640 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0312980 0x5565c0319480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609e2ae640 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 0x5565c0318f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 -- 172.21.15.27:0/1614508456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 msgr2=0x5565c0318f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 0x5565c0318f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 -- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0312980 msgr2=0x5565c0319480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0312980 0x5565c0319480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 -- 172.21.15.27:0/1614508456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c00517e0 con 0x5565c031c010 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609daad640 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0312980 0x5565c0319480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f609eaaf640 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c031c010 0x5565c03199c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6090022a30 tx=0x7f6090019ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ac3f749446c142 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.068+0000 7f6081ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f6081ffb640 1 -- 172.21.15.27:0/1614508456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6090027040 con 0x5565c031c010 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f6081ffb640 1 -- 172.21.15.27:0/1614508456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6090022ad0 con 0x5565c031c010 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 1 -- 172.21.15.27:0/1614508456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6090019d30 con 0x5565c031c010 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f6081ffb640 1 -- 172.21.15.27:0/1614508456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f609003b050 con 0x5565c031c010 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 10 client.16749 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 1 -- 172.21.15.27:0/1614508456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5565c00b1f70 con 0x5565c031c010 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f6081ffb640 1 -- 172.21.15.27:0/1614508456 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6090019d30 con 0x5565c031c010 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f6081ffb640 1 -- 172.21.15.27:0/1614508456 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f609002c480 con 0x5565c031c010 2024-04-25T01:30:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 10 client.16749 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 10 client.16749 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 20 client.16749 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5565c032c0b0 0x5565c032e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 4 client.16749 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.069+0000 7f60a3217740 1 -- 172.21.15.27:0/1614508456 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565c02495a0 con 0x5565c032c0b0 2024-04-25T01:30:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.070+0000 7f609e2ae640 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5565c032c0b0 0x5565c032e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.070+0000 7f609e2ae640 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5565c032c0b0 0x5565c032e4a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7936ed8425a41b3 server_cookie=a53ec0c8e0cd0a25 in_seq=0 out_seq=0 2024-04-25T01:30:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.070+0000 7f6081ffb640 10 client.16749 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f6081ffb640 1 -- 172.21.15.27:0/1614508456 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565c02495a0 con 0x5565c032c0b0 2024-04-25T01:30:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f6081ffb640 10 client.16749 handle_command_reply: tid=0 2024-04-25T01:30:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 client.16749 shutdown 2024-04-25T01:30:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 -- 172.21.15.27:0/1614508456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c031c010 msgr2=0x5565c03199c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c031c010 0x5565c03199c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6090022a30 tx=0x7f6090019ee0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 -- 172.21.15.27:0/1614508456 shutdown_connections 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5565c032c0b0 0x5565c032e4a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c031c010 0x5565c03199c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f6081ffb640 0 client.16749 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0312980 0x5565c0319480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 --2- 172.21.15.27:0/1614508456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 0x5565c0318f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 -- 172.21.15.27:0/1614508456 >> 172.21.15.27:0/1614508456 conn(0x5565c02d9950 msgr2=0x5565c03135f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 -- 172.21.15.27:0/1614508456 shutdown_connections 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 1 -- 172.21.15.27:0/1614508456 wait complete. 2024-04-25T01:30:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.071+0000 7f60a3217740 20 client.16749 trim_cache size 0 max 16384 2024-04-25T01:30:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.089+0000 7f60a3217740 1 Processor -- start 2024-04-25T01:30:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.089+0000 7f60a3217740 1 -- start start 2024-04-25T01:30:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.089+0000 7f60a3217740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0327d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.089+0000 7f60a3217740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c03197e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.089+0000 7f60a3217740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c030c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.089+0000 7f60a3217740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f60981b3800 con 0x5565c0312980 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.089+0000 7f60a3217740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f60981b10d0 con 0x5565c0311290 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f60a3217740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60981aef20 con 0x5565c031c010 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c03197e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c03197e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56408/0 (socket says 172.21.15.27:56408) 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 -- 172.21.15.27:0/3326137982 learned_addr learned my addr 172.21.15.27:0/3326137982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609eaaf640 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c030c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 -- 172.21.15.27:0/3326137982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 msgr2=0x5565c030c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c030c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 -- 172.21.15.27:0/3326137982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 msgr2=0x5565c0327d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609e2ae640 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0327d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0327d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 -- 172.21.15.27:0/3326137982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565bff49170 con 0x5565c0312980 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609eaaf640 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c030c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f609daad640 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c03197e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f609400b890 tx=0x7f609400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cd15a0fd5d0200 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f6083fff640 1 -- 172.21.15.27:0/3326137982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094019070 con 0x5565c0312980 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f6083fff640 1 -- 172.21.15.27:0/3326137982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6094011d20 con 0x5565c0312980 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.090+0000 7f6083fff640 1 -- 172.21.15.27:0/3326137982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60940143e0 con 0x5565c0312980 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 -- 172.21.15.27:0/3326137982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 msgr2=0x5565c03197e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c03197e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f609400b890 tx=0x7f609400bd50 comp rx=0 tx=0).stop 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 -- 172.21.15.27:0/3326137982 shutdown_connections 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c031c010 0x5565c030c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c03197e0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 --2- 172.21.15.27:0/3326137982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c0311290 0x5565c0327d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 -- 172.21.15.27:0/3326137982 >> 172.21.15.27:0/3326137982 conn(0x5565c02d9950 msgr2=0x5565c030c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 -- 172.21.15.27:0/3326137982 shutdown_connections 2024-04-25T01:30:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 -- 172.21.15.27:0/3326137982 wait complete. 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 -- start start 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.091+0000 7f60a3217740 1 Processor -- start 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f60a3217740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 0x5565c03245e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f60a3217740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c031f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f60a3217740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c031c010 0x5565c031fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f60a3217740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f60981af590 con 0x5565c0312980 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f60a3217740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f60981b0a00 con 0x5565c031c010 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f60a3217740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60981afc60 con 0x5565c0311290 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f60a3217740 10 client.0 _ll_register_callbacks cb 0x7f6090025070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c031f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c031f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56418/0 (socket says 172.21.15.27:56418) 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 -- 172.21.15.27:0/1142088383 learned_addr learned my addr 172.21.15.27:0/1142088383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609eaaf640 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c031c010 0x5565c031fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609e2ae640 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 0x5565c03245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 -- 172.21.15.27:0/1142088383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 msgr2=0x5565c03245e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 0x5565c03245e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 -- 172.21.15.27:0/1142088383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c031c010 msgr2=0x5565c031fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c031c010 0x5565c031fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 -- 172.21.15.27:0/1142088383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c03206a0 con 0x5565c0312980 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f609daad640 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c031f950 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6094006e10 tx=0x7f6094016e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0d69d52f09359e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f6081ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f6081ffb640 1 -- 172.21.15.27:0/1142088383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094019040 con 0x5565c0312980 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f6081ffb640 1 -- 172.21.15.27:0/1142088383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f609401e450 con 0x5565c0312980 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.092+0000 7f60a3217740 1 -- 172.21.15.27:0/1142088383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6094011d20 con 0x5565c0312980 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f6081ffb640 1 -- 172.21.15.27:0/1142088383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f609401e650 con 0x5565c0312980 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f60a3217740 10 client.16755 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f60a3217740 1 -- 172.21.15.27:0/1142088383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5565c0392340 con 0x5565c0312980 2024-04-25T01:30:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f6081ffb640 1 -- 172.21.15.27:0/1142088383 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6094011d20 con 0x5565c0312980 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f6081ffb640 1 -- 172.21.15.27:0/1142088383 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6094029840 con 0x5565c0312980 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f60a3217740 10 client.16755 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f60a3217740 10 client.16755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f60a3217740 20 client.16755 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f60a3217740 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5565c0392950 0x5565c0394d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f60a3217740 4 client.16755 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f60a3217740 1 -- 172.21.15.27:0/1142088383 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5565c0282800 con 0x5565c0392950 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.093+0000 7f609e2ae640 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5565c0392950 0x5565c0394d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.094+0000 7f609e2ae640 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5565c0392950 0x5565c0394d70 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c892d6451e1a9d2c server_cookie=7ac682cb9251b781 in_seq=0 out_seq=0 2024-04-25T01:30:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.094+0000 7f6081ffb640 10 client.16755 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:24.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f6081ffb640 1 -- 172.21.15.27:0/1142088383 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5565c0282800 con 0x5565c0392950 2024-04-25T01:30:24.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f6081ffb640 10 client.16755 handle_command_reply: tid=0 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 client.16755 shutdown 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 -- 172.21.15.27:0/1142088383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 msgr2=0x5565c031f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c031f950 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6094006e10 tx=0x7f6094016e70 comp rx=0 tx=0).stop 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 -- 172.21.15.27:0/1142088383 shutdown_connections 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5565c0392950 0x5565c0394d70 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5565c031c010 0x5565c031fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565c0312980 0x5565c031f950 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f6081ffb640 0 client.16755 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 --2- 172.21.15.27:0/1142088383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5565c0311290 0x5565c03245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 -- 172.21.15.27:0/1142088383 >> 172.21.15.27:0/1142088383 conn(0x5565c02d9950 msgr2=0x5565c0312d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 -- 172.21.15.27:0/1142088383 shutdown_connections 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 1 -- 172.21.15.27:0/1142088383 wait complete. 2024-04-25T01:30:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.095+0000 7f60a3217740 20 client.16755 trim_cache size 0 max 16384 2024-04-25T01:30:24.079 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:30:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:30:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "53e106bf-9132-45bc-b667-f3add96bd116", 2024-04-25T01:30:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:30:24.079 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 -- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60840a2d80 msgr2=0x7f60840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60840a2d80 0x7f60840a5240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6090009900 tx=0x7f609001b000 comp rx=0 tx=0).stop 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 -- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098069dc0 msgr2=0x7f6098100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098069dc0 0x7f6098100670 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f608c009a50 tx=0x7f608c004270 comp rx=0 tx=0).stop 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 -- 172.21.15.27:0/2583767994 shutdown_connections 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6098107730 0x7f60981010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f609806a780 0x7f6098100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f60840a2d80 0x7f60840a5240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 --2- 172.21.15.27:0/2583767994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6098069dc0 0x7f6098100670 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 -- 172.21.15.27:0/2583767994 >> 172.21.15.27:0/2583767994 conn(0x7f60980776a0 msgr2=0x7f60980fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 -- 172.21.15.27:0/2583767994 shutdown_connections 2024-04-25T01:30:24.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.096+0000 7f60a0512640 1 -- 172.21.15.27:0/2583767994 wait complete. 2024-04-25T01:30:24.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:30:24.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 -- 172.21.15.27:0/3323089641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc100fe0 msgr2=0x7f2afc10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 --2- 172.21.15.27:0/3323089641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc100fe0 0x7f2afc10d4f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009cd0 tx=0x7f2aec02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 -- 172.21.15.27:0/3323089641 shutdown_connections 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 --2- 172.21.15.27:0/3323089641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc100fe0 0x7f2afc10d4f0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 --2- 172.21.15.27:0/3323089641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2afc100620 0x7f2afc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 --2- 172.21.15.27:0/3323089641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2afc106630 0x7f2afc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 -- 172.21.15.27:0/3323089641 >> 172.21.15.27:0/3323089641 conn(0x7f2afc0776a0 msgr2=0x7f2afc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 -- 172.21.15.27:0/3323089641 shutdown_connections 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.213+0000 7f2b042da640 1 -- 172.21.15.27:0/3323089641 wait complete. 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b042da640 1 Processor -- start 2024-04-25T01:30:24.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b042da640 1 -- start start 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b042da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2afc100620 0x7f2afc197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b042da640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2afc100fe0 0x7f2afc1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b042da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc106630 0x7f2afc19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b042da640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc110a40 con 0x7f2afc106630 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b042da640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc1108c0 con 0x7f2afc100fe0 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b042da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc110bc0 con 0x7f2afc100620 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02877640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc106630 0x7f2afc19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02877640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc106630 0x7f2afc19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56442/0 (socket says 172.21.15.27:56442) 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02877640 1 -- 172.21.15.27:0/1248858486 learned_addr learned my addr 172.21.15.27:0/1248858486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b01875640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2afc100fe0 0x7f2afc1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02076640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2afc100620 0x7f2afc197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02877640 1 -- 172.21.15.27:0/1248858486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2afc100620 msgr2=0x7f2afc197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02877640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2afc100620 0x7f2afc197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02877640 1 -- 172.21.15.27:0/1248858486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2afc100fe0 msgr2=0x7f2afc1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02877640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2afc100fe0 0x7f2afc1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02877640 1 -- 172.21.15.27:0/1248858486 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2afc19cf50 con 0x7f2afc106630 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b01875640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2afc100fe0 0x7f2afc1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.214+0000 7f2b02076640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2afc100620 0x7f2afc197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.215+0000 7f2b02877640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc106630 0x7f2afc19c820 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2aec005cc0 tx=0x7f2aec003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.215+0000 7f2afb7fe640 1 -- 172.21.15.27:0/1248858486 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec004040 con 0x7f2afc106630 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.215+0000 7f2afb7fe640 1 -- 172.21.15.27:0/1248858486 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aec0046c0 con 0x7f2afc106630 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.215+0000 7f2b042da640 1 -- 172.21.15.27:0/1248858486 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2afc19d1e0 con 0x7f2afc106630 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.215+0000 7f2afb7fe640 1 -- 172.21.15.27:0/1248858486 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec046480 con 0x7f2afc106630 2024-04-25T01:30:24.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.215+0000 7f2b042da640 1 -- 172.21.15.27:0/1248858486 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2afc1a4ad0 con 0x7f2afc106630 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.217+0000 7f2b06fee740 1 Processor -- start 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.217+0000 7f2afb7fe640 1 -- 172.21.15.27:0/1248858486 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2aec0305a0 con 0x7f2afc106630 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b06fee740 1 -- start start 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df595a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df5a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b06fee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b4df4fa6b0 con 0x55b4df595630 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b06fee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b4df4a2880 con 0x55b4df595f50 2024-04-25T01:30:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b06fee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b4df4f6670 con 0x55b4df5a0620 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b01875640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df595a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b01875640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df595a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56456/0 (socket says 172.21.15.27:56456) 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b01875640 1 -- 172.21.15.27:0/293521271 learned_addr learned my addr 172.21.15.27:0/293521271 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b02076640 1 --2- 172.21.15.27:0/293521271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2afb7fe640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2ae40a2d00 0x7f2ae40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b01875640 1 -- 172.21.15.27:0/293521271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 msgr2=0x55b4df5a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b01875640 1 --2- 172.21.15.27:0/293521271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df5a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b01875640 1 -- 172.21.15.27:0/293521271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 msgr2=0x55b4df5a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b01875640 1 --2- 172.21.15.27:0/293521271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b01875640 1 -- 172.21.15.27:0/293521271 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4df2d87d0 con 0x55b4df595630 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.218+0000 7f2b02877640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2ae40a2d00 0x7f2ae40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.219+0000 7f2afb7fe640 1 -- 172.21.15.27:0/1248858486 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2aec02e030 con 0x7f2afc106630 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.219+0000 7f2b01875640 1 --2- 172.21.15.27:0/293521271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df595a10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2af0008cf0 tx=0x7f2af0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b5a782833ae14e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.219+0000 7f2af8bfe640 1 -- 172.21.15.27:0/293521271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0016070 con 0x55b4df595630 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.219+0000 7f2af8bfe640 1 -- 172.21.15.27:0/293521271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2af0005700 con 0x55b4df595630 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.219+0000 7f2af8bfe640 1 -- 172.21.15.27:0/293521271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0012720 con 0x55b4df595630 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- 172.21.15.27:0/293521271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 msgr2=0x55b4df595a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b02877640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2ae40a2d00 0x7f2ae40a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0bcf80 tx=0x7f2aec0095c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 --2- 172.21.15.27:0/293521271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df595a10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2af0008cf0 tx=0x7f2af0010740 comp rx=0 tx=0).stop 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- 172.21.15.27:0/293521271 shutdown_connections 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 --2- 172.21.15.27:0/293521271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df5a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 --2- 172.21.15.27:0/293521271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 --2- 172.21.15.27:0/293521271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df595a10 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- 172.21.15.27:0/293521271 >> 172.21.15.27:0/293521271 conn(0x55b4df591950 msgr2=0x55b4df591d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- 172.21.15.27:0/293521271 shutdown_connections 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- 172.21.15.27:0/293521271 wait complete. 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- start start 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 Processor -- start 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df59d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df595f50 0x55b4df59da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df5a0620 0x55b4df59dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b4df5a3410 con 0x55b4df5a0620 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b4df4b5f50 con 0x55b4df595630 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.220+0000 7f2b06fee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b4df5a3590 con 0x55b4df595f50 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b06fee740 10 client.0 _ll_register_callbacks cb 0x55b4df4f1000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df5a0620 0x55b4df59dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df5a0620 0x55b4df59dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56460/0 (socket says 172.21.15.27:56460) 2024-04-25T01:30:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 -- 172.21.15.27:0/299813262 learned_addr learned my addr 172.21.15.27:0/299813262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df59d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b02076640 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df595f50 0x55b4df59da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 -- 172.21.15.27:0/299813262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df595f50 msgr2=0x55b4df59da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df595f50 0x55b4df59da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 -- 172.21.15.27:0/299813262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 msgr2=0x55b4df59d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df59d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 -- 172.21.15.27:0/299813262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4df20e3f0 con 0x55b4df5a0620 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b02076640 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df595f50 0x55b4df59da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b01875640 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df5a0620 0x55b4df59dfa0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f2af0008470 tx=0x7f2af0004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec7d22a2650ee33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2ae1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/299813262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af001f070 con 0x55b4df5a0620 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/299813262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2af00053b0 con 0x55b4df5a0620 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b06fee740 1 -- 172.21.15.27:0/299813262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2af0005700 con 0x55b4df5a0620 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/299813262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2af0031050 con 0x55b4df5a0620 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b06fee740 10 client.16767 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.221+0000 7f2b06fee740 1 -- 172.21.15.27:0/299813262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b4df1ce0f0 con 0x55b4df5a0620 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/299813262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2af0005700 con 0x55b4df5a0620 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/299813262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2af001a960 con 0x55b4df5a0620 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2b06fee740 10 client.16767 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2b06fee740 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2b06fee740 20 client.16767 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2b06fee740 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b4df5b07f0 0x55b4df5b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2b06fee740 4 client.16767 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2b06fee740 1 -- 172.21.15.27:0/299813262 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4df507bf0 con 0x55b4df5b07f0 2024-04-25T01:30:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.222+0000 7f2b02076640 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b4df5b07f0 0x55b4df5b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b02076640 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b4df5b07f0 0x55b4df5b2be0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc31c63e1ed8bbec server_cookie=502e58506498119d in_seq=0 out_seq=0 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2ae1ffb640 10 client.16767 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/299813262 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4df507bf0 con 0x55b4df5b07f0 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2ae1ffb640 10 client.16767 handle_command_reply: tid=0 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 client.16767 shutdown 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 -- 172.21.15.27:0/299813262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df5a0620 msgr2=0x55b4df59dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df5a0620 0x55b4df59dfa0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f2af0008470 tx=0x7f2af0004df0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 -- 172.21.15.27:0/299813262 shutdown_connections 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b4df5b07f0 0x55b4df5b2be0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df5a0620 0x55b4df59dfa0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2ae1ffb640 0 client.16767 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df595f50 0x55b4df59da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 --2- 172.21.15.27:0/299813262 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df59d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.223+0000 7f2b06fee740 1 -- 172.21.15.27:0/299813262 >> 172.21.15.27:0/299813262 conn(0x55b4df591950 msgr2=0x55b4df597650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.224+0000 7f2b06fee740 1 -- 172.21.15.27:0/299813262 shutdown_connections 2024-04-25T01:30:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.224+0000 7f2b06fee740 1 -- 172.21.15.27:0/299813262 wait complete. 2024-04-25T01:30:24.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.224+0000 7f2b06fee740 20 client.16767 trim_cache size 0 max 16384 2024-04-25T01:30:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b06fee740 1 Processor -- start 2024-04-25T01:30:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b06fee740 1 -- start start 2024-04-25T01:30:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df5ac300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595f50 0x55b4df59d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df597d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b06fee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc1b13a0 con 0x55b4df595f50 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b06fee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc1aec70 con 0x55b4df595630 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b06fee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc1acac0 con 0x55b4df5a0620 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b02076640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595f50 0x55b4df59d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b02076640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595f50 0x55b4df59d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56468/0 (socket says 172.21.15.27:56468) 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.241+0000 7f2b02076640 1 -- 172.21.15.27:0/3980101972 learned_addr learned my addr 172.21.15.27:0/3980101972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b01875640 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df5ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b01875640 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df597d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b02076640 1 -- 172.21.15.27:0/3980101972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 msgr2=0x55b4df597d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b02076640 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df597d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b02076640 1 -- 172.21.15.27:0/3980101972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 msgr2=0x55b4df5ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b02076640 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df5ac300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b02076640 1 -- 172.21.15.27:0/3980101972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4df5b2f50 con 0x55b4df595f50 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b01875640 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df5ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b02076640 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595f50 0x55b4df59d4e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x55b4df59cd80 tx=0x7f2af4011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963809e81946fab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2ae3fff640 1 -- 172.21.15.27:0/3980101972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af4021030 con 0x55b4df595f50 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2ae3fff640 1 -- 172.21.15.27:0/3980101972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2af4005020 con 0x55b4df595f50 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2ae3fff640 1 -- 172.21.15.27:0/3980101972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af4004bc0 con 0x55b4df595f50 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b06fee740 1 -- 172.21.15.27:0/3980101972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595f50 msgr2=0x55b4df59d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.242+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595f50 0x55b4df59d4e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x55b4df59cd80 tx=0x7f2af4011980 comp rx=0 tx=0).stop 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 -- 172.21.15.27:0/3980101972 shutdown_connections 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df597d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595f50 0x55b4df59d4e0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3980101972 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595630 0x55b4df5ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 -- 172.21.15.27:0/3980101972 >> 172.21.15.27:0/3980101972 conn(0x55b4df591950 msgr2=0x55b4df59c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 -- 172.21.15.27:0/3980101972 shutdown_connections 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 -- 172.21.15.27:0/3980101972 wait complete. 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 -- start start 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 Processor -- start 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df5a8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df5a4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc1ad130 con 0x55b4df595630 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc1ae5a0 con 0x55b4df595f50 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2afc1ad800 con 0x55b4df5a0620 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b06fee740 10 client.0 _ll_register_callbacks cb 0x7f2af400b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b01875640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df5a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b01875640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df5a8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56480/0 (socket says 172.21.15.27:56480) 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b01875640 1 -- 172.21.15.27:0/3935182097 learned_addr learned my addr 172.21.15.27:0/3935182097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.243+0000 7f2b02076640 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b01875640 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df5a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b01875640 1 -- 172.21.15.27:0/3935182097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 msgr2=0x55b4df5a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b01875640 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df5a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b01875640 1 -- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 msgr2=0x55b4df5a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b01875640 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b01875640 1 -- 172.21.15.27:0/3935182097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4df5a4880 con 0x55b4df595630 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b02076640 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b01875640 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df5a8960 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2af0008b00 tx=0x7f2af0008310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef96abd75be0a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2ae1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/3935182097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af003e060 con 0x55b4df595630 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/3935182097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2af00055c0 con 0x55b4df595630 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b06fee740 1 -- 172.21.15.27:0/3935182097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2af4005020 con 0x55b4df595630 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/3935182097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2af001f050 con 0x55b4df595630 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b06fee740 10 client.16773 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2b06fee740 1 -- 172.21.15.27:0/3935182097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b4df6169c0 con 0x55b4df595630 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/3935182097 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2af4005020 con 0x55b4df595630 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.244+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/3935182097 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2af001a400 con 0x55b4df595630 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b06fee740 10 client.16773 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b06fee740 10 client.16773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b06fee740 20 client.16773 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b4df617050 0x55b4df619470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b06fee740 4 client.16773 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:30:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b06fee740 1 -- 172.21.15.27:0/3935182097 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b4df501f40 con 0x55b4df617050 2024-04-25T01:30:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b02076640 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b4df617050 0x55b4df619470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b02076640 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b4df617050 0x55b4df619470 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba8002676a7306e server_cookie=2a2f9cb40ef4169a in_seq=0 out_seq=0 2024-04-25T01:30:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2ae1ffb640 10 client.16773 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2ae1ffb640 1 -- 172.21.15.27:0/3935182097 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b4df501f40 con 0x55b4df617050 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2ae1ffb640 10 client.16773 handle_command_reply: tid=0 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.245+0000 7f2b06fee740 1 client.16773 shutdown 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 -- 172.21.15.27:0/3935182097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 msgr2=0x55b4df5a8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df5a8960 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2af0008b00 tx=0x7f2af0008310 comp rx=0 tx=0).stop 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 -- 172.21.15.27:0/3935182097 shutdown_connections 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b4df617050 0x55b4df619470 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b4df5a0620 0x55b4df5a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b4df595f50 0x55b4df5a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2ae1ffb640 0 client.16773 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:24.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 --2- 172.21.15.27:0/3935182097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4df595630 0x55b4df5a8960 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 -- 172.21.15.27:0/3935182097 >> 172.21.15.27:0/3935182097 conn(0x55b4df591950 msgr2=0x55b4df590260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 -- 172.21.15.27:0/3935182097 shutdown_connections 2024-04-25T01:30:24.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 1 -- 172.21.15.27:0/3935182097 wait complete. 2024-04-25T01:30:24.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b06fee740 20 client.16773 trim_cache size 0 max 16384 2024-04-25T01:30:24.232 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout: "53e106bf-9132-45bc-b667-f3add96bd116": { 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "53e106bf-9132-45bc-b667-f3add96bd116", 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:30:24.233 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:30:24.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 -- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2ae40a2d00 msgr2=0x7f2ae40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2ae40a2d00 0x7f2ae40a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0bcf80 tx=0x7f2aec0095c0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 -- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc106630 msgr2=0x7f2afc19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:24.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc106630 0x7f2afc19c820 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2aec005cc0 tx=0x7f2aec003970 comp rx=0 tx=0).stop 2024-04-25T01:30:24.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 -- 172.21.15.27:0/1248858486 shutdown_connections 2024-04-25T01:30:24.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f2ae40a2d00 0x7f2ae40a51c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc106630 0x7f2afc19c820 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2afc100fe0 0x7f2afc1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 --2- 172.21.15.27:0/1248858486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2afc100620 0x7f2afc197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 -- 172.21.15.27:0/1248858486 >> 172.21.15.27:0/1248858486 conn(0x7f2afc0776a0 msgr2=0x7f2afc069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 -- 172.21.15.27:0/1248858486 shutdown_connections 2024-04-25T01:30:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:24.246+0000 7f2b042da640 1 -- 172.21.15.27:0/1248858486 wait complete. 2024-04-25T01:30:24.237 INFO:tasks.cephfs.filesystem:scrub status for tag:53e106bf-9132-45bc-b667-f3add96bd116 - {'path': '/', 'tag': '53e106bf-9132-45bc-b667-f3add96bd116', 'options': 'recursive,force'} 2024-04-25T01:30:24.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:24 smithi027 ceph-mon[29197]: pgmap v152: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:24.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:24 smithi153 ceph-mon[32621]: pgmap v152: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:24.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:24 smithi139 ceph-mon[32760]: pgmap v152: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:25.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:25 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:30:25.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:25 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:25.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:25 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:30:25.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:25 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:30:25.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:25 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:25.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:25 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:25.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:25 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:30:25.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:25 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:25.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:25 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:30:25.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:25 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:30:25.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:25 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:25.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:25 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:25.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:25 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:30:25.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:25 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:25.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:25 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:30:25.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:25 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:30:25.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:25 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:25.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:25 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:26.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:26 smithi027 ceph-mon[29197]: pgmap v153: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:26.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:26 smithi153 ceph-mon[32621]: pgmap v153: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:26.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:26 smithi139 ceph-mon[32760]: pgmap v153: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:28.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:28 smithi027 ceph-mon[29197]: pgmap v154: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:28.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:28 smithi153 ceph-mon[32621]: pgmap v154: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:28.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:28 smithi139 ceph-mon[32760]: pgmap v154: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:30.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:30 smithi027 ceph-mon[29197]: pgmap v155: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 9 op/s 2024-04-25T01:30:30.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:30 smithi153 ceph-mon[32621]: pgmap v155: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 9 op/s 2024-04-25T01:30:30.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:30 smithi139 ceph-mon[32760]: pgmap v155: 129 pgs: 129 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 9 op/s 2024-04-25T01:30:32.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:32 smithi027 ceph-mon[29197]: pgmap v156: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:32.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:32 smithi153 ceph-mon[32621]: pgmap v156: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:32.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:32 smithi139 ceph-mon[32760]: pgmap v156: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:33 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:30:33.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:33 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:30:33.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:33 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:30:33.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:33 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:30:33.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:33 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:30:33.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:33 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:30:34.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:34 smithi027 ceph-mon[29197]: pgmap v157: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:30:34.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:34 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:34.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:34 smithi153 ceph-mon[32621]: pgmap v157: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:30:34.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:34 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:34.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:34 smithi139 ceph-mon[32760]: pgmap v157: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:30:34.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:34 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:35.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:35 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:35.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:35 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:35.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:35 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:30:35.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:35 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:35.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:35 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:35.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:35 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:30:35.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:35 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:35.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:35 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:35.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:35 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:30:36.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:36 smithi153 ceph-mon[32621]: pgmap v158: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:30:36.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:36 smithi139 ceph-mon[32760]: pgmap v158: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:30:36.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:36 smithi027 ceph-mon[29197]: pgmap v158: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:30:38.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:38 smithi153 ceph-mon[32621]: pgmap v159: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:38.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:38 smithi139 ceph-mon[32760]: pgmap v159: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:38.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:38 smithi027 ceph-mon[29197]: pgmap v159: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:40.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:40 smithi027 ceph-mon[29197]: pgmap v160: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:40 smithi153 ceph-mon[32621]: pgmap v160: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:40 smithi139 ceph-mon[32760]: pgmap v160: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:42.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:42 smithi153 ceph-mon[32621]: pgmap v161: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:30:42.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:42 smithi139 ceph-mon[32760]: pgmap v161: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:30:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:42 smithi027 ceph-mon[29197]: pgmap v161: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:30:44.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:44 smithi153 ceph-mon[32621]: pgmap v162: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:44.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:44 smithi139 ceph-mon[32760]: pgmap v162: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:44.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:44 smithi027 ceph-mon[29197]: pgmap v162: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:46.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:46 smithi153 ceph-mon[32621]: pgmap v163: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:46.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:46 smithi139 ceph-mon[32760]: pgmap v163: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:46 smithi027 ceph-mon[29197]: pgmap v163: 129 pgs: 129 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:48.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:48 smithi153 ceph-mon[32621]: pgmap v164: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:48.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:48 smithi139 ceph-mon[32760]: pgmap v164: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:48.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:48 smithi027 ceph-mon[29197]: pgmap v164: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:50.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:50 smithi153 ceph-mon[32621]: pgmap v165: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:50.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:50 smithi139 ceph-mon[32760]: pgmap v165: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:50.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:50 smithi027 ceph-mon[29197]: pgmap v165: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.5 KiB/s wr, 9 op/s 2024-04-25T01:30:52.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:52 smithi153 ceph-mon[32621]: pgmap v166: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:30:52.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:52 smithi139 ceph-mon[32760]: pgmap v166: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:30:52.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:52 smithi027 ceph-mon[29197]: pgmap v166: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:30:54.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:30:54.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:30:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 -- 172.21.15.27:0/2037804765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f847406a700 msgr2=0x7f847410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 --2- 172.21.15.27:0/2037804765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f847406a700 0x7f847410e5e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f846c009cd0 tx=0x7f846c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 -- 172.21.15.27:0/2037804765 shutdown_connections 2024-04-25T01:30:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 --2- 172.21.15.27:0/2037804765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f847406a700 0x7f847410e5e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 --2- 172.21.15.27:0/2037804765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8474069d40 0x7f847406a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 --2- 172.21.15.27:0/2037804765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8474107720 0x7f8474107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 -- 172.21.15.27:0/2037804765 >> 172.21.15.27:0/2037804765 conn(0x7f8474076440 msgr2=0x7f8474076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 -- 172.21.15.27:0/2037804765 shutdown_connections 2024-04-25T01:30:54.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.378+0000 7f847a1dc640 1 -- 172.21.15.27:0/2037804765 wait complete. 2024-04-25T01:30:54.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f847a1dc640 1 Processor -- start 2024-04-25T01:30:54.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f847a1dc640 1 -- start start 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f847a1dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8474069d40 0x7f8474198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f847a1dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f847406a700 0x7f8474198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f847a1dc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8474107720 0x7f847419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f847a1dc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8474111b30 con 0x7f847406a700 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f847a1dc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f84741119b0 con 0x7f8474107720 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f847a1dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8474111cb0 con 0x7f8474069d40 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f8472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f847406a700 0x7f8474198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f8472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f847406a700 0x7f8474198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51116/0 (socket says 172.21.15.27:51116) 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f8472ffd640 1 -- 172.21.15.27:0/373587204 learned_addr learned my addr 172.21.15.27:0/373587204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f84737fe640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8474069d40 0x7f8474198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.379+0000 7f8473fff640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8474107720 0x7f847419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8472ffd640 1 -- 172.21.15.27:0/373587204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8474069d40 msgr2=0x7f8474198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8472ffd640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8474069d40 0x7f8474198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8472ffd640 1 -- 172.21.15.27:0/373587204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8474107720 msgr2=0x7f847419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8472ffd640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8474107720 0x7f847419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8472ffd640 1 -- 172.21.15.27:0/373587204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847419cfd0 con 0x7f847406a700 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f84737fe640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8474069d40 0x7f8474198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8472ffd640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f847406a700 0x7f8474198570 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f846800ea60 tx=0x7f846800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8470ff9640 1 -- 172.21.15.27:0/373587204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846800cde0 con 0x7f847406a700 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f847a1dc640 1 -- 172.21.15.27:0/373587204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f847419d2c0 con 0x7f847406a700 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8470ff9640 1 -- 172.21.15.27:0/373587204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84680035f0 con 0x7f847406a700 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f8470ff9640 1 -- 172.21.15.27:0/373587204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8468010670 con 0x7f847406a700 2024-04-25T01:30:54.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.380+0000 7f847a1dc640 1 -- 172.21.15.27:0/373587204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84741a4b50 con 0x7f847406a700 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.382+0000 7f8470ff9640 1 -- 172.21.15.27:0/373587204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f84680037a0 con 0x7f847406a700 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f847cfc6740 1 Processor -- start 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f847cfc6740 1 -- start start 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e400ff910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e40110e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f847cfc6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560e400646b0 con 0x560e400ffe50 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f847cfc6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560e4000c880 con 0x560e400ff530 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f847cfc6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560e40060670 con 0x560e4010ea30 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f8470ff9640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f845c0a2d00 0x7f845c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f84737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f84737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4010e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51124/0 (socket says 172.21.15.27:51124) 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f8473fff640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f845c0a2d00 0x7f845c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f84737fe640 1 -- 172.21.15.27:0/2607135728 learned_addr learned my addr 172.21.15.27:0/2607135728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f8472ffd640 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e40110e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f8473fff640 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e400ff910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f8470ff9640 1 -- 172.21.15.27:0/373587204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f84680cfc20 con 0x7f847406a700 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f84737fe640 1 -- 172.21.15.27:0/2607135728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 msgr2=0x560e40110e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f84737fe640 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e40110e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f84737fe640 1 -- 172.21.15.27:0/2607135728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 msgr2=0x560e400ff910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f84737fe640 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e400ff910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.383+0000 7f84737fe640 1 -- 172.21.15.27:0/2607135728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e3fe427d0 con 0x560e400ffe50 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.384+0000 7f8472ffd640 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e40110e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.384+0000 7f8473fff640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f845c0a2d00 0x7f845c0a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f846c009a60 tx=0x7f846c033000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.384+0000 7f84737fe640 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4010e4f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8464009cb0 tx=0x7f84640117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17885b93947b9d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.384+0000 7f8459ffb640 1 -- 172.21.15.27:0/2607135728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464012450 con 0x560e400ffe50 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.384+0000 7f8459ffb640 1 -- 172.21.15.27:0/2607135728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8464012ad0 con 0x560e400ffe50 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.384+0000 7f8459ffb640 1 -- 172.21.15.27:0/2607135728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846400fb60 con 0x560e400ffe50 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 -- 172.21.15.27:0/2607135728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 msgr2=0x560e4010e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4010e4f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8464009cb0 tx=0x7f84640117b0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 -- 172.21.15.27:0/2607135728 shutdown_connections 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e40110e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4010e4f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 --2- 172.21.15.27:0/2607135728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e400ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 -- 172.21.15.27:0/2607135728 >> 172.21.15.27:0/2607135728 conn(0x560e400fb950 msgr2=0x560e400fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 -- 172.21.15.27:0/2607135728 shutdown_connections 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 -- 172.21.15.27:0/2607135728 wait complete. 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 -- start start 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.385+0000 7f847cfc6740 1 Processor -- start 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ff530 0x560e4010b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e400ffe50 0x560e4010be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4010ea30 0x560e4010c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f847cfc6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560e401117f0 con 0x560e400ff530 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f847cfc6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560e4001ff50 con 0x560e4010ea30 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f847cfc6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560e40111970 con 0x560e400ffe50 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f847cfc6740 10 client.0 _ll_register_callbacks cb 0x560e4005b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ff530 0x560e4010b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ff530 0x560e4010b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51140/0 (socket says 172.21.15.27:51140) 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 -- 172.21.15.27:0/1539273194 learned_addr learned my addr 172.21.15.27:0/1539273194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f8472ffd640 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e400ffe50 0x560e4010be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f8473fff640 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4010ea30 0x560e4010c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 -- 172.21.15.27:0/1539273194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e400ffe50 msgr2=0x560e4010be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e400ffe50 0x560e4010be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 -- 172.21.15.27:0/1539273194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4010ea30 msgr2=0x560e4010c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4010ea30 0x560e4010c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 -- 172.21.15.27:0/1539273194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e3fd783f0 con 0x560e400ff530 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f8472ffd640 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e400ffe50 0x560e4010be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f84737fe640 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ff530 0x560e4010b8e0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f846400f7c0 tx=0x7f846400f4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205ddd5fe36a809a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f8452ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f8452ffd640 1 -- 172.21.15.27:0/1539273194 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464028020 con 0x560e400ff530 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f8452ffd640 1 -- 172.21.15.27:0/1539273194 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f846400fcb0 con 0x560e400ff530 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.386+0000 7f847cfc6740 1 -- 172.21.15.27:0/1539273194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8464012ad0 con 0x560e400ff530 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f8452ffd640 1 -- 172.21.15.27:0/1539273194 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8464033050 con 0x560e400ff530 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f847cfc6740 10 client.16785 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f847cfc6740 1 -- 172.21.15.27:0/1539273194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560e3fd380f0 con 0x560e400ff530 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f8452ffd640 1 -- 172.21.15.27:0/1539273194 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8464012ad0 con 0x560e400ff530 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f8452ffd640 1 -- 172.21.15.27:0/1539273194 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8464004850 con 0x560e400ff530 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f847cfc6740 10 client.16785 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f847cfc6740 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f847cfc6740 20 client.16785 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560e4011ec50 0x560e40121040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f847cfc6740 4 client.16785 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f847cfc6740 1 -- 172.21.15.27:0/1539273194 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e40071bf0 con 0x560e4011ec50 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.387+0000 7f8472ffd640 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560e4011ec50 0x560e40121040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.388+0000 7f8472ffd640 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560e4011ec50 0x560e40121040 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83f508c48853694 server_cookie=44e8affbfd522020 in_seq=0 out_seq=0 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.388+0000 7f8452ffd640 10 client.16785 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.388+0000 7f8452ffd640 1 -- 172.21.15.27:0/1539273194 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e40071bf0 con 0x560e4011ec50 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.388+0000 7f8452ffd640 10 client.16785 handle_command_reply: tid=0 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.388+0000 7f847cfc6740 1 client.16785 shutdown 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 -- 172.21.15.27:0/1539273194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ff530 msgr2=0x560e4010b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ff530 0x560e4010b8e0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f846400f7c0 tx=0x7f846400f4e0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 -- 172.21.15.27:0/1539273194 shutdown_connections 2024-04-25T01:30:54.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560e4011ec50 0x560e40121040 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e4010ea30 0x560e4010c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f8452ffd640 0 client.16785 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e400ffe50 0x560e4010be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1539273194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ff530 0x560e4010b8e0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 -- 172.21.15.27:0/1539273194 >> 172.21.15.27:0/1539273194 conn(0x560e400fb950 msgr2=0x560e401016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 -- 172.21.15.27:0/1539273194 shutdown_connections 2024-04-25T01:30:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 1 -- 172.21.15.27:0/1539273194 wait complete. 2024-04-25T01:30:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.389+0000 7f847cfc6740 20 client.16785 trim_cache size 0 max 16384 2024-04-25T01:30:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.406+0000 7f847cfc6740 1 Processor -- start 2024-04-25T01:30:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.406+0000 7f847cfc6740 1 -- start start 2024-04-25T01:30:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e4010e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4011a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e4010a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f847cfc6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84741af3c0 con 0x560e400ffe50 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f847cfc6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f84741acc90 con 0x560e400ff530 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f847cfc6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84741aaae0 con 0x560e4010ea30 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4011a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4011a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51152/0 (socket says 172.21.15.27:51152) 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 -- 172.21.15.27:0/2995913373 learned_addr learned my addr 172.21.15.27:0/2995913373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 -- 172.21.15.27:0/2995913373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 msgr2=0x560e4010a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 --2- 172.21.15.27:0/2995913373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e4010a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 -- 172.21.15.27:0/2995913373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 msgr2=0x560e4010e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 --2- 172.21.15.27:0/2995913373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e4010e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 -- 172.21.15.27:0/2995913373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e400fb780 con 0x560e400ffe50 2024-04-25T01:30:54.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f8472ffd640 1 --2- 172.21.15.27:0/2995913373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4011a6f0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8468098210 tx=0x7f846809b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeeea7e930ec8b92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f84597fa640 1 -- 172.21.15.27:0/2995913373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846809a370 con 0x560e400ffe50 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f84597fa640 1 -- 172.21.15.27:0/2995913373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f846809c4b0 con 0x560e400ffe50 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.407+0000 7f84597fa640 1 -- 172.21.15.27:0/2995913373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8468003b30 con 0x560e400ffe50 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 -- 172.21.15.27:0/2995913373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 msgr2=0x560e4011a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 --2- 172.21.15.27:0/2995913373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4011a6f0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8468098210 tx=0x7f846809b910 comp rx=0 tx=0).stop 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 -- 172.21.15.27:0/2995913373 shutdown_connections 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 --2- 172.21.15.27:0/2995913373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e4010a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 --2- 172.21.15.27:0/2995913373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e4011a6f0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 --2- 172.21.15.27:0/2995913373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e4010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 -- 172.21.15.27:0/2995913373 >> 172.21.15.27:0/2995913373 conn(0x560e400fb950 msgr2=0x560e4010a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 -- 172.21.15.27:0/2995913373 shutdown_connections 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 -- 172.21.15.27:0/2995913373 wait complete. 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.408+0000 7f847cfc6740 1 -- start start 2024-04-25T01:30:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f847cfc6740 1 Processor -- start 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e40116fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e40112330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f847cfc6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e40112870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f847cfc6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f84741ab150 con 0x560e400ffe50 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f847cfc6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f84741ac5c0 con 0x560e400ff530 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f847cfc6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84741ab820 con 0x560e4010ea30 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f847cfc6740 10 client.0 _ll_register_callbacks cb 0x7f8468005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e40112330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e40112330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51156/0 (socket says 172.21.15.27:51156) 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 -- 172.21.15.27:0/1971434107 learned_addr learned my addr 172.21.15.27:0/1971434107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f84737fe640 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e40116fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 -- 172.21.15.27:0/1971434107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 msgr2=0x560e40112870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e40112870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 -- 172.21.15.27:0/1971434107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 msgr2=0x560e40116fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e40116fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 -- 172.21.15.27:0/1971434107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e401066c0 con 0x560e400ffe50 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8472ffd640 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e40112330 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f846800ea30 tx=0x7f84680095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5c4b9d4b62e9a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8452ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.409+0000 7f8452ffd640 1 -- 172.21.15.27:0/1971434107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846809bdb0 con 0x560e400ffe50 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f8452ffd640 1 -- 172.21.15.27:0/1971434107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8468003b30 con 0x560e400ffe50 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 1 -- 172.21.15.27:0/1971434107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f846809c4b0 con 0x560e400ffe50 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f8452ffd640 1 -- 172.21.15.27:0/1971434107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f84680cd260 con 0x560e400ffe50 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 10 client.16791 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 1 -- 172.21.15.27:0/1971434107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560e40184e00 con 0x560e400ffe50 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f8452ffd640 1 -- 172.21.15.27:0/1971434107 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f846809c4b0 con 0x560e400ffe50 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f8452ffd640 1 -- 172.21.15.27:0/1971434107 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8468086040 con 0x560e400ffe50 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 10 client.16791 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 20 client.16791 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560e40185420 0x560e40187810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 4 client.16791 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f847cfc6740 1 -- 172.21.15.27:0/1971434107 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560e4006bf40 con 0x560e40185420 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.410+0000 7f84737fe640 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560e40185420 0x560e40187810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f84737fe640 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560e40185420 0x560e40187810 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3264847a03ab0f server_cookie=821f3c0198be6df3 in_seq=0 out_seq=0 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f8452ffd640 10 client.16791 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f8452ffd640 1 -- 172.21.15.27:0/1971434107 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560e4006bf40 con 0x560e40185420 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f8452ffd640 10 client.16791 handle_command_reply: tid=0 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 client.16791 shutdown 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 -- 172.21.15.27:0/1971434107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 msgr2=0x560e40112330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e40112330 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f846800ea30 tx=0x7f84680095b0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 -- 172.21.15.27:0/1971434107 shutdown_connections 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560e40185420 0x560e40187810 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e4010ea30 0x560e40112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f8452ffd640 0 client.16791 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560e400ffe50 0x560e40112330 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 --2- 172.21.15.27:0/1971434107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560e400ff530 0x560e40116fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 -- 172.21.15.27:0/1971434107 >> 172.21.15.27:0/1971434107 conn(0x560e400fb950 msgr2=0x560e4010a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 -- 172.21.15.27:0/1971434107 shutdown_connections 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 1 -- 172.21.15.27:0/1971434107 wait complete. 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.411+0000 7f847cfc6740 20 client.16791 trim_cache size 0 max 16384 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:30:54.395 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:30:54.396 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:30:54.396 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:30:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 -- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f845c0a2d00 msgr2=0x7f845c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f845c0a2d00 0x7f845c0a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f846c009a60 tx=0x7f846c033000 comp rx=0 tx=0).stop 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 -- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f847406a700 msgr2=0x7f8474198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f847406a700 0x7f8474198570 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f846800ea60 tx=0x7f846800ef20 comp rx=0 tx=0).stop 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 -- 172.21.15.27:0/373587204 shutdown_connections 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8474107720 0x7f847419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f845c0a2d00 0x7f845c0a51c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f847406a700 0x7f8474198570 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 --2- 172.21.15.27:0/373587204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8474069d40 0x7f8474198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 -- 172.21.15.27:0/373587204 >> 172.21.15.27:0/373587204 conn(0x7f8474076440 msgr2=0x7f84740fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 -- 172.21.15.27:0/373587204 shutdown_connections 2024-04-25T01:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.412+0000 7f847a1dc640 1 -- 172.21.15.27:0/373587204 wait complete. 2024-04-25T01:30:54.400 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:30:54.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 -- 172.21.15.27:0/1271568781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c10bd60 msgr2=0x7f104c10e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1271568781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c10bd60 0x7f104c10e1d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1044009ca0 tx=0x7f104402a950 comp rx=0 tx=0).stop 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 -- 172.21.15.27:0/1271568781 shutdown_connections 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1271568781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c10bd60 0x7f104c10e1d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1271568781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f104c103340 0x7f104c10b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1271568781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f104c102a20 0x7f104c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 -- 172.21.15.27:0/1271568781 >> 172.21.15.27:0/1271568781 conn(0x7f104c0777a0 msgr2=0x7f104c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 -- 172.21.15.27:0/1271568781 shutdown_connections 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 -- 172.21.15.27:0/1271568781 wait complete. 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 Processor -- start 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.526+0000 7f1053e8c640 1 -- start start 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1053e8c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f104c102a20 0x7f104c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1053e8c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c103340 0x7f104c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1053e8c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f104c10bd60 0x7f104c19c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1053e8c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f104c111bf0 con 0x7f104c103340 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1053e8c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f104c111a70 con 0x7f104c10bd60 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1053e8c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f104c111d70 con 0x7f104c102a20 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c103340 0x7f104c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c103340 0x7f104c198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51176/0 (socket says 172.21.15.27:51176) 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 -- 172.21.15.27:0/1417774605 learned_addr learned my addr 172.21.15.27:0/1417774605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051c28640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f104c102a20 0x7f104c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 -- 172.21.15.27:0/1417774605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f104c102a20 msgr2=0x7f104c198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f104c102a20 0x7f104c198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1052429640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f104c10bd60 0x7f104c19c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 -- 172.21.15.27:0/1417774605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f104c10bd60 msgr2=0x7f104c19c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f104c10bd60 0x7f104c19c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 -- 172.21.15.27:0/1417774605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104c19d100 con 0x7f104c103340 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051c28640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f104c102a20 0x7f104c198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1052429640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f104c10bd60 0x7f104c19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1051427640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c103340 0x7f104c198650 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f103c00b4a0 tx=0x7f103c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1042ffd640 1 -- 172.21.15.27:0/1417774605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c014070 con 0x7f104c103340 2024-04-25T01:30:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1042ffd640 1 -- 172.21.15.27:0/1417774605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f103c010af0 con 0x7f104c103340 2024-04-25T01:30:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1042ffd640 1 -- 172.21.15.27:0/1417774605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c017d10 con 0x7f104c103340 2024-04-25T01:30:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f104c19d3f0 con 0x7f104c103340 2024-04-25T01:30:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.527+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f104c1a4c80 con 0x7f104c103340 2024-04-25T01:30:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.528+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104c0689b0 con 0x7f104c103340 2024-04-25T01:30:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.530+0000 7f1042ffd640 1 -- 172.21.15.27:0/1417774605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f103c017070 con 0x7f104c103340 2024-04-25T01:30:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.531+0000 7f1042ffd640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10380a2d00 0x7f10380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.531+0000 7f1051c28640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10380a2d00 0x7f10380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.531+0000 7f1042ffd640 1 -- 172.21.15.27:0/1417774605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f103c0ceee0 con 0x7f104c103340 2024-04-25T01:30:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.531+0000 7f1051c28640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10380a2d00 0x7f10380a51c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1048006fd0 tx=0x7f1048008040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.535+0000 7f1042ffd640 1 -- 172.21.15.27:0/1417774605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f103c09b880 con 0x7f104c103340 2024-04-25T01:30:54.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:54 smithi027 ceph-mon[29197]: pgmap v167: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.761+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f104c199460 con 0x7f104c103340 2024-04-25T01:30:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.762+0000 7f1042ffd640 1 -- 172.21.15.27:0/1417774605 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7f104c199460 con 0x7f104c103340 2024-04-25T01:30:54.740 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:30:54.740 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:30:54.740 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10380a2d00 msgr2=0x7f10380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10380a2d00 0x7f10380a51c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1048006fd0 tx=0x7f1048008040 comp rx=0 tx=0).stop 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c103340 msgr2=0x7f104c198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c103340 0x7f104c198650 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f103c00b4a0 tx=0x7f103c00b960 comp rx=0 tx=0).stop 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 shutdown_connections 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f104c10bd60 0x7f104c19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f10380a2d00 0x7f10380a51c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104c103340 0x7f104c198650 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 --2- 172.21.15.27:0/1417774605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f104c102a20 0x7f104c198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 >> 172.21.15.27:0/1417774605 conn(0x7f104c0777a0 msgr2=0x7f104c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 shutdown_connections 2024-04-25T01:30:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.764+0000 7f1053e8c640 1 -- 172.21.15.27:0/1417774605 wait complete. 2024-04-25T01:30:54.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:30:54.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:54 smithi153 ceph-mon[32621]: pgmap v167: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:54.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:54 smithi139 ceph-mon[32760]: pgmap v167: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 -- 172.21.15.27:0/1271925977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198100fe0 msgr2=0x7f319810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 --2- 172.21.15.27:0/1271925977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198100fe0 0x7f319810d4f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3190009930 tx=0x7f319002a420 comp rx=0 tx=0).stop 2024-04-25T01:30:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 -- 172.21.15.27:0/1271925977 shutdown_connections 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 --2- 172.21.15.27:0/1271925977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198100fe0 0x7f319810d4f0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 --2- 172.21.15.27:0/1271925977 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3198100620 0x7f3198100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 --2- 172.21.15.27:0/1271925977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3198106630 0x7f3198106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 -- 172.21.15.27:0/1271925977 >> 172.21.15.27:0/1271925977 conn(0x7f31980776a0 msgr2=0x7f3198077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 -- 172.21.15.27:0/1271925977 shutdown_connections 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.879+0000 7f319fa76640 1 -- 172.21.15.27:0/1271925977 wait complete. 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319fa76640 1 Processor -- start 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319fa76640 1 -- start start 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319fa76640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3198100620 0x7f3198196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319fa76640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3198100fe0 0x7f3198197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319fa76640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198106630 0x7f319819b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319fa76640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3198110a90 con 0x7f3198106630 2024-04-25T01:30:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319fa76640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3198110910 con 0x7f3198100fe0 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319fa76640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3198110c10 con 0x7f3198100620 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319e013640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198106630 0x7f319819b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319e013640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198106630 0x7f319819b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51184/0 (socket says 172.21.15.27:51184) 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319e013640 1 -- 172.21.15.27:0/221758188 learned_addr learned my addr 172.21.15.27:0/221758188 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319d011640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3198100fe0 0x7f3198197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319d812640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3198100620 0x7f3198196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319e013640 1 -- 172.21.15.27:0/221758188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3198100620 msgr2=0x7f3198196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319e013640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3198100620 0x7f3198196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319e013640 1 -- 172.21.15.27:0/221758188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3198100fe0 msgr2=0x7f3198197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319e013640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3198100fe0 0x7f3198197450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.880+0000 7f319e013640 1 -- 172.21.15.27:0/221758188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319819bf00 con 0x7f3198106630 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.881+0000 7f319e013640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198106630 0x7f319819b7d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3190004770 tx=0x7f3190003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.881+0000 7f318effd640 1 -- 172.21.15.27:0/221758188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190035070 con 0x7f3198106630 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.881+0000 7f319fa76640 1 -- 172.21.15.27:0/221758188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319819c190 con 0x7f3198106630 2024-04-25T01:30:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.881+0000 7f318effd640 1 -- 172.21.15.27:0/221758188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3190031a90 con 0x7f3198106630 2024-04-25T01:30:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.881+0000 7f318effd640 1 -- 172.21.15.27:0/221758188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190038b60 con 0x7f3198106630 2024-04-25T01:30:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.881+0000 7f319fa76640 1 -- 172.21.15.27:0/221758188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3198069d40 con 0x7f3198106630 2024-04-25T01:30:54.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.883+0000 7f318effd640 1 -- 172.21.15.27:0/221758188 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3190031c40 con 0x7f3198106630 2024-04-25T01:30:54.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.883+0000 7f31a2785740 1 Processor -- start 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f31a2785740 1 -- start start 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f31a2785740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1c580 0x56027ef1c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f31a2785740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1cea0 0x56027ef27030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f31a2785740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef29960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f31a2785740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56027ee81670 con 0x56027ef1c580 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f31a2785740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56027ee29970 con 0x56027ef1cea0 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f31a2785740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56027ee7d630 con 0x56027ef27570 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f318effd640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f31840a2d00 0x7f31840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1c580 0x56027ef1c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1c580 0x56027ef1c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51188/0 (socket says 172.21.15.27:51188) 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 -- 172.21.15.27:0/518794686 learned_addr learned my addr 172.21.15.27:0/518794686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319e013640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f31840a2d00 0x7f31840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d812640 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1cea0 0x56027ef27030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef29960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f318effd640 1 -- 172.21.15.27:0/221758188 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f31900f1820 con 0x7f3198106630 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 -- 172.21.15.27:0/518794686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 msgr2=0x56027ef29960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef29960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 -- 172.21.15.27:0/518794686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1cea0 msgr2=0x56027ef27030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1cea0 0x56027ef27030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.884+0000 7f319d011640 1 -- 172.21.15.27:0/518794686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56027ee4d400 con 0x56027ef1c580 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.885+0000 7f319e013640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f31840a2d00 0x7f31840a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f31900eec90 tx=0x7f31900a8040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.885+0000 7f319d011640 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1c580 0x56027ef1c960 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3188005cf0 tx=0x7f318800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7468efb659016de8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.885+0000 7f3183fff640 1 -- 172.21.15.27:0/518794686 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188004030 con 0x56027ef1c580 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.885+0000 7f3183fff640 1 -- 172.21.15.27:0/518794686 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3188005500 con 0x56027ef1c580 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.885+0000 7f3183fff640 1 -- 172.21.15.27:0/518794686 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188004b60 con 0x56027ef1c580 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- 172.21.15.27:0/518794686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1c580 msgr2=0x56027ef1c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1c580 0x56027ef1c960 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3188005cf0 tx=0x7f318800b840 comp rx=0 tx=0).stop 2024-04-25T01:30:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- 172.21.15.27:0/518794686 shutdown_connections 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef29960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1cea0 0x56027ef27030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 --2- 172.21.15.27:0/518794686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1c580 0x56027ef1c960 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- 172.21.15.27:0/518794686 >> 172.21.15.27:0/518794686 conn(0x56027ef188a0 msgr2=0x56027ef18c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- 172.21.15.27:0/518794686 shutdown_connections 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- 172.21.15.27:0/518794686 wait complete. 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- start start 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 Processor -- start 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef26f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef24200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef247a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56027ef2a360 con 0x56027ef1cea0 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56027ee3d050 con 0x56027ef1c580 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.886+0000 7f31a2785740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56027ef2a4e0 con 0x56027ef27570 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f31a2785740 10 client.0 _ll_register_callbacks cb 0x56027eb94be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef24200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef24200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51190/0 (socket says 172.21.15.27:51190) 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 -- 172.21.15.27:0/1291249987 learned_addr learned my addr 172.21.15.27:0/1291249987 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d011640 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef247a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d011640 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef26f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 -- 172.21.15.27:0/1291249987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 msgr2=0x56027ef247a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef247a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 -- 172.21.15.27:0/1291249987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 msgr2=0x56027ef26f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef26f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 -- 172.21.15.27:0/1291249987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56027ee0edc0 con 0x56027ef1cea0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d011640 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef26f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f319d812640 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef24200 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f319400be50 tx=0x7f3194007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2347f23883bf9552 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f31817fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f31817fa640 1 -- 172.21.15.27:0/1291249987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3194007bd0 con 0x56027ef1cea0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f31817fa640 1 -- 172.21.15.27:0/1291249987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3194007d70 con 0x56027ef1cea0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f31a2785740 1 -- 172.21.15.27:0/1291249987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3188005500 con 0x56027ef1cea0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f31817fa640 1 -- 172.21.15.27:0/1291249987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3194019410 con 0x56027ef1cea0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f31a2785740 10 client.16809 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.887+0000 7f31a2785740 1 -- 172.21.15.27:0/1291249987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56027eb953f0 con 0x56027ef1cea0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f31817fa640 1 -- 172.21.15.27:0/1291249987 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3188005500 con 0x56027ef1cea0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f31817fa640 1 -- 172.21.15.27:0/1291249987 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3194018690 con 0x56027ef1cea0 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f31a2785740 10 client.16809 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f31a2785740 10 client.16809 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f31a2785740 20 client.16809 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f31a2785740 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027ef37700 0x56027ef39b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f31a2785740 4 client.16809 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f31a2785740 1 -- 172.21.15.27:0/1291249987 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56027ee8ebb0 con 0x56027ef37700 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.888+0000 7f319d011640 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027ef37700 0x56027ef39b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.889+0000 7f319d011640 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027ef37700 0x56027ef39b60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83383ed136696d8e server_cookie=898cf852e5d55ea in_seq=0 out_seq=0 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.889+0000 7f31817fa640 10 client.16809 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.889+0000 7f31817fa640 1 -- 172.21.15.27:0/1291249987 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56027ee8ebb0 con 0x56027ef37700 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.889+0000 7f31817fa640 10 client.16809 handle_command_reply: tid=0 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.889+0000 7f31a2785740 1 client.16809 shutdown 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.889+0000 7f31a2785740 1 -- 172.21.15.27:0/1291249987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 msgr2=0x56027ef24200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.889+0000 7f31a2785740 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef24200 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f319400be50 tx=0x7f3194007680 comp rx=0 tx=0).stop 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 1 -- 172.21.15.27:0/1291249987 shutdown_connections 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027ef37700 0x56027ef39b60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31817fa640 0 client.16809 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:30:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef24200 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 1 --2- 172.21.15.27:0/1291249987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef26f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 1 -- 172.21.15.27:0/1291249987 >> 172.21.15.27:0/1291249987 conn(0x56027ef188a0 msgr2=0x56027ef17b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 1 -- 172.21.15.27:0/1291249987 shutdown_connections 2024-04-25T01:30:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 1 -- 172.21.15.27:0/1291249987 wait complete. 2024-04-25T01:30:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.890+0000 7f31a2785740 20 client.16809 trim_cache size 0 max 16384 2024-04-25T01:30:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f31a2785740 1 Processor -- start 2024-04-25T01:30:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f31a2785740 1 -- start start 2024-04-25T01:30:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f31a2785740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef33220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f31a2785740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef1cea0 0x56027ef26a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f31a2785740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef27570 0x56027ef3f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f31a2785740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31981af270 con 0x56027ef27570 2024-04-25T01:30:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f31a2785740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f31981acb40 con 0x56027ef1c580 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f31a2785740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31981aa990 con 0x56027ef1cea0 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f319d011640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef27570 0x56027ef3f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f319d011640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef27570 0x56027ef3f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51202/0 (socket says 172.21.15.27:51202) 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.903+0000 7f319d011640 1 -- 172.21.15.27:0/1419528718 learned_addr learned my addr 172.21.15.27:0/1419528718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d011640 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef33220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d812640 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef1cea0 0x56027ef26a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d011640 1 -- 172.21.15.27:0/1419528718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef1cea0 msgr2=0x56027ef26a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d011640 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef1cea0 0x56027ef26a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d011640 1 -- 172.21.15.27:0/1419528718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 msgr2=0x56027ef33220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d011640 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef33220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d011640 1 -- 172.21.15.27:0/1419528718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56027ec5f7d0 con 0x56027ef27570 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d812640 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef1cea0 0x56027ef26a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f319d011640 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef27570 0x56027ef3f430 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x56027ef1ed40 tx=0x7f3188004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d285faad6a1f10e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f31837fe640 1 -- 172.21.15.27:0/1419528718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188029340 con 0x56027ef27570 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f31837fe640 1 -- 172.21.15.27:0/1419528718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3188010460 con 0x56027ef27570 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f31837fe640 1 -- 172.21.15.27:0/1419528718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318801b650 con 0x56027ef27570 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f31a2785740 1 -- 172.21.15.27:0/1419528718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef27570 msgr2=0x56027ef3f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.904+0000 7f31a2785740 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef27570 0x56027ef3f430 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x56027ef1ed40 tx=0x7f3188004560 comp rx=0 tx=0).stop 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 -- 172.21.15.27:0/1419528718 shutdown_connections 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef27570 0x56027ef3f430 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef1cea0 0x56027ef26a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 --2- 172.21.15.27:0/1419528718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef33220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 -- 172.21.15.27:0/1419528718 >> 172.21.15.27:0/1419528718 conn(0x56027ef188a0 msgr2=0x56027ef1e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 -- 172.21.15.27:0/1419528718 shutdown_connections 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 -- 172.21.15.27:0/1419528718 wait complete. 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 -- start start 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 Processor -- start 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef2b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef2b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef2bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31981ab000 con 0x56027ef1cea0 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f31981ac470 con 0x56027ef1c580 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31981ab6d0 con 0x56027ef27570 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f31a2785740 10 client.0 _ll_register_callbacks cb 0x7f3194009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f319d812640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef2b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f319d812640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef2b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51216/0 (socket says 172.21.15.27:51216) 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.905+0000 7f319d812640 1 -- 172.21.15.27:0/1136114551 learned_addr learned my addr 172.21.15.27:0/1136114551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d011640 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef2bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d011640 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef2b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d812640 1 -- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 msgr2=0x56027ef2bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d812640 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef2bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d812640 1 -- 172.21.15.27:0/1136114551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 msgr2=0x56027ef2b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d812640 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef2b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d812640 1 -- 172.21.15.27:0/1136114551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56027efa5560 con 0x56027ef1cea0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d011640 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef2bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f319d812640 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef2b740 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f3194000f80 tx=0x7f319400ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538cd492b77e07cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f31817fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f31817fa640 1 -- 172.21.15.27:0/1136114551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319402c020 con 0x56027ef1cea0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f31817fa640 1 -- 172.21.15.27:0/1136114551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f31940185d0 con 0x56027ef1cea0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f31a2785740 1 -- 172.21.15.27:0/1136114551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3188010460 con 0x56027ef1cea0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f31817fa640 1 -- 172.21.15.27:0/1136114551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3194009c60 con 0x56027ef1cea0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f31a2785740 10 client.16815 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.906+0000 7f31a2785740 1 -- 172.21.15.27:0/1136114551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56027efa67e0 con 0x56027ef1cea0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31817fa640 1 -- 172.21.15.27:0/1136114551 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3188010460 con 0x56027ef1cea0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31817fa640 1 -- 172.21.15.27:0/1136114551 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3194028480 con 0x56027ef1cea0 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31a2785740 10 client.16815 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31a2785740 10 client.16815 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31a2785740 20 client.16815 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31a2785740 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027efa6fe0 0x56027efa9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31a2785740 4 client.16815 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31a2785740 1 -- 172.21.15.27:0/1136114551 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56027ee88f00 con 0x56027efa6fe0 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f319d011640 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027efa6fe0 0x56027efa9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f31817fa640 10 client.16815 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.907+0000 7f319d011640 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027efa6fe0 0x56027efa9400 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b1a0c8b09f51b68 server_cookie=23e30be10eb63a26 in_seq=0 out_seq=0 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31817fa640 1 -- 172.21.15.27:0/1136114551 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56027ee88f00 con 0x56027efa6fe0 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31817fa640 10 client.16815 handle_command_reply: tid=0 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 client.16815 shutdown 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 -- 172.21.15.27:0/1136114551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 msgr2=0x56027ef2b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef2b740 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f3194000f80 tx=0x7f319400ba70 comp rx=0 tx=0).stop 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 -- 172.21.15.27:0/1136114551 shutdown_connections 2024-04-25T01:30:54.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027efa6fe0 0x56027efa9400 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027ef27570 0x56027ef2bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027ef1cea0 0x56027ef2b740 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 --2- 172.21.15.27:0/1136114551 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027ef1c580 0x56027ef2b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31817fa640 0 client.16815 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 -- 172.21.15.27:0/1136114551 >> 172.21.15.27:0/1136114551 conn(0x56027ef188a0 msgr2=0x56027ef1ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 -- 172.21.15.27:0/1136114551 shutdown_connections 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 1 -- 172.21.15.27:0/1136114551 wait complete. 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f31a2785740 20 client.16815 trim_cache size 0 max 16384 2024-04-25T01:30:54.892 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:30:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f319fa76640 1 -- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f31840a2d00 msgr2=0x7f31840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f319fa76640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f31840a2d00 0x7f31840a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f31900eec90 tx=0x7f31900a8040 comp rx=0 tx=0).stop 2024-04-25T01:30:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f319fa76640 1 -- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198106630 msgr2=0x7f319819b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f319fa76640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198106630 0x7f319819b7d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3190004770 tx=0x7f3190003b90 comp rx=0 tx=0).stop 2024-04-25T01:30:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f319fa76640 1 -- 172.21.15.27:0/221758188 shutdown_connections 2024-04-25T01:30:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.908+0000 7f319fa76640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f31840a2d00 0x7f31840a51c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.909+0000 7f319fa76640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3198106630 0x7f319819b7d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.909+0000 7f319fa76640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3198100fe0 0x7f3198197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.909+0000 7f319fa76640 1 --2- 172.21.15.27:0/221758188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3198100620 0x7f3198196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.909+0000 7f319fa76640 1 -- 172.21.15.27:0/221758188 >> 172.21.15.27:0/221758188 conn(0x7f31980776a0 msgr2=0x7f31981030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.909+0000 7f319fa76640 1 -- 172.21.15.27:0/221758188 shutdown_connections 2024-04-25T01:30:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:54.909+0000 7f319fa76640 1 -- 172.21.15.27:0/221758188 wait complete. 2024-04-25T01:30:54.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 -- 172.21.15.27:0/74697057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 msgr2=0x7f43d810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 --2- 172.21.15.27:0/74697057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 0x7f43d810d4f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f43d0009cd0 tx=0x7f43d002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 -- 172.21.15.27:0/74697057 shutdown_connections 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 --2- 172.21.15.27:0/74697057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 0x7f43d810d4f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 --2- 172.21.15.27:0/74697057 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43d8100620 0x7f43d8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 --2- 172.21.15.27:0/74697057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43d8106630 0x7f43d8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 -- 172.21.15.27:0/74697057 >> 172.21.15.27:0/74697057 conn(0x7f43d80776a0 msgr2=0x7f43d8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 -- 172.21.15.27:0/74697057 shutdown_connections 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 -- 172.21.15.27:0/74697057 wait complete. 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 Processor -- start 2024-04-25T01:30:55.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.023+0000 7f43de664640 1 -- start start 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43de664640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43d8100620 0x7f43d819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43de664640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 0x7f43d819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43de664640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43d8106630 0x7f43d819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43de664640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8110a90 con 0x7f43d8100fe0 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43de664640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8110910 con 0x7f43d8106630 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43de664640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8110c10 con 0x7f43d8100620 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 0x7f43d819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 0x7f43d819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51236/0 (socket says 172.21.15.27:51236) 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 -- 172.21.15.27:0/666353696 learned_addr learned my addr 172.21.15.27:0/666353696 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43dcc01640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43d8106630 0x7f43d819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d7fff640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43d8100620 0x7f43d819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 -- 172.21.15.27:0/666353696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43d8100620 msgr2=0x7f43d819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43d8100620 0x7f43d819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 -- 172.21.15.27:0/666353696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43d8106630 msgr2=0x7f43d819ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43d8106630 0x7f43d819ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 -- 172.21.15.27:0/666353696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d819f1a0 con 0x7f43d8100fe0 2024-04-25T01:30:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d7fff640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43d8100620 0x7f43d819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.024+0000 7f43d77fe640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 0x7f43d819a6f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f43c000d950 tx=0x7f43c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.025+0000 7f43d57fa640 1 -- 172.21.15.27:0/666353696 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c0017030 con 0x7f43d8100fe0 2024-04-25T01:30:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.025+0000 7f43d57fa640 1 -- 172.21.15.27:0/666353696 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c0004020 con 0x7f43d8100fe0 2024-04-25T01:30:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.025+0000 7f43de664640 1 -- 172.21.15.27:0/666353696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d819f490 con 0x7f43d8100fe0 2024-04-25T01:30:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.025+0000 7f43d57fa640 1 -- 172.21.15.27:0/666353696 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c00047a0 con 0x7f43d8100fe0 2024-04-25T01:30:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.025+0000 7f43de664640 1 -- 172.21.15.27:0/666353696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d81a6d20 con 0x7f43d8100fe0 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43d57fa640 1 -- 172.21.15.27:0/666353696 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f43c00041d0 con 0x7f43d8100fe0 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43e1373740 1 Processor -- start 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43e1373740 1 -- start start 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43e1373740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df15960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43e1373740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df20030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43e1373740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df22960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43e1373740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55974de7a670 con 0x55974df20570 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43e1373740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55974de22970 con 0x55974df15580 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43e1373740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55974de76630 con 0x55974df15ea0 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43d57fa640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43c40a2d00 0x7f43c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df22960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df22960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51244/0 (socket says 172.21.15.27:51244) 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.027+0000 7f43d77fe640 1 -- 172.21.15.27:0/1582644255 learned_addr learned my addr 172.21.15.27:0/1582644255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43dcc01640 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df15960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43d7fff640 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df20030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43dcc01640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43c40a2d00 0x7f43c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43d57fa640 1 -- 172.21.15.27:0/666353696 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f43c00cf1a0 con 0x7f43d8100fe0 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43d77fe640 1 -- 172.21.15.27:0/1582644255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 msgr2=0x55974df20030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43d77fe640 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df20030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43d77fe640 1 -- 172.21.15.27:0/1582644255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 msgr2=0x55974df15960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43d77fe640 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df15960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43d77fe640 1 -- 172.21.15.27:0/1582644255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55974de46400 con 0x55974df20570 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43d77fe640 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df22960 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f43c0097790 tx=0x7f43c0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb02c07e57502344 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43dcc01640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43c40a2d00 0x7f43c40a51c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f43d0005e30 tx=0x7f43d0005ef0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43ce7fc640 1 -- 172.21.15.27:0/1582644255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c009b330 con 0x55974df20570 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43ce7fc640 1 -- 172.21.15.27:0/1582644255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c009ba30 con 0x55974df20570 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.028+0000 7f43ce7fc640 1 -- 172.21.15.27:0/1582644255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c009b590 con 0x55974df20570 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 -- 172.21.15.27:0/1582644255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 msgr2=0x55974df22960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df22960 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f43c0097790 tx=0x7f43c0021000 comp rx=0 tx=0).stop 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 -- 172.21.15.27:0/1582644255 shutdown_connections 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df22960 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df20030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 --2- 172.21.15.27:0/1582644255 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df15960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 -- 172.21.15.27:0/1582644255 >> 172.21.15.27:0/1582644255 conn(0x55974df118a0 msgr2=0x55974df11c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 -- 172.21.15.27:0/1582644255 shutdown_connections 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 -- 172.21.15.27:0/1582644255 wait complete. 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.029+0000 7f43e1373740 1 -- start start 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43e1373740 1 Processor -- start 2024-04-25T01:30:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43e1373740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15580 0x55974df1ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43e1373740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15ea0 0x55974df1d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43e1373740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df1d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43e1373740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55974df232a0 con 0x55974df20570 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43e1373740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55974de36050 con 0x55974df15ea0 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43e1373740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55974df23420 con 0x55974df15580 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43e1373740 10 client.0 _ll_register_callbacks cb 0x55974db8dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df1d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df1d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51252/0 (socket says 172.21.15.27:51252) 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 -- 172.21.15.27:0/4096829251 learned_addr learned my addr 172.21.15.27:0/4096829251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43d7fff640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15580 0x55974df1ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43d77fe640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15ea0 0x55974df1d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 -- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15580 msgr2=0x55974df1ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15580 0x55974df1ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 -- 172.21.15.27:0/4096829251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15ea0 msgr2=0x55974df1d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15ea0 0x55974df1d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 -- 172.21.15.27:0/4096829251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55974de07dc0 con 0x55974df20570 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43d77fe640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15ea0 0x55974df1d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43d7fff640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15580 0x55974df1ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43dcc01640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df1d770 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f43d0009430 tx=0x7f43d0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51c2dc8a596929a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.030+0000 7f43b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43b77fe640 1 -- 172.21.15.27:0/4096829251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d004f070 con 0x55974df20570 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43b77fe640 1 -- 172.21.15.27:0/4096829251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f43d0004720 con 0x55974df20570 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 1 -- 172.21.15.27:0/4096829251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43c009ba30 con 0x55974df20570 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43b77fe640 1 -- 172.21.15.27:0/4096829251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f43d0031d60 con 0x55974df20570 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 10 client.16827 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 1 -- 172.21.15.27:0/4096829251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55974db8e3f0 con 0x55974df20570 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43b77fe640 1 -- 172.21.15.27:0/4096829251 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c009ba30 con 0x55974df20570 2024-04-25T01:30:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43b77fe640 1 -- 172.21.15.27:0/4096829251 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f43d0030070 con 0x55974df20570 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 10 client.16827 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 10 client.16827 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 20 client.16827 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55974df30550 0x55974df32980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 4 client.16827 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43e1373740 1 -- 172.21.15.27:0/4096829251 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55974de87bb0 con 0x55974df30550 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.031+0000 7f43d7fff640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55974df30550 0x55974df32980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.032+0000 7f43b77fe640 10 client.16827 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.032+0000 7f43d7fff640 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55974df30550 0x55974df32980 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5cb114c7690f3df server_cookie=efe0961743bd317c in_seq=0 out_seq=0 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43b77fe640 1 -- 172.21.15.27:0/4096829251 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55974de87bb0 con 0x55974df30550 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43b77fe640 10 client.16827 handle_command_reply: tid=0 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 client.16827 shutdown 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 -- 172.21.15.27:0/4096829251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 msgr2=0x55974df1d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df1d770 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f43d0009430 tx=0x7f43d0009360 comp rx=0 tx=0).stop 2024-04-25T01:30:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 -- 172.21.15.27:0/4096829251 shutdown_connections 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55974df30550 0x55974df32980 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df1d770 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15ea0 0x55974df1d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43b77fe640 0 client.16827 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 --2- 172.21.15.27:0/4096829251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15580 0x55974df1ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 -- 172.21.15.27:0/4096829251 >> 172.21.15.27:0/4096829251 conn(0x55974df118a0 msgr2=0x55974df10b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 -- 172.21.15.27:0/4096829251 shutdown_connections 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 1 -- 172.21.15.27:0/4096829251 wait complete. 2024-04-25T01:30:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.033+0000 7f43e1373740 20 client.16827 trim_cache size 0 max 16384 2024-04-25T01:30:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.046+0000 7f43e1373740 1 Processor -- start 2024-04-25T01:30:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.046+0000 7f43e1373740 1 -- start start 2024-04-25T01:30:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.046+0000 7f43e1373740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df2c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.046+0000 7f43e1373740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df15ea0 0x55974df1fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.046+0000 7f43e1373740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df20570 0x55974df38250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.046+0000 7f43e1373740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81b3730 con 0x55974df15ea0 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.046+0000 7f43e1373740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81b1000 con 0x55974df15580 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.046+0000 7f43e1373740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81aee50 con 0x55974df20570 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df15ea0 0x55974df1fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df15ea0 0x55974df1fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51268/0 (socket says 172.21.15.27:51268) 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 -- 172.21.15.27:0/4108298765 learned_addr learned my addr 172.21.15.27:0/4108298765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 -- 172.21.15.27:0/4108298765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df20570 msgr2=0x55974df38250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 --2- 172.21.15.27:0/4108298765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df20570 0x55974df38250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 -- 172.21.15.27:0/4108298765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 msgr2=0x55974df2c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 --2- 172.21.15.27:0/4108298765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df2c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 -- 172.21.15.27:0/4108298765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55974dc587d0 con 0x55974df15ea0 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43d77fe640 1 --2- 172.21.15.27:0/4108298765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df15ea0 0x55974df1fa30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f43c00ccb60 tx=0x7f43c00cf130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0a60856865f13c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43cdffb640 1 -- 172.21.15.27:0/4108298765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c009ba30 con 0x55974df15ea0 2024-04-25T01:30:55.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43cdffb640 1 -- 172.21.15.27:0/4108298765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c00a3fa0 con 0x55974df15ea0 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.047+0000 7f43cdffb640 1 -- 172.21.15.27:0/4108298765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c009b590 con 0x55974df15ea0 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- 172.21.15.27:0/4108298765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df15ea0 msgr2=0x55974df1fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 --2- 172.21.15.27:0/4108298765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df15ea0 0x55974df1fa30 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f43c00ccb60 tx=0x7f43c00cf130 comp rx=0 tx=0).stop 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- 172.21.15.27:0/4108298765 shutdown_connections 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 --2- 172.21.15.27:0/4108298765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df20570 0x55974df38250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 --2- 172.21.15.27:0/4108298765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df15ea0 0x55974df1fa30 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 --2- 172.21.15.27:0/4108298765 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df2c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- 172.21.15.27:0/4108298765 >> 172.21.15.27:0/4108298765 conn(0x55974df118a0 msgr2=0x55974df18790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- 172.21.15.27:0/4108298765 shutdown_connections 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- 172.21.15.27:0/4108298765 wait complete. 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- start start 2024-04-25T01:30:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 Processor -- start 2024-04-25T01:30:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df242a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df247e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df24d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81af4c0 con 0x55974df20570 2024-04-25T01:30:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81b0930 con 0x55974df15580 2024-04-25T01:30:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.048+0000 7f43e1373740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81afb90 con 0x55974df15ea0 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43e1373740 10 client.0 _ll_register_callbacks cb 0x7f43d00318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df24d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df24d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51282/0 (socket says 172.21.15.27:51282) 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43d7fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df242a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 -- 172.21.15.27:0/3580289604 learned_addr learned my addr 172.21.15.27:0/3580289604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43d77fe640 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df247e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 -- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 msgr2=0x55974df247e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df247e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 -- 172.21.15.27:0/3580289604 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 msgr2=0x55974df242a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df242a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 -- 172.21.15.27:0/3580289604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55974df95cc0 con 0x55974df20570 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43d77fe640 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df247e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43dcc01640 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df24d20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f43d0031cc0 tx=0x7f43d0030300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4ec55449c26649 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43b77fe640 1 -- 172.21.15.27:0/3580289604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0065060 con 0x55974df20570 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43b77fe640 1 -- 172.21.15.27:0/3580289604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f43d0004720 con 0x55974df20570 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43e1373740 1 -- 172.21.15.27:0/3580289604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55974de81f00 con 0x55974df20570 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43b77fe640 1 -- 172.21.15.27:0/3580289604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f43d004f040 con 0x55974df20570 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43e1373740 10 client.16833 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.049+0000 7f43e1373740 1 -- 172.21.15.27:0/3580289604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55974df96fa0 con 0x55974df20570 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43b77fe640 1 -- 172.21.15.27:0/3580289604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55974de81f00 con 0x55974df20570 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43b77fe640 1 -- 172.21.15.27:0/3580289604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f43d005f070 con 0x55974df20570 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43e1373740 10 client.16833 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43e1373740 10 client.16833 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43e1373740 20 client.16833 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43e1373740 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55974df976c0 0x55974df99ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43e1373740 4 client.16833 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43e1373740 1 -- 172.21.15.27:0/3580289604 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55974ddff820 con 0x55974df976c0 2024-04-25T01:30:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43d7fff640 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55974df976c0 0x55974df99ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43b77fe640 10 client.16833 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.050+0000 7f43d7fff640 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55974df976c0 0x55974df99ae0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4922f17a85acce87 server_cookie=9e36622a0ff68d9e in_seq=0 out_seq=0 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43b77fe640 1 -- 172.21.15.27:0/3580289604 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55974ddff820 con 0x55974df976c0 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43b77fe640 10 client.16833 handle_command_reply: tid=0 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 client.16833 shutdown 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 -- 172.21.15.27:0/3580289604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 msgr2=0x55974df24d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df24d20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f43d0031cc0 tx=0x7f43d0030300 comp rx=0 tx=0).stop 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 -- 172.21.15.27:0/3580289604 shutdown_connections 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55974df976c0 0x55974df99ae0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55974df20570 0x55974df24d20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43b77fe640 0 client.16833 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55974df15ea0 0x55974df247e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 --2- 172.21.15.27:0/3580289604 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55974df15580 0x55974df242a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 -- 172.21.15.27:0/3580289604 >> 172.21.15.27:0/3580289604 conn(0x55974df118a0 msgr2=0x55974df16cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 -- 172.21.15.27:0/3580289604 shutdown_connections 2024-04-25T01:30:55.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 1 -- 172.21.15.27:0/3580289604 wait complete. 2024-04-25T01:30:55.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43e1373740 20 client.16833 trim_cache size 0 max 16384 2024-04-25T01:30:55.035 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43de664640 1 -- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43c40a2d00 msgr2=0x7f43c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43de664640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43c40a2d00 0x7f43c40a51c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f43d0005e30 tx=0x7f43d0005ef0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43de664640 1 -- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 msgr2=0x7f43d819a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.051+0000 7f43de664640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 0x7f43d819a6f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f43c000d950 tx=0x7f43c000de10 comp rx=0 tx=0).stop 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.052+0000 7f43de664640 1 -- 172.21.15.27:0/666353696 shutdown_connections 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.052+0000 7f43de664640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43d8106630 0x7f43d819ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.052+0000 7f43de664640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43c40a2d00 0x7f43c40a51c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.052+0000 7f43de664640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d8100fe0 0x7f43d819a6f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.052+0000 7f43de664640 1 --2- 172.21.15.27:0/666353696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43d8100620 0x7f43d819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.052+0000 7f43de664640 1 -- 172.21.15.27:0/666353696 >> 172.21.15.27:0/666353696 conn(0x7f43d80776a0 msgr2=0x7f43d8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.052+0000 7f43de664640 1 -- 172.21.15.27:0/666353696 shutdown_connections 2024-04-25T01:30:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.052+0000 7f43de664640 1 -- 172.21.15.27:0/666353696 wait complete. 2024-04-25T01:30:55.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:30:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 -- 172.21.15.27:0/2491882650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c101710 msgr2=0x7f852c101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 --2- 172.21.15.27:0/2491882650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c101710 0x7f852c101b90 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f8520009930 tx=0x7f852002a420 comp rx=0 tx=0).stop 2024-04-25T01:30:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 -- 172.21.15.27:0/2491882650 shutdown_connections 2024-04-25T01:30:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 --2- 172.21.15.27:0/2491882650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f852c1020d0 0x7f852c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 --2- 172.21.15.27:0/2491882650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c101710 0x7f852c101b90 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 --2- 172.21.15.27:0/2491882650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c107720 0x7f852c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 -- 172.21.15.27:0/2491882650 >> 172.21.15.27:0/2491882650 conn(0x7f852c0fd180 msgr2=0x7f852c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 -- 172.21.15.27:0/2491882650 shutdown_connections 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 -- 172.21.15.27:0/2491882650 wait complete. 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.166+0000 7f85308d4640 1 Processor -- start 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f85308d4640 1 -- start start 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f85308d4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f852c101710 0x7f852c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f85308d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c1020d0 0x7f852c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f85308d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c107720 0x7f852c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f85308d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f852c111b30 con 0x7f852c1020d0 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f85308d4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f852c1119b0 con 0x7f852c101710 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f85308d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f852c111cb0 con 0x7f852c107720 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c1020d0 0x7f852c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c1020d0 0x7f852c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51308/0 (socket says 172.21.15.27:51308) 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 -- 172.21.15.27:0/2821374062 learned_addr learned my addr 172.21.15.27:0/2821374062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f852ad9d640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c107720 0x7f852c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f852a59c640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f852c101710 0x7f852c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 -- 172.21.15.27:0/2821374062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c107720 msgr2=0x7f852c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c107720 0x7f852c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 -- 172.21.15.27:0/2821374062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f852c101710 msgr2=0x7f852c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f852c101710 0x7f852c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 -- 172.21.15.27:0/2821374062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852c19d020 con 0x7f852c1020d0 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f852a59c640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f852c101710 0x7f852c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.167+0000 7f8529d9b640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c1020d0 0x7f852c198570 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f8520009230 tx=0x7f8520004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.168+0000 7f851b7fe640 1 -- 172.21.15.27:0/2821374062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852003f030 con 0x7f852c1020d0 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.168+0000 7f851b7fe640 1 -- 172.21.15.27:0/2821374062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8520003ad0 con 0x7f852c1020d0 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.168+0000 7f85308d4640 1 -- 172.21.15.27:0/2821374062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852c19d2b0 con 0x7f852c1020d0 2024-04-25T01:30:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.168+0000 7f851b7fe640 1 -- 172.21.15.27:0/2821374062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520035070 con 0x7f852c1020d0 2024-04-25T01:30:55.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.168+0000 7f85308d4640 1 -- 172.21.15.27:0/2821374062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f852c1a4ba0 con 0x7f852c1020d0 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.170+0000 7f851b7fe640 1 -- 172.21.15.27:0/2821374062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8520042030 con 0x7f852c1020d0 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f85335e3740 1 Processor -- start 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f85335e3740 1 -- start start 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f85335e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f85335e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f85335e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f85335e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ad8eeb4670 con 0x55ad8ef4fea0 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f85335e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ad8ee5c970 con 0x55ad8ef4f580 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f85335e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ad8eeb0630 con 0x55ad8ef5a570 2024-04-25T01:30:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f852a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f852a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51320/0 (socket says 172.21.15.27:51320) 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f852a59c640 1 -- 172.21.15.27:0/3186672927 learned_addr learned my addr 172.21.15.27:0/3186672927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f852ad9d640 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f851b7fe640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d00 0x7f85140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f8529d9b640 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f852a59c640 1 -- 172.21.15.27:0/3186672927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 msgr2=0x55ad8ef5c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.171+0000 7f852ad9d640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d00 0x7f85140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f852a59c640 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f852a59c640 1 -- 172.21.15.27:0/3186672927 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 msgr2=0x55ad8ef4f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f852a59c640 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f852a59c640 1 -- 172.21.15.27:0/3186672927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad8ee80400 con 0x55ad8ef4fea0 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f851b7fe640 1 -- 172.21.15.27:0/2821374062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f85200efce0 con 0x7f852c1020d0 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f8529d9b640 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f852a59c640 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5a030 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f851c00bcc0 tx=0x7f851c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded0afc83249de80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f852ad9d640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d00 0x7f85140a51c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f85240098b0 tx=0x7f8524004ab0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f8518ff9640 1 -- 172.21.15.27:0/3186672927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c015070 con 0x55ad8ef4fea0 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f8518ff9640 1 -- 172.21.15.27:0/3186672927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f851c01a940 con 0x55ad8ef4fea0 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.172+0000 7f8518ff9640 1 -- 172.21.15.27:0/3186672927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c011ae0 con 0x55ad8ef4fea0 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- 172.21.15.27:0/3186672927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 msgr2=0x55ad8ef5a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5a030 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f851c00bcc0 tx=0x7f851c00f730 comp rx=0 tx=0).stop 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- 172.21.15.27:0/3186672927 shutdown_connections 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5a030 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 --2- 172.21.15.27:0/3186672927 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- 172.21.15.27:0/3186672927 >> 172.21.15.27:0/3186672927 conn(0x55ad8ef4b8a0 msgr2=0x55ad8ef4bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- 172.21.15.27:0/3186672927 shutdown_connections 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- 172.21.15.27:0/3186672927 wait complete. 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- start start 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 Processor -- start 2024-04-25T01:30:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef59ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef57720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ad8ef5d360 con 0x55ad8ef4fea0 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ad8ee70050 con 0x55ad8ef4f580 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ad8ef5d4e0 con 0x55ad8ef5a570 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.173+0000 7f85335e3740 10 client.0 _ll_register_callbacks cb 0x55ad8ebc7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51336/0 (socket says 172.21.15.27:51336) 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 -- 172.21.15.27:0/338336251 learned_addr learned my addr 172.21.15.27:0/338336251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f852ad9d640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef57720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f852a59c640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef59ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 -- 172.21.15.27:0/338336251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 msgr2=0x55ad8ef57720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef57720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 -- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 msgr2=0x55ad8ef59ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef59ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 -- 172.21.15.27:0/338336251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad8ee41dc0 con 0x55ad8ef4fea0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f852ad9d640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef57720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f852a59c640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef59ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8529d9b640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef571c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f85200ed6a0 tx=0x7f8520041ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93bbefbde039829 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8511ffb640 1 -- 172.21.15.27:0/338336251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520049040 con 0x55ad8ef4fea0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8511ffb640 1 -- 172.21.15.27:0/338336251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f85200ed320 con 0x55ad8ef4fea0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f85335e3740 1 -- 172.21.15.27:0/338336251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f851c01a940 con 0x55ad8ef4fea0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8511ffb640 1 -- 172.21.15.27:0/338336251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f85200bc570 con 0x55ad8ef4fea0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f85335e3740 10 client.16845 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f85335e3740 1 -- 172.21.15.27:0/338336251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ad8ebc83f0 con 0x55ad8ef4fea0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.174+0000 7f8511ffb640 1 -- 172.21.15.27:0/338336251 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f851c01a940 con 0x55ad8ef4fea0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f8511ffb640 1 -- 172.21.15.27:0/338336251 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f85200b84e0 con 0x55ad8ef4fea0 2024-04-25T01:30:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f85335e3740 10 client.16845 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f85335e3740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f85335e3740 20 client.16845 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f85335e3740 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ad8ef6a530 0x55ad8ef6c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f85335e3740 4 client.16845 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f85335e3740 1 -- 172.21.15.27:0/338336251 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad8eec1bb0 con 0x55ad8ef6a530 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f852a59c640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ad8ef6a530 0x55ad8ef6c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f8511ffb640 10 client.16845 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.175+0000 7f852a59c640 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ad8ef6a530 0x55ad8ef6c960 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46b0159289f182f4 server_cookie=b2b2619a6f850aa6 in_seq=0 out_seq=0 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f8511ffb640 1 -- 172.21.15.27:0/338336251 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad8eec1bb0 con 0x55ad8ef6a530 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f8511ffb640 10 client.16845 handle_command_reply: tid=0 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 client.16845 shutdown 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 -- 172.21.15.27:0/338336251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 msgr2=0x55ad8ef571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef571c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f85200ed6a0 tx=0x7f8520041ab0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 -- 172.21.15.27:0/338336251 shutdown_connections 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ad8ef6a530 0x55ad8ef6c960 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef57720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef571c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 --2- 172.21.15.27:0/338336251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef59ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f8511ffb640 0 client.16845 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 -- 172.21.15.27:0/338336251 >> 172.21.15.27:0/338336251 conn(0x55ad8ef4b8a0 msgr2=0x55ad8ef4ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 -- 172.21.15.27:0/338336251 shutdown_connections 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 1 -- 172.21.15.27:0/338336251 wait complete. 2024-04-25T01:30:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.176+0000 7f85335e3740 20 client.16845 trim_cache size 0 max 16384 2024-04-25T01:30:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.189+0000 7f85335e3740 1 Processor -- start 2024-04-25T01:30:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.189+0000 7f85335e3740 1 -- start start 2024-04-25T01:30:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f85335e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f85335e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef517e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f85335e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef5a570 0x55ad8ef59a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f85335e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f852c1b1490 con 0x55ad8ef5a570 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f85335e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f852c1aed60 con 0x55ad8ef4f580 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f85335e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f852c1acbb0 con 0x55ad8ef4fea0 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef5a570 0x55ad8ef59a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef5a570 0x55ad8ef59a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51340/0 (socket says 172.21.15.27:51340) 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 -- 172.21.15.27:0/266815065 learned_addr learned my addr 172.21.15.27:0/266815065 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852a59c640 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f8529d9b640 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 -- 172.21.15.27:0/266815065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef4fea0 msgr2=0x55ad8ef517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 -- 172.21.15.27:0/266815065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 msgr2=0x55ad8ef4b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 -- 172.21.15.27:0/266815065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad8ef57a90 con 0x55ad8ef5a570 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f8529d9b640 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852a59c640 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f852ad9d640 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef5a570 0x55ad8ef59a00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8524003c80 tx=0x7f8524009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9371936fbd7de32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f8513fff640 1 -- 172.21.15.27:0/266815065 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852401a040 con 0x55ad8ef5a570 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f8513fff640 1 -- 172.21.15.27:0/266815065 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8524028070 con 0x55ad8ef5a570 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.190+0000 7f8513fff640 1 -- 172.21.15.27:0/266815065 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524023d30 con 0x55ad8ef5a570 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- 172.21.15.27:0/266815065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef5a570 msgr2=0x55ad8ef59a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef5a570 0x55ad8ef59a00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8524003c80 tx=0x7f8524009ad0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- 172.21.15.27:0/266815065 shutdown_connections 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef5a570 0x55ad8ef59a00 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 --2- 172.21.15.27:0/266815065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef4b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- 172.21.15.27:0/266815065 >> 172.21.15.27:0/266815065 conn(0x55ad8ef4b8a0 msgr2=0x55ad8ef52660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- 172.21.15.27:0/266815065 shutdown_connections 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- 172.21.15.27:0/266815065 wait complete. 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- start start 2024-04-25T01:30:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 Processor -- start 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef62be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f852c1ad220 con 0x55ad8ef4fea0 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f852c1ae690 con 0x55ad8ef4f580 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.191+0000 7f85335e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f852c1ad8f0 con 0x55ad8ef5a570 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f85335e3740 10 client.0 _ll_register_callbacks cb 0x7f8520009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51350/0 (socket says 172.21.15.27:51350) 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 -- 172.21.15.27:0/2958775325 learned_addr learned my addr 172.21.15.27:0/2958775325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f852a59c640 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef62be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f852ad9d640 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 -- 172.21.15.27:0/2958775325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 msgr2=0x55ad8ef5fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 -- 172.21.15.27:0/2958775325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 msgr2=0x55ad8ef62be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef62be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 -- 172.21.15.27:0/2958775325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad8ec927d0 con 0x55ad8ef4fea0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8529d9b640 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5f760 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f85200ba120 tx=0x7f85200ed550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ceac6a7ff86ca78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8511ffb640 1 -- 172.21.15.27:0/2958775325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85200d8020 con 0x55ad8ef4fea0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f85335e3740 1 -- 172.21.15.27:0/2958775325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8524028070 con 0x55ad8ef4fea0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8511ffb640 1 -- 172.21.15.27:0/2958775325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8520031dd0 con 0x55ad8ef4fea0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f8511ffb640 1 -- 172.21.15.27:0/2958775325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8520049040 con 0x55ad8ef4fea0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f85335e3740 10 client.16851 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.192+0000 7f85335e3740 1 -- 172.21.15.27:0/2958775325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ad8ec927d0 con 0x55ad8ef4fea0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f8511ffb640 1 -- 172.21.15.27:0/2958775325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8524028070 con 0x55ad8ef4fea0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f8511ffb640 1 -- 172.21.15.27:0/2958775325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f85200a1020 con 0x55ad8ef4fea0 2024-04-25T01:30:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f85335e3740 10 client.16851 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f85335e3740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f85335e3740 20 client.16851 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f85335e3740 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ad8efd1c80 0x55ad8efd40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f85335e3740 4 client.16851 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f85335e3740 1 -- 172.21.15.27:0/2958775325 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad8eebbf00 con 0x55ad8efd1c80 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f852a59c640 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ad8efd1c80 0x55ad8efd40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f852a59c640 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ad8efd1c80 0x55ad8efd40a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c53104311554aa9 server_cookie=eae88c4c670776b5 in_seq=0 out_seq=0 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.193+0000 7f8511ffb640 10 client.16851 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f8511ffb640 1 -- 172.21.15.27:0/2958775325 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad8eebbf00 con 0x55ad8efd1c80 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f8511ffb640 10 client.16851 handle_command_reply: tid=0 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 client.16851 shutdown 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 -- 172.21.15.27:0/2958775325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 msgr2=0x55ad8ef5f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5f760 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f85200ba120 tx=0x7f85200ed550 comp rx=0 tx=0).stop 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 -- 172.21.15.27:0/2958775325 shutdown_connections 2024-04-25T01:30:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ad8efd1c80 0x55ad8efd40a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ad8ef5a570 0x55ad8ef5fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad8ef4fea0 0x55ad8ef5f760 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f8511ffb640 0 client.16851 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 --2- 172.21.15.27:0/2958775325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ad8ef4f580 0x55ad8ef62be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 -- 172.21.15.27:0/2958775325 >> 172.21.15.27:0/2958775325 conn(0x55ad8ef4b8a0 msgr2=0x55ad8ef65270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 -- 172.21.15.27:0/2958775325 shutdown_connections 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 1 -- 172.21.15.27:0/2958775325 wait complete. 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85335e3740 20 client.16851 trim_cache size 0 max 16384 2024-04-25T01:30:55.178 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:30:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 -- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d00 msgr2=0x7f85140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d00 0x7f85140a51c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f85240098b0 tx=0x7f8524004ab0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 -- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c1020d0 msgr2=0x7f852c198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c1020d0 0x7f852c198570 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f8520009230 tx=0x7f8520004270 comp rx=0 tx=0).stop 2024-04-25T01:30:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 -- 172.21.15.27:0/2821374062 shutdown_connections 2024-04-25T01:30:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c107720 0x7f852c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d00 0x7f85140a51c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f852c1020d0 0x7f852c198570 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 --2- 172.21.15.27:0/2821374062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f852c101710 0x7f852c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.194+0000 7f85308d4640 1 -- 172.21.15.27:0/2821374062 >> 172.21.15.27:0/2821374062 conn(0x7f852c0fd180 msgr2=0x7f852c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.195+0000 7f85308d4640 1 -- 172.21.15.27:0/2821374062 shutdown_connections 2024-04-25T01:30:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:55.195+0000 7f85308d4640 1 -- 172.21.15.27:0/2821374062 wait complete. 2024-04-25T01:30:55.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:55 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1417774605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:55.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:55 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1417774605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:55.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:55 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1417774605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:56.183 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:30:56.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:56.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 -- 172.21.15.27:0/1921534388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f901066b0 msgr2=0x7f3f90106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1921534388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f901066b0 0x7f3f90106a90 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3f78009930 tx=0x7f3f7802a420 comp rx=0 tx=0).stop 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 -- 172.21.15.27:0/1921534388 shutdown_connections 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1921534388 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f90100fd0 0x7f3f9010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1921534388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f90100610 0x7f3f90100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1921534388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f901066b0 0x7f3f90106a90 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 -- 172.21.15.27:0/1921534388 >> 172.21.15.27:0/1921534388 conn(0x7f3f90077740 msgr2=0x7f3f90077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 -- 172.21.15.27:0/1921534388 shutdown_connections 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.325+0000 7f3f96fee640 1 -- 172.21.15.27:0/1921534388 wait complete. 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f96fee640 1 Processor -- start 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f96fee640 1 -- start start 2024-04-25T01:30:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f96fee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f90100610 0x7f3f90196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f96fee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f90100fd0 0x7f3f901974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f96fee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f901066b0 0x7f3f9019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f96fee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f90110a30 con 0x7f3f90100610 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f96fee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901108b0 con 0x7f3f901066b0 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f96fee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f90110bb0 con 0x7f3f90100fd0 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f94d8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f90100610 0x7f3f90196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f94d8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f90100610 0x7f3f90196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51366/0 (socket says 172.21.15.27:51366) 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f94d8a640 1 -- 172.21.15.27:0/1664517759 learned_addr learned my addr 172.21.15.27:0/1664517759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f9558b640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f901066b0 0x7f3f9019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f8ffff640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f90100fd0 0x7f3f901974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f94d8a640 1 -- 172.21.15.27:0/1664517759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f90100fd0 msgr2=0x7f3f901974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f90100fd0 0x7f3f901974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f94d8a640 1 -- 172.21.15.27:0/1664517759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f901066b0 msgr2=0x7f3f9019b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f901066b0 0x7f3f9019b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f94d8a640 1 -- 172.21.15.27:0/1664517759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9019bf60 con 0x7f3f90100610 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f8ffff640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f90100fd0 0x7f3f901974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.326+0000 7f3f9558b640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f901066b0 0x7f3f9019b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.327+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f90100610 0x7f3f90196f70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3f78005940 tx=0x7f3f78004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.327+0000 7f3f8dffb640 1 -- 172.21.15.27:0/1664517759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7803f030 con 0x7f3f90100610 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.327+0000 7f3f8dffb640 1 -- 172.21.15.27:0/1664517759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f78003ad0 con 0x7f3f90100610 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.327+0000 7f3f96fee640 1 -- 172.21.15.27:0/1664517759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9019c1f0 con 0x7f3f90100610 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.327+0000 7f3f8dffb640 1 -- 172.21.15.27:0/1664517759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78035070 con 0x7f3f90100610 2024-04-25T01:30:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.327+0000 7f3f96fee640 1 -- 172.21.15.27:0/1664517759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f901a3ae0 con 0x7f3f90100610 2024-04-25T01:30:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.329+0000 7f3f8dffb640 1 -- 172.21.15.27:0/1664517759 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3f78042030 con 0x7f3f90100610 2024-04-25T01:30:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f99ddc740 1 Processor -- start 2024-04-25T01:30:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f99ddc740 1 -- start start 2024-04-25T01:30:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f8dffb640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a53a50 0x5630f0a51ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a52410 0x5630f0a52890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a61970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5630f09b42c0 con 0x5630f0a5f580 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5630f095c910 con 0x5630f0a52410 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5630f09b0280 con 0x5630f0a53a50 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f9558b640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f8dffb640 1 -- 172.21.15.27:0/1664517759 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3f780eff20 con 0x7f3f90100610 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f9558b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a61970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f9558b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a61970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51370/0 (socket says 172.21.15.27:51370) 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f9558b640 1 -- 172.21.15.27:0/2214051455 learned_addr learned my addr 172.21.15.27:0/2214051455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f8ffff640 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a52410 0x5630f0a52890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.330+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a53a50 0x5630f0a51ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f9558b640 1 -- 172.21.15.27:0/2214051455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a53a50 msgr2=0x5630f0a51ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f9558b640 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a53a50 0x5630f0a51ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f9558b640 1 -- 172.21.15.27:0/2214051455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a52410 msgr2=0x5630f0a52890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f9558b640 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a52410 0x5630f0a52890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f9558b640 1 -- 172.21.15.27:0/2214051455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630f096dfb0 con 0x5630f0a5f580 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a53a50 0x5630f0a51ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f9558b640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3f880149e0 tx=0x7f3f8800cba0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f9558b640 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a61970 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f3f880117b0 tx=0x7f3f8801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3146956c5cde6cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f86ffd640 1 -- 172.21.15.27:0/2214051455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88022070 con 0x5630f0a5f580 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f86ffd640 1 -- 172.21.15.27:0/2214051455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f8800fc60 con 0x5630f0a5f580 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.331+0000 7f3f86ffd640 1 -- 172.21.15.27:0/2214051455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88014350 con 0x5630f0a5f580 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 -- 172.21.15.27:0/2214051455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 msgr2=0x5630f0a61970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a61970 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f3f880117b0 tx=0x7f3f8801b000 comp rx=0 tx=0).stop 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 -- 172.21.15.27:0/2214051455 shutdown_connections 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a61970 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a52410 0x5630f0a52890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/2214051455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a53a50 0x5630f0a51ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 -- 172.21.15.27:0/2214051455 >> 172.21.15.27:0/2214051455 conn(0x5630f0a18950 msgr2=0x5630f0a4ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 -- 172.21.15.27:0/2214051455 shutdown_connections 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 -- 172.21.15.27:0/2214051455 wait complete. 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 -- start start 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.332+0000 7f3f99ddc740 1 Processor -- start 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a5c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a53a50 0x5630f0a5c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a5f580 0x5630f0a5cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5630f0a62280 con 0x5630f0a53a50 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5630f096fe90 con 0x5630f0a5f580 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5630f0a62400 con 0x5630f0a52410 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a53a50 0x5630f0a5c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f99ddc740 10 client.0 _ll_register_callbacks cb 0x5630f09ab970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a53a50 0x5630f0a5c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51386/0 (socket says 172.21.15.27:51386) 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 -- 172.21.15.27:0/1503235484 learned_addr learned my addr 172.21.15.27:0/1503235484 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f9558b640 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a5f580 0x5630f0a5cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a5c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 -- 172.21.15.27:0/1503235484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 msgr2=0x5630f0a5c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a5c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 -- 172.21.15.27:0/1503235484 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a5f580 msgr2=0x5630f0a5cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a5f580 0x5630f0a5cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 -- 172.21.15.27:0/1503235484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630f07907e0 con 0x5630f0a53a50 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a5c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f8ffff640 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a53a50 0x5630f0a5c9e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f3f80012750 tx=0x7f3f80012c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59174340a858e2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f6ffff640 1 -- 172.21.15.27:0/1503235484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80012e60 con 0x5630f0a53a50 2024-04-25T01:30:56.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1503235484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f8800fc60 con 0x5630f0a53a50 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.333+0000 7f3f6ffff640 1 -- 172.21.15.27:0/1503235484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3f80010d60 con 0x5630f0a53a50 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f6ffff640 1 -- 172.21.15.27:0/1503235484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f8001b440 con 0x5630f0a53a50 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f99ddc740 10 client.16863 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1503235484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5630f07f0f70 con 0x5630f0a53a50 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f6ffff640 1 -- 172.21.15.27:0/1503235484 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f8800fc60 con 0x5630f0a53a50 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f6ffff640 1 -- 172.21.15.27:0/1503235484 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3f80019070 con 0x5630f0a53a50 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f99ddc740 10 client.16863 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f99ddc740 10 client.16863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f99ddc740 20 client.16863 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5630f0a6f530 0x5630f0a71980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f99ddc740 4 client.16863 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1503235484 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630f09885a0 con 0x5630f0a6f530 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.334+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5630f0a6f530 0x5630f0a71980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.335+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5630f0a6f530 0x5630f0a71980 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d095ae7ae3b6bb0 server_cookie=9d83886b56d6deb0 in_seq=0 out_seq=0 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.335+0000 7f3f6ffff640 10 client.16863 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.335+0000 7f3f6ffff640 1 -- 172.21.15.27:0/1503235484 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5630f09885a0 con 0x5630f0a6f530 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.335+0000 7f3f6ffff640 10 client.16863 handle_command_reply: tid=0 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 client.16863 shutdown 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1503235484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a53a50 msgr2=0x5630f0a5c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a53a50 0x5630f0a5c9e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f3f80012750 tx=0x7f3f80012c10 comp rx=0 tx=0).stop 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1503235484 shutdown_connections 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5630f0a6f530 0x5630f0a71980 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a5f580 0x5630f0a5cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a53a50 0x5630f0a5c9e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1503235484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a5c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f6ffff640 0 client.16863 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1503235484 >> 172.21.15.27:0/1503235484 conn(0x5630f0a18950 msgr2=0x5630f0a54830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1503235484 shutdown_connections 2024-04-25T01:30:56.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1503235484 wait complete. 2024-04-25T01:30:56.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.336+0000 7f3f99ddc740 20 client.16863 trim_cache size 0 max 16384 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f99ddc740 1 Processor -- start 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f99ddc740 1 -- start start 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a52410 0x5630f0a6b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 0x5630f0a4dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a5f580 0x5630f0a4b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901af300 con 0x5630f0a52410 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901acbd0 con 0x5630f0a53a50 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901aaa20 con 0x5630f0a5f580 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f94d8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a52410 0x5630f0a6b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f94d8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a52410 0x5630f0a6b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51394/0 (socket says 172.21.15.27:51394) 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f94d8a640 1 -- 172.21.15.27:0/1423680165 learned_addr learned my addr 172.21.15.27:0/1423680165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f8ffff640 1 --2- 172.21.15.27:0/1423680165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 0x5630f0a4dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f94d8a640 1 -- 172.21.15.27:0/1423680165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a5f580 msgr2=0x5630f0a4b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1423680165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a5f580 0x5630f0a4b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f94d8a640 1 -- 172.21.15.27:0/1423680165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 msgr2=0x5630f0a4dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1423680165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 0x5630f0a4dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.354+0000 7f3f94d8a640 1 -- 172.21.15.27:0/1423680165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630f0688170 con 0x5630f0a52410 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/1423680165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a52410 0x5630f0a6b230 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3f780bbdb0 tx=0x7f3f780dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fbf453d4db940f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f867fc640 1 -- 172.21.15.27:0/1423680165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78049040 con 0x5630f0a52410 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f867fc640 1 -- 172.21.15.27:0/1423680165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f780bc0b0 con 0x5630f0a52410 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f867fc640 1 -- 172.21.15.27:0/1423680165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f780ba440 con 0x5630f0a52410 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1423680165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a52410 msgr2=0x5630f0a6b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1423680165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a52410 0x5630f0a6b230 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3f780bbdb0 tx=0x7f3f780dc040 comp rx=0 tx=0).stop 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1423680165 shutdown_connections 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1423680165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a5f580 0x5630f0a4b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1423680165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 0x5630f0a4dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/1423680165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a52410 0x5630f0a6b230 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1423680165 >> 172.21.15.27:0/1423680165 conn(0x5630f0a18950 msgr2=0x5630f0a4b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1423680165 shutdown_connections 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.355+0000 7f3f99ddc740 1 -- 172.21.15.27:0/1423680165 wait complete. 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 1 -- start start 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 1 Processor -- start 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a64710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 0x5630f0a64c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a65190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901ab090 con 0x5630f0a5f580 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901ac500 con 0x5630f0a53a50 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901ab760 con 0x5630f0a52410 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f99ddc740 10 client.0 _ll_register_callbacks cb 0x7f3f80009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f9558b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a65190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f9558b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a65190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51402/0 (socket says 172.21.15.27:51402) 2024-04-25T01:30:56.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.356+0000 7f3f9558b640 1 -- 172.21.15.27:0/3099727803 learned_addr learned my addr 172.21.15.27:0/3099727803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a64710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f9558b640 1 -- 172.21.15.27:0/3099727803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 msgr2=0x5630f0a64710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f8ffff640 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 0x5630f0a64c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f9558b640 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a64710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f9558b640 1 -- 172.21.15.27:0/3099727803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 msgr2=0x5630f0a64c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f9558b640 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 0x5630f0a64c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f9558b640 1 -- 172.21.15.27:0/3099727803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630f0ad5d60 con 0x5630f0a5f580 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a64710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f9558b640 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a65190 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3f880117b0 tx=0x7f3f8800fe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d5330800c09bea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f6ffff640 1 -- 172.21.15.27:0/3099727803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88034060 con 0x5630f0a5f580 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f6ffff640 1 -- 172.21.15.27:0/3099727803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3f88005040 con 0x5630f0a5f580 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f99ddc740 1 -- 172.21.15.27:0/3099727803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f780bc0b0 con 0x5630f0a5f580 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f6ffff640 1 -- 172.21.15.27:0/3099727803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f8803b050 con 0x5630f0a5f580 2024-04-25T01:30:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f99ddc740 10 client.16869 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f99ddc740 1 -- 172.21.15.27:0/3099727803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5630f0ad7080 con 0x5630f0a5f580 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f6ffff640 1 -- 172.21.15.27:0/3099727803 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f780bc0b0 con 0x5630f0a5f580 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.357+0000 7f3f6ffff640 1 -- 172.21.15.27:0/3099727803 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3f88004980 con 0x5630f0a5f580 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f99ddc740 10 client.16869 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f99ddc740 10 client.16869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f99ddc740 20 client.16869 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5630f0ad7410 0x5630f0ad9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f99ddc740 4 client.16869 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f99ddc740 1 -- 172.21.15.27:0/3099727803 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5630f09c1800 con 0x5630f0ad7410 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5630f0ad7410 0x5630f0ad9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f6ffff640 10 client.16869 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.358+0000 7f3f94d8a640 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5630f0ad7410 0x5630f0ad9800 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb801d6ca21aca90 server_cookie=c286e725c34a595d in_seq=0 out_seq=0 2024-04-25T01:30:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.359+0000 7f3f6ffff640 1 -- 172.21.15.27:0/3099727803 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5630f09c1800 con 0x5630f0ad7410 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.359+0000 7f3f6ffff640 10 client.16869 handle_command_reply: tid=0 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.359+0000 7f3f99ddc740 1 client.16869 shutdown 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 -- 172.21.15.27:0/3099727803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 msgr2=0x5630f0a65190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a65190 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3f880117b0 tx=0x7f3f8800fe90 comp rx=0 tx=0).stop 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 -- 172.21.15.27:0/3099727803 shutdown_connections 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5630f0ad7410 0x5630f0ad9800 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630f0a5f580 0x5630f0a65190 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5630f0a53a50 0x5630f0a64c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 --2- 172.21.15.27:0/3099727803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630f0a52410 0x5630f0a64710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f6ffff640 0 client.16869 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 -- 172.21.15.27:0/3099727803 >> 172.21.15.27:0/3099727803 conn(0x5630f0a18950 msgr2=0x5630f0a54830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 -- 172.21.15.27:0/3099727803 shutdown_connections 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 1 -- 172.21.15.27:0/3099727803 wait complete. 2024-04-25T01:30:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.360+0000 7f3f99ddc740 20 client.16869 trim_cache size 0 max 16384 2024-04-25T01:30:56.343 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:30:56.343 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:30:56.343 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "29d25996-8de1-4cc9-875c-6b2f9dc43e4a", 2024-04-25T01:30:56.343 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:30:56.343 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 -- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f7c0a2d00 msgr2=0x7f3f7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3f880149e0 tx=0x7f3f8800cba0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 -- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f90100610 msgr2=0x7f3f90196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f90100610 0x7f3f90196f70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3f78005940 tx=0x7f3f78004270 comp rx=0 tx=0).stop 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 -- 172.21.15.27:0/1664517759 shutdown_connections 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3f901066b0 0x7f3f9019b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f90100fd0 0x7f3f901974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 --2- 172.21.15.27:0/1664517759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f90100610 0x7f3f90196f70 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 -- 172.21.15.27:0/1664517759 >> 172.21.15.27:0/1664517759 conn(0x7f3f90077740 msgr2=0x7f3f900ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 -- 172.21.15.27:0/1664517759 shutdown_connections 2024-04-25T01:30:56.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.361+0000 7f3f96fee640 1 -- 172.21.15.27:0/1664517759 wait complete. 2024-04-25T01:30:56.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:30:56.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:30:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.476+0000 7fbf0418b640 1 -- 172.21.15.27:0/1192915462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 msgr2=0x7fbefc10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.476+0000 7fbf0418b640 1 --2- 172.21.15.27:0/1192915462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 0x7fbefc10e1a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009ca0 tx=0x7fbef802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.476+0000 7fbf0418b640 1 -- 172.21.15.27:0/1192915462 shutdown_connections 2024-04-25T01:30:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 --2- 172.21.15.27:0/1192915462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 0x7fbefc10e1a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 --2- 172.21.15.27:0/1192915462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbefc1032e0 0x7fbefc10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 --2- 172.21.15.27:0/1192915462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbefc1029c0 0x7fbefc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 -- 172.21.15.27:0/1192915462 >> 172.21.15.27:0/1192915462 conn(0x7fbefc077740 msgr2=0x7fbefc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 -- 172.21.15.27:0/1192915462 shutdown_connections 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 -- 172.21.15.27:0/1192915462 wait complete. 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 Processor -- start 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 -- start start 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbefc1029c0 0x7fbefc100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbefc1032e0 0x7fbefc100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 0x7fbefc101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc111b00 con 0x7fbefc10bd00 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc111980 con 0x7fbefc1032e0 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.477+0000 7fbf0418b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc111c80 con 0x7fbefc1029c0 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 0x7fbefc101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 0x7fbefc101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51424/0 (socket says 172.21.15.27:51424) 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 -- 172.21.15.27:0/2658413618 learned_addr learned my addr 172.21.15.27:0/2658413618 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf01726640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbefc1032e0 0x7fbefc100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf01f27640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbefc1029c0 0x7fbefc100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 -- 172.21.15.27:0/2658413618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbefc1029c0 msgr2=0x7fbefc100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbefc1029c0 0x7fbefc100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 -- 172.21.15.27:0/2658413618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbefc1032e0 msgr2=0x7fbefc100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbefc1032e0 0x7fbefc100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 -- 172.21.15.27:0/2658413618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc1a64a0 con 0x7fbefc10bd00 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf01f27640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbefc1029c0 0x7fbefc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf02728640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 0x7fbefc101090 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbef8006010 tx=0x7fbef8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbef2ffd640 1 -- 172.21.15.27:0/2658413618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8047070 con 0x7fbefc10bd00 2024-04-25T01:30:56.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbef2ffd640 1 -- 172.21.15.27:0/2658413618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef8003e50 con 0x7fbefc10bd00 2024-04-25T01:30:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf0418b640 1 -- 172.21.15.27:0/2658413618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbefc1a6730 con 0x7fbefc10bd00 2024-04-25T01:30:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbef2ffd640 1 -- 172.21.15.27:0/2658413618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8033050 con 0x7fbefc10bd00 2024-04-25T01:30:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.478+0000 7fbf0418b640 1 -- 172.21.15.27:0/2658413618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbefc1a6c70 con 0x7fbefc10bd00 2024-04-25T01:30:56.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.481+0000 7fbef2ffd640 1 -- 172.21.15.27:0/2658413618 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbef804a070 con 0x7fbefc10bd00 2024-04-25T01:30:56.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.481+0000 7fbf06f70740 1 Processor -- start 2024-04-25T01:30:56.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf06f70740 1 -- start start 2024-04-25T01:30:56.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba33a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 0x55692ba3e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba3e620 0x55692ba40a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf06f70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55692b9986b0 con 0x55692ba33630 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf06f70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55692b940880 con 0x55692ba33f50 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf06f70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55692b994670 con 0x55692ba3e620 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba33a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba33a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51432/0 (socket says 172.21.15.27:51432) 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01726640 1 -- 172.21.15.27:0/3773945515 learned_addr learned my addr 172.21.15.27:0/3773945515 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01f27640 1 --2- 172.21.15.27:0/3773945515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 0x55692ba3e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbef2ffd640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbee80a2db0 0x7fbee80a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01726640 1 -- 172.21.15.27:0/3773945515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba3e620 msgr2=0x55692ba40a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01726640 1 --2- 172.21.15.27:0/3773945515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba3e620 0x55692ba40a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01726640 1 -- 172.21.15.27:0/3773945515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 msgr2=0x55692ba3e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01726640 1 --2- 172.21.15.27:0/3773945515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 0x55692ba3e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf01726640 1 -- 172.21.15.27:0/3773945515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55692b7767d0 con 0x55692ba33630 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbf02728640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbee80a2db0 0x7fbee80a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.482+0000 7fbef2ffd640 1 -- 172.21.15.27:0/2658413618 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fbef80f0390 con 0x7fbefc10bd00 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf01726640 1 --2- 172.21.15.27:0/3773945515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba33a10 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fbeec004610 tx=0x7fbeec00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9cca6716e79186 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbee7fff640 1 -- 172.21.15.27:0/3773945515 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec003bd0 con 0x55692ba33630 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbee7fff640 1 -- 172.21.15.27:0/3773945515 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeec016030 con 0x55692ba33630 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbee7fff640 1 -- 172.21.15.27:0/3773945515 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec004db0 con 0x55692ba33630 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf02728640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbee80a2db0 0x7fbee80a5270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbef8049000 tx=0x7fbef8035c50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf06f70740 1 -- 172.21.15.27:0/3773945515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 msgr2=0x55692ba33a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf06f70740 1 --2- 172.21.15.27:0/3773945515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba33a10 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fbeec004610 tx=0x7fbeec00bc20 comp rx=0 tx=0).stop 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf06f70740 1 -- 172.21.15.27:0/3773945515 shutdown_connections 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf06f70740 1 --2- 172.21.15.27:0/3773945515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba3e620 0x55692ba40a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf06f70740 1 --2- 172.21.15.27:0/3773945515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 0x55692ba3e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf06f70740 1 --2- 172.21.15.27:0/3773945515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba33a10 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.483+0000 7fbf06f70740 1 -- 172.21.15.27:0/3773945515 >> 172.21.15.27:0/3773945515 conn(0x55692ba2f950 msgr2=0x55692ba2fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 -- 172.21.15.27:0/3773945515 shutdown_connections 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 -- 172.21.15.27:0/3773945515 wait complete. 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 -- start start 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 Processor -- start 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33630 0x55692ba3b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 0x55692ba3bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba3e620 0x55692ba3c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55692ba413b0 con 0x55692ba3e620 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55692b953f50 con 0x55692ba33f50 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55692ba41530 con 0x55692ba33630 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf06f70740 10 client.0 _ll_register_callbacks cb 0x55692b98f000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.484+0000 7fbf01726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba3e620 0x55692ba3c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba3e620 0x55692ba3c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51436/0 (socket says 172.21.15.27:51436) 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 -- 172.21.15.27:0/2091633646 learned_addr learned my addr 172.21.15.27:0/2091633646 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33630 0x55692ba3b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01f27640 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 0x55692ba3bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 -- 172.21.15.27:0/2091633646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33630 msgr2=0x55692ba3b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33630 0x55692ba3b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 -- 172.21.15.27:0/2091633646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 msgr2=0x55692ba3bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 0x55692ba3bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 -- 172.21.15.27:0/2091633646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55692b6ac3f0 con 0x55692ba3e620 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf01726640 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba3e620 0x55692ba3c050 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fbeec005fd0 tx=0x7fbeec004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7916165d50b278 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbee57fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbee57fa640 1 -- 172.21.15.27:0/2091633646 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec024040 con 0x55692ba3e620 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbee57fa640 1 -- 172.21.15.27:0/2091633646 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fbeec01b670 con 0x55692ba3e620 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf06f70740 1 -- 172.21.15.27:0/2091633646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeec016030 con 0x55692ba3e620 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbee57fa640 1 -- 172.21.15.27:0/2091633646 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbeec01b870 con 0x55692ba3e620 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf06f70740 10 client.16881 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbf06f70740 1 -- 172.21.15.27:0/2091633646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55692b66c0f0 con 0x55692ba3e620 2024-04-25T01:30:56.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.485+0000 7fbee57fa640 1 -- 172.21.15.27:0/2091633646 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeec016030 con 0x55692ba3e620 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbee57fa640 1 -- 172.21.15.27:0/2091633646 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbeec00fa60 con 0x55692ba3e620 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbf06f70740 10 client.16881 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbf06f70740 10 client.16881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbf06f70740 20 client.16881 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55692ba4e7b0 0x55692ba50ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbf06f70740 4 client.16881 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbf06f70740 1 -- 172.21.15.27:0/2091633646 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55692b9a5bf0 con 0x55692ba4e7b0 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbf01f27640 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55692ba4e7b0 0x55692ba50ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbf01f27640 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55692ba4e7b0 0x55692ba50ba0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb4d3fa72be9ea3c server_cookie=ca0851eea8581a2a in_seq=0 out_seq=0 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.486+0000 7fbee57fa640 10 client.16881 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbee57fa640 1 -- 172.21.15.27:0/2091633646 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55692b9a5bf0 con 0x55692ba4e7b0 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbee57fa640 10 client.16881 handle_command_reply: tid=0 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 client.16881 shutdown 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 -- 172.21.15.27:0/2091633646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba3e620 msgr2=0x55692ba3c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba3e620 0x55692ba3c050 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fbeec005fd0 tx=0x7fbeec004990 comp rx=0 tx=0).stop 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 -- 172.21.15.27:0/2091633646 shutdown_connections 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55692ba4e7b0 0x55692ba50ba0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba3e620 0x55692ba3c050 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba33f50 0x55692ba3bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2091633646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33630 0x55692ba3b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbee57fa640 0 client.16881 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.487+0000 7fbf06f70740 1 -- 172.21.15.27:0/2091633646 >> 172.21.15.27:0/2091633646 conn(0x55692ba2f950 msgr2=0x55692ba35650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.488+0000 7fbf06f70740 1 -- 172.21.15.27:0/2091633646 shutdown_connections 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.488+0000 7fbf06f70740 1 -- 172.21.15.27:0/2091633646 wait complete. 2024-04-25T01:30:56.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.488+0000 7fbf06f70740 20 client.16881 trim_cache size 0 max 16384 2024-04-25T01:30:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf06f70740 1 Processor -- start 2024-04-25T01:30:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf06f70740 1 -- start start 2024-04-25T01:30:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba4a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 0x55692ba3b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 0x55692ba35d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf06f70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc1b36a0 con 0x55692ba33630 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf06f70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc1b0f70 con 0x55692ba3e620 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf06f70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc1aedc0 con 0x55692ba33f50 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf01726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba4a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf01726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba4a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51442/0 (socket says 172.21.15.27:51442) 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf01726640 1 -- 172.21.15.27:0/2709619160 learned_addr learned my addr 172.21.15.27:0/2709619160 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf01f27640 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 0x55692ba3b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.506+0000 7fbf01726640 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 0x55692ba35d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.507+0000 7fbf01726640 1 -- 172.21.15.27:0/2709619160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 msgr2=0x55692ba3b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.507+0000 7fbf01726640 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 0x55692ba3b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.507+0000 7fbf01726640 1 -- 172.21.15.27:0/2709619160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 msgr2=0x55692ba35d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.507+0000 7fbf01726640 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 0x55692ba35d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.507+0000 7fbf01726640 1 -- 172.21.15.27:0/2709619160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55692ba33d80 con 0x55692ba33630 2024-04-25T01:30:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.507+0000 7fbf01726640 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba4a270 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fbeec004520 tx=0x7fbeec00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8779a33267af614 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.508+0000 7fbee77fe640 1 -- 172.21.15.27:0/2709619160 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec02a560 con 0x55692ba33630 2024-04-25T01:30:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.508+0000 7fbee77fe640 1 -- 172.21.15.27:0/2709619160 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeec016070 con 0x55692ba33630 2024-04-25T01:30:56.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.508+0000 7fbee77fe640 1 -- 172.21.15.27:0/2709619160 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec01b470 con 0x55692ba33630 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 -- 172.21.15.27:0/2709619160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 msgr2=0x55692ba4a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba4a270 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fbeec004520 tx=0x7fbeec00bc20 comp rx=0 tx=0).stop 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 -- 172.21.15.27:0/2709619160 shutdown_connections 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 0x55692ba35d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 0x55692ba3b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 --2- 172.21.15.27:0/2709619160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba4a270 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 -- 172.21.15.27:0/2709619160 >> 172.21.15.27:0/2709619160 conn(0x55692ba2f950 msgr2=0x55692ba2eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 -- 172.21.15.27:0/2709619160 shutdown_connections 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 -- 172.21.15.27:0/2709619160 wait complete. 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.509+0000 7fbf06f70740 1 -- start start 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf06f70740 1 Processor -- start 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba384e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 0x55692ba38a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf06f70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 0x55692ba38f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf06f70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc1af430 con 0x55692ba33630 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf06f70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc1b08a0 con 0x55692ba3e620 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf06f70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc1afb00 con 0x55692ba33f50 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf06f70740 10 client.0 _ll_register_callbacks cb 0x7fbef400b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba384e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51454/0 (socket says 172.21.15.27:51454) 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 -- 172.21.15.27:0/1946532113 learned_addr learned my addr 172.21.15.27:0/1946532113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01f27640 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 0x55692ba38a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 0x55692ba38f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 -- 172.21.15.27:0/1946532113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 msgr2=0x55692ba38a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 0x55692ba38a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 -- 172.21.15.27:0/1946532113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 msgr2=0x55692ba38f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 0x55692ba38f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.510+0000 7fbf01726640 1 -- 172.21.15.27:0/1946532113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55692bab3dd0 con 0x55692ba33630 2024-04-25T01:30:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf01726640 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba384e0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbeec010010 tx=0x7fbeec02af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cff10a0a748770 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbee57fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbee57fa640 1 -- 172.21.15.27:0/1946532113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec003ff0 con 0x55692ba33630 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbee57fa640 1 -- 172.21.15.27:0/1946532113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fbeec01ba10 con 0x55692ba33630 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 1 -- 172.21.15.27:0/1946532113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeec016070 con 0x55692ba33630 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbee57fa640 1 -- 172.21.15.27:0/1946532113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbeec01bc10 con 0x55692ba33630 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 10 client.16887 fetch_fsmap learned FSMap version 27 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 1 -- 172.21.15.27:0/1946532113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55692bab50e0 con 0x55692ba33630 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbee57fa640 1 -- 172.21.15.27:0/1946532113 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeec016070 con 0x55692ba33630 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbee57fa640 1 -- 172.21.15.27:0/1946532113 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbeec005320 con 0x55692ba33630 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 10 client.16887 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 10 client.16887 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 20 client.16887 populate_metadata read hostname 'smithi027' 2024-04-25T01:30:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55692bab5690 0x55692bab7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 4 client.16887 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.511+0000 7fbf06f70740 1 -- 172.21.15.27:0/1946532113 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55692b99ff40 con 0x55692bab5690 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.512+0000 7fbf01f27640 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55692bab5690 0x55692bab7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.512+0000 7fbee57fa640 10 client.16887 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.512+0000 7fbf01f27640 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55692bab5690 0x55692bab7a80 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37c97ce3bb27826e server_cookie=d5d5b78f2cb8e9d0 in_seq=0 out_seq=0 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.512+0000 7fbee57fa640 1 -- 172.21.15.27:0/1946532113 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55692b99ff40 con 0x55692bab5690 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.512+0000 7fbee57fa640 10 client.16887 handle_command_reply: tid=0 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.512+0000 7fbf06f70740 1 client.16887 shutdown 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 -- 172.21.15.27:0/1946532113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 msgr2=0x55692ba384e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba384e0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbeec010010 tx=0x7fbeec02af20 comp rx=0 tx=0).stop 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 -- 172.21.15.27:0/1946532113 shutdown_connections 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55692bab5690 0x55692bab7a80 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55692ba3e620 0x55692ba38f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbee57fa640 0 client.16887 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:30:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55692ba33f50 0x55692ba38a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 --2- 172.21.15.27:0/1946532113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55692ba33630 0x55692ba384e0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 -- 172.21.15.27:0/1946532113 >> 172.21.15.27:0/1946532113 conn(0x55692ba2f950 msgr2=0x55692ba39b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 -- 172.21.15.27:0/1946532113 shutdown_connections 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 1 -- 172.21.15.27:0/1946532113 wait complete. 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf06f70740 20 client.16887 trim_cache size 0 max 16384 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "29d25996-8de1-4cc9-875c-6b2f9dc43e4a": { 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "29d25996-8de1-4cc9-875c-6b2f9dc43e4a", 2024-04-25T01:30:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:30:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:30:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:30:56.497 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:30:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 -- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbee80a2db0 msgr2=0x7fbee80a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbee80a2db0 0x7fbee80a5270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbef8049000 tx=0x7fbef8035c50 comp rx=0 tx=0).stop 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 -- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 msgr2=0x7fbefc101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 0x7fbefc101090 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbef8006010 tx=0x7fbef8003ce0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 -- 172.21.15.27:0/2658413618 shutdown_connections 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbee80a2db0 0x7fbee80a5270 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbefc10bd00 0x7fbefc101090 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbefc1032e0 0x7fbefc100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 --2- 172.21.15.27:0/2658413618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbefc1029c0 0x7fbefc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.513+0000 7fbf0418b640 1 -- 172.21.15.27:0/2658413618 >> 172.21.15.27:0/2658413618 conn(0x7fbefc077740 msgr2=0x7fbefc0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.514+0000 7fbf0418b640 1 -- 172.21.15.27:0/2658413618 shutdown_connections 2024-04-25T01:30:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:30:56.514+0000 7fbf0418b640 1 -- 172.21.15.27:0/2658413618 wait complete. 2024-04-25T01:30:56.501 INFO:tasks.cephfs.filesystem:scrub status for tag:29d25996-8de1-4cc9-875c-6b2f9dc43e4a - {'path': '/', 'tag': '29d25996-8de1-4cc9-875c-6b2f9dc43e4a', 'options': 'recursive,force'} 2024-04-25T01:30:56.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:56 smithi153 ceph-mon[32621]: pgmap v168: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:56.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:56 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:30:56.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:56 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:56.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:56 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:30:56.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:56 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:56.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:56 smithi139 ceph-mon[32760]: pgmap v168: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:56.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:56 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:30:56.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:56 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:56.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:56 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:30:56.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:56 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:56.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:56 smithi027 ceph-mon[29197]: pgmap v168: 129 pgs: 129 active+clean; 2.8 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:30:56.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:56 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:30:56.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:56 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:56.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:56 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:30:56.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:56 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:57.480 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.480 INFO:tasks.workunit.client.0.smithi027.stdout:189/600: open fd = 3 2024-04-25T01:30:57.480 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.480 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.480 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.480 INFO:tasks.workunit.client.0.smithi027.stdout:190/600: open fd = 3 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:191/600: open fd = 3 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:192/600: open fd = 3 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:193/600: open fd = 3 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.481 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:194/600: open fd = 3 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:195/600: open fd = 3 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:196/600: open fd = 3 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:197/600: open fd = 3 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.482 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:198/600: open fd = 3 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:199/600: open fd = 3 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:200/600: open fd = 3 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:201/600: open fd = 3 2024-04-25T01:30:57.483 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:202/600: open fd = 3 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:203/600: open fd = 3 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:204/600: open fd = 3 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:205/600: open fd = 3 2024-04-25T01:30:57.484 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:206/600: open fd = 3 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:207/600: open fd = 3 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:208/600: open fd = 3 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:209/600: open fd = 3 2024-04-25T01:30:57.485 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:210/600: open fd = 3 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:211/600: open fd = 3 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:212/600: open fd = 3 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.486 INFO:tasks.workunit.client.0.smithi027.stdout:213/600: open fd = 3 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:214/600: open fd = 3 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:215/600: open fd = 3 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:216/600: open fd = 3 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.487 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:217/600: open fd = 3 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:218/600: open fd = 3 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:219/600: open fd = 3 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:220/600: open fd = 3 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.488 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:221/600: open fd = 3 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:222/600: open fd = 3 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:223/600: open fd = 3 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:224/600: open fd = 3 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.489 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:225/600: open fd = 3 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:226/600: open fd = 3 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:227/600: open fd = 3 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:228/600: open fd = 3 2024-04-25T01:30:57.490 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:229/600: open fd = 3 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:230/600: open fd = 3 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:231/600: open fd = 3 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:232/600: open fd = 3 2024-04-25T01:30:57.491 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:233/600: open fd = 3 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:234/600: open fd = 3 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:235/600: open fd = 3 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:236/600: open fd = 3 2024-04-25T01:30:57.492 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:237/600: open fd = 3 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:238/600: open fd = 3 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:239/600: open fd = 3 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.493 INFO:tasks.workunit.client.0.smithi027.stdout:240/600: open fd = 3 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:241/600: open fd = 3 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:242/600: open fd = 3 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:243/600: open fd = 3 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.494 INFO:tasks.workunit.client.0.smithi027.stdout:244/600: open fd = 3 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:245/600: open fd = 3 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:246/600: open fd = 3 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:247/600: open fd = 3 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.495 INFO:tasks.workunit.client.0.smithi027.stdout:248/600: open fd = 3 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:249/600: open fd = 3 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:250/600: open fd = 3 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.496 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:30:57.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:57 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:30:57.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:57 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:57.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:57 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:30:57.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:57 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:57.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:57 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:30:57.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:57 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:30:58.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:30:58 smithi153 ceph-mon[32621]: pgmap v169: 129 pgs: 129 active+clean; 2.8 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:58.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:30:58 smithi139 ceph-mon[32760]: pgmap v169: 129 pgs: 129 active+clean; 2.8 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:30:58.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:30:58 smithi027 ceph-mon[29197]: pgmap v169: 129 pgs: 129 active+clean; 2.8 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:31:00.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:00 smithi153 ceph-mon[32621]: pgmap v170: 129 pgs: 129 active+clean; 2.8 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:31:00.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:00 smithi139 ceph-mon[32760]: pgmap v170: 129 pgs: 129 active+clean; 2.8 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:31:00.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:00 smithi027 ceph-mon[29197]: pgmap v170: 129 pgs: 129 active+clean; 2.8 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T01:31:02.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:02 smithi153 ceph-mon[32621]: pgmap v171: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:31:02.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:02 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:31:02.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:02 smithi139 ceph-mon[32760]: pgmap v171: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:31:02.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:02 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:31:02.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:02 smithi027 ceph-mon[29197]: pgmap v171: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 34 KiB/s wr, 29 op/s 2024-04-25T01:31:02.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:02 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:31:03.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:03 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:31:03.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:03 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:31:03.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:03 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:31:04.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:04 smithi153 ceph-mon[32621]: pgmap v172: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-25T01:31:04.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:04 smithi139 ceph-mon[32760]: pgmap v172: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-25T01:31:04.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:04 smithi027 ceph-mon[29197]: pgmap v172: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-25T01:31:06.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:06 smithi153 ceph-mon[32621]: pgmap v173: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-25T01:31:06.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:06 smithi139 ceph-mon[32760]: pgmap v173: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-25T01:31:06.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:06 smithi027 ceph-mon[29197]: pgmap v173: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-25T01:31:08.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:08 smithi153 ceph-mon[32621]: pgmap v174: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 37 KiB/s wr, 29 op/s 2024-04-25T01:31:08.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:08 smithi139 ceph-mon[32760]: pgmap v174: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 37 KiB/s wr, 29 op/s 2024-04-25T01:31:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:08 smithi027 ceph-mon[29197]: pgmap v174: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 37 KiB/s wr, 29 op/s 2024-04-25T01:31:10.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:10 smithi153 ceph-mon[32621]: pgmap v175: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 25 op/s 2024-04-25T01:31:10.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:10 smithi139 ceph-mon[32760]: pgmap v175: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 25 op/s 2024-04-25T01:31:10.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:10 smithi027 ceph-mon[29197]: pgmap v175: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 25 op/s 2024-04-25T01:31:12.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:12 smithi153 ceph-mon[32621]: pgmap v176: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:31:12.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:12 smithi139 ceph-mon[32760]: pgmap v176: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:31:12.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:12 smithi027 ceph-mon[29197]: pgmap v176: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 29 op/s 2024-04-25T01:31:14.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:14 smithi139 ceph-mon[32760]: pgmap v177: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:31:14.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:14 smithi027 ceph-mon[29197]: pgmap v177: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:31:15.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:14 smithi153 ceph-mon[32621]: pgmap v177: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:31:16.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:16 smithi139 ceph-mon[32760]: pgmap v178: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:31:16.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:16 smithi027 ceph-mon[29197]: pgmap v178: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:31:17.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:16 smithi153 ceph-mon[32621]: pgmap v178: 129 pgs: 129 active+clean; 2.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:31:18.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:18 smithi139 ceph-mon[32760]: pgmap v179: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:18.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:18 smithi027 ceph-mon[29197]: pgmap v179: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:19.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:18 smithi153 ceph-mon[32621]: pgmap v179: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:20.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:20 smithi139 ceph-mon[32760]: pgmap v180: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:31:20.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:20 smithi027 ceph-mon[29197]: pgmap v180: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:31:21.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:20 smithi153 ceph-mon[32621]: pgmap v180: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:31:22.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:22 smithi139 ceph-mon[32760]: pgmap v181: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:31:22.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:22 smithi027 ceph-mon[29197]: pgmap v181: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:31:23.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:22 smithi153 ceph-mon[32621]: pgmap v181: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:31:24.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:24 smithi027 ceph-mon[29197]: pgmap v182: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:25.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:24 smithi153 ceph-mon[32621]: pgmap v182: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:25.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:24 smithi139 ceph-mon[32760]: pgmap v182: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:26.502 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:31:26.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:31:26.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 -- 172.21.15.27:0/3679756854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a20100fd0 msgr2=0x7f0a2010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 --2- 172.21.15.27:0/3679756854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a20100fd0 0x7f0a2010d4e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009cd0 tx=0x7f0a1002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 -- 172.21.15.27:0/3679756854 shutdown_connections 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 --2- 172.21.15.27:0/3679756854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a20100fd0 0x7f0a2010d4e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 --2- 172.21.15.27:0/3679756854 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a20100610 0x7f0a20100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 --2- 172.21.15.27:0/3679756854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a201066b0 0x7f0a20106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 -- 172.21.15.27:0/3679756854 >> 172.21.15.27:0/3679756854 conn(0x7f0a20077740 msgr2=0x7f0a20077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 -- 172.21.15.27:0/3679756854 shutdown_connections 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 -- 172.21.15.27:0/3679756854 wait complete. 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.643+0000 7f0a252e1640 1 Processor -- start 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a252e1640 1 -- start start 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a252e1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a20100610 0x7f0a20198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a252e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a20100fd0 0x7f0a20199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a252e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a201066b0 0x7f0a201a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a252e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201109f0 con 0x7f0a201066b0 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a252e1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20110870 con 0x7f0a20100610 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a252e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20110b70 con 0x7f0a20100fd0 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a201066b0 0x7f0a201a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a201066b0 0x7f0a201a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55690/0 (socket says 172.21.15.27:55690) 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1effd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a20100610 0x7f0a20198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1f7fe640 1 -- 172.21.15.27:0/2574377587 learned_addr learned my addr 172.21.15.27:0/2574377587 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a20100fd0 0x7f0a20199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1f7fe640 1 -- 172.21.15.27:0/2574377587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a20100fd0 msgr2=0x7f0a20199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a20100fd0 0x7f0a20199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1f7fe640 1 -- 172.21.15.27:0/2574377587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a20100610 msgr2=0x7f0a20198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a20100610 0x7f0a20198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1f7fe640 1 -- 172.21.15.27:0/2574377587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a201a43b0 con 0x7f0a201066b0 2024-04-25T01:31:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.644+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a20100610 0x7f0a20198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:26.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.645+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a201066b0 0x7f0a201a3e70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0a10005cc0 tx=0x7f0a10032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:26.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.645+0000 7f0a07fff640 1 -- 172.21.15.27:0/2574377587 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10004040 con 0x7f0a201066b0 2024-04-25T01:31:26.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.645+0000 7f0a07fff640 1 -- 172.21.15.27:0/2574377587 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a100046c0 con 0x7f0a201066b0 2024-04-25T01:31:26.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.645+0000 7f0a252e1640 1 -- 172.21.15.27:0/2574377587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a201a4580 con 0x7f0a201066b0 2024-04-25T01:31:26.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.645+0000 7f0a07fff640 1 -- 172.21.15.27:0/2574377587 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10030e00 con 0x7f0a201066b0 2024-04-25T01:31:26.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.645+0000 7f0a252e1640 1 -- 172.21.15.27:0/2574377587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a201a49e0 con 0x7f0a201066b0 2024-04-25T01:31:26.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.647+0000 7f0a07fff640 1 -- 172.21.15.27:0/2574377587 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0a100305a0 con 0x7f0a201066b0 2024-04-25T01:31:26.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.647+0000 7f0a07fff640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0a080a2d00 0x7f0a080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.647+0000 7f0a27ff5740 1 Processor -- start 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a27ff5740 1 -- start start 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0a080a2d00 0x7f0a080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a07fff640 1 -- 172.21.15.27:0/2574377587 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0a1002e030 con 0x7f0a201066b0 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 0x55d000968740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968d10 0x55d0009774f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000977a30 0x55d000979e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d0008cd6b0 con 0x55d000968d10 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d000875880 con 0x55d000968360 2024-04-25T01:31:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d0008c9670 con 0x55d000977a30 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968d10 0x55d0009774f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968d10 0x55d0009774f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55702/0 (socket says 172.21.15.27:55702) 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1e7fc640 1 -- 172.21.15.27:0/1295000428 learned_addr learned my addr 172.21.15.27:0/1295000428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1effd640 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 0x55d000968740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000977a30 0x55d000979e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0a080a2d00 0x7f0a080a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0a18005fd0 tx=0x7f0a1800b390 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1e7fc640 1 -- 172.21.15.27:0/1295000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000977a30 msgr2=0x55d000979e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000977a30 0x55d000979e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1e7fc640 1 -- 172.21.15.27:0/1295000428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 msgr2=0x55d000968740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 0x55d000968740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.648+0000 7f0a1e7fc640 1 -- 172.21.15.27:0/1295000428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0006ab7d0 con 0x55d000968d10 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.649+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968d10 0x55d0009774f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009c90 tx=0x7f0a14011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b593035337357814 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.649+0000 7f0a057fa640 1 -- 172.21.15.27:0/1295000428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1401e070 con 0x55d000968d10 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.649+0000 7f0a057fa640 1 -- 172.21.15.27:0/1295000428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a14019b10 con 0x55d000968d10 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.649+0000 7f0a057fa640 1 -- 172.21.15.27:0/1295000428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400ea70 con 0x55d000968d10 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.649+0000 7f0a27ff5740 1 -- 172.21.15.27:0/1295000428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968d10 msgr2=0x55d0009774f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.649+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968d10 0x55d0009774f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009c90 tx=0x7f0a14011810 comp rx=0 tx=0).stop 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.649+0000 7f0a27ff5740 1 -- 172.21.15.27:0/1295000428 shutdown_connections 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000977a30 0x55d000979e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968d10 0x55d0009774f0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/1295000428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 0x55d000968740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 -- 172.21.15.27:0/1295000428 >> 172.21.15.27:0/1295000428 conn(0x55d000964950 msgr2=0x55d000964d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 -- 172.21.15.27:0/1295000428 shutdown_connections 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 -- 172.21.15.27:0/1295000428 wait complete. 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 -- start start 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 Processor -- start 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009749e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000974f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 0x55d000975460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d00097a7f0 con 0x55d000968360 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d000888f50 con 0x55d000977a30 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d00097a970 con 0x55d000968d10 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a27ff5740 10 client.0 _ll_register_callbacks cb 0x55d0008c4000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009749e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009749e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55718/0 (socket says 172.21.15.27:55718) 2024-04-25T01:31:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a1effd640 1 -- 172.21.15.27:0/2283143236 learned_addr learned my addr 172.21.15.27:0/2283143236 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000974f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.650+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 0x55d000975460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a1effd640 1 -- 172.21.15.27:0/2283143236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 msgr2=0x55d000974f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000974f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a1effd640 1 -- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 msgr2=0x55d000975460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 0x55d000975460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a1effd640 1 -- 172.21.15.27:0/2283143236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0005e13f0 con 0x55d000968360 2024-04-25T01:31:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 0x55d000975460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009749e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f0a18011af0 tx=0x7f0a18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52749f4f83686d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a02ff5640 1 -- 172.21.15.27:0/2283143236 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18005020 con 0x55d000968360 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a02ff5640 1 -- 172.21.15.27:0/2283143236 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0a180051c0 con 0x55d000968360 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2283143236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a14019b10 con 0x55d000968360 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a02ff5640 1 -- 172.21.15.27:0/2283143236 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0a180053c0 con 0x55d000968360 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a27ff5740 10 client.16899 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2283143236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0005a10f0 con 0x55d000968360 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.651+0000 7f0a02ff5640 1 -- 172.21.15.27:0/2283143236 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a14019b10 con 0x55d000968360 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a02ff5640 1 -- 172.21.15.27:0/2283143236 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0a18026070 con 0x55d000968360 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a27ff5740 10 client.16899 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a27ff5740 10 client.16899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a27ff5740 20 client.16899 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d000987e90 0x55d00098a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a27ff5740 4 client.16899 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2283143236 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0008dabf0 con 0x55d000987e90 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d000987e90 0x55d00098a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a02ff5640 10 client.16899 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.652+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d000987e90 0x55d00098a2d0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59820152e6433d67 server_cookie=663b04dc408dacb3 in_seq=0 out_seq=0 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a02ff5640 1 -- 172.21.15.27:0/2283143236 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0008dabf0 con 0x55d000987e90 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a02ff5640 10 client.16899 handle_command_reply: tid=0 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 client.16899 shutdown 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2283143236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 msgr2=0x55d0009749e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009749e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f0a18011af0 tx=0x7f0a18004730 comp rx=0 tx=0).stop 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2283143236 shutdown_connections 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d000987e90 0x55d00098a2d0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 0x55d000975460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000974f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2283143236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009749e0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a02ff5640 0 client.16899 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2283143236 >> 172.21.15.27:0/2283143236 conn(0x55d000964950 msgr2=0x55d00096a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2283143236 shutdown_connections 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2283143236 wait complete. 2024-04-25T01:31:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.653+0000 7f0a27ff5740 20 client.16899 trim_cache size 0 max 16384 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.671+0000 7f0a27ff5740 1 Processor -- start 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.671+0000 7f0a27ff5740 1 -- start start 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009836f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000963970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 0x55d00098fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201af210 con 0x55d000968360 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201acae0 con 0x55d000977a30 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201aa930 con 0x55d000968d10 2024-04-25T01:31:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009836f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009836f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55730/0 (socket says 172.21.15.27:55730) 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 -- 172.21.15.27:0/2027181927 learned_addr learned my addr 172.21.15.27:0/2027181927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 -- 172.21.15.27:0/2027181927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 msgr2=0x55d000963970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2027181927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000963970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 -- 172.21.15.27:0/2027181927 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 msgr2=0x55d00098fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2027181927 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 0x55d00098fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 -- 172.21.15.27:0/2027181927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d000964780 con 0x55d000968360 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a1effd640 1 --2- 172.21.15.27:0/2027181927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009836f0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0a180046c0 tx=0x7f0a1800bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b30202f7c61f9c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a04ff9640 1 -- 172.21.15.27:0/2027181927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1803a020 con 0x55d000968360 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a04ff9640 1 -- 172.21.15.27:0/2027181927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a180273f0 con 0x55d000968360 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.672+0000 7f0a04ff9640 1 -- 172.21.15.27:0/2027181927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a180363f0 con 0x55d000968360 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2027181927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 msgr2=0x55d0009836f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2027181927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009836f0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0a180046c0 tx=0x7f0a1800bc90 comp rx=0 tx=0).stop 2024-04-25T01:31:26.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2027181927 shutdown_connections 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2027181927 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000977a30 0x55d00098fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2027181927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000963970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/2027181927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000968360 0x55d0009836f0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2027181927 >> 172.21.15.27:0/2027181927 conn(0x55d000964950 msgr2=0x55d000963260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2027181927 shutdown_connections 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 -- 172.21.15.27:0/2027181927 wait complete. 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 -- start start 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.673+0000 7f0a27ff5740 1 Processor -- start 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 0x55d00097fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000971210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000977a30 0x55d000971750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201aafa0 con 0x55d000977a30 2024-04-25T01:31:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201ac410 con 0x55d000968360 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201ab670 con 0x55d000968d10 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 10 client.0 _ll_register_callbacks cb 0x7f0a180114e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000977a30 0x55d000971750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000977a30 0x55d000971750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55738/0 (socket says 172.21.15.27:55738) 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 -- 172.21.15.27:0/3837821250 learned_addr learned my addr 172.21.15.27:0/3837821250 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1effd640 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 0x55d00097fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000971210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 -- 172.21.15.27:0/3837821250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 msgr2=0x55d000971210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000971210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 -- 172.21.15.27:0/3837821250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 msgr2=0x55d00097fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 0x55d00097fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 -- 172.21.15.27:0/3837821250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d000971d90 con 0x55d000977a30 2024-04-25T01:31:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1e7fc640 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000971210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a1f7fe640 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000977a30 0x55d000971750 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0a100ee800 tx=0x7f0a100099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e0d0bca5cf876e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a02ff5640 1 -- 172.21.15.27:0/3837821250 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10031e60 con 0x55d000977a30 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a02ff5640 1 -- 172.21.15.27:0/3837821250 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0a100b9750 con 0x55d000977a30 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 1 -- 172.21.15.27:0/3837821250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a180273f0 con 0x55d000977a30 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a02ff5640 1 -- 172.21.15.27:0/3837821250 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0a100dd4a0 con 0x55d000977a30 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 10 client.16905 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.674+0000 7f0a27ff5740 1 -- 172.21.15.27:0/3837821250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0009ee290 con 0x55d000977a30 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a02ff5640 1 -- 172.21.15.27:0/3837821250 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a180273f0 con 0x55d000977a30 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a02ff5640 1 -- 172.21.15.27:0/3837821250 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0a1004e070 con 0x55d000977a30 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a27ff5740 10 client.16905 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a27ff5740 10 client.16905 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a27ff5740 20 client.16905 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d0009ee9b0 0x55d0009f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a27ff5740 4 client.16905 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a27ff5740 1 -- 172.21.15.27:0/3837821250 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d0008d4f40 con 0x55d0009ee9b0 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a1effd640 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d0009ee9b0 0x55d0009f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a1effd640 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d0009ee9b0 0x55d0009f0dd0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f846a4516a8504dd server_cookie=879914ab07d5d110 in_seq=0 out_seq=0 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.675+0000 7f0a02ff5640 10 client.16905 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a02ff5640 1 -- 172.21.15.27:0/3837821250 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d0008d4f40 con 0x55d0009ee9b0 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a02ff5640 10 client.16905 handle_command_reply: tid=0 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 client.16905 shutdown 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 -- 172.21.15.27:0/3837821250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000977a30 msgr2=0x55d000971750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000977a30 0x55d000971750 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0a100ee800 tx=0x7f0a100099f0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 -- 172.21.15.27:0/3837821250 shutdown_connections 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d0009ee9b0 0x55d0009f0dd0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d000977a30 0x55d000971750 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d000968d10 0x55d000971210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a02ff5640 0 client.16905 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 --2- 172.21.15.27:0/3837821250 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d000968360 0x55d00097fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 -- 172.21.15.27:0/3837821250 >> 172.21.15.27:0/3837821250 conn(0x55d000964950 msgr2=0x55d000963af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 -- 172.21.15.27:0/3837821250 shutdown_connections 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 1 -- 172.21.15.27:0/3837821250 wait complete. 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a27ff5740 20 client.16905 trim_cache size 0 max 16384 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:31:26.661 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:31:26.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 -- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0a080a2d00 msgr2=0x7f0a080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0a080a2d00 0x7f0a080a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0a18005fd0 tx=0x7f0a1800b390 comp rx=0 tx=0).stop 2024-04-25T01:31:26.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 -- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a201066b0 msgr2=0x7f0a201a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a201066b0 0x7f0a201a3e70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0a10005cc0 tx=0x7f0a10032f40 comp rx=0 tx=0).stop 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 -- 172.21.15.27:0/2574377587 shutdown_connections 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f0a080a2d00 0x7f0a080a51c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a201066b0 0x7f0a201a3e70 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a20100fd0 0x7f0a20199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 --2- 172.21.15.27:0/2574377587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0a20100610 0x7f0a20198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.676+0000 7f0a252e1640 1 -- 172.21.15.27:0/2574377587 >> 172.21.15.27:0/2574377587 conn(0x7f0a20077740 msgr2=0x7f0a200ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.677+0000 7f0a252e1640 1 -- 172.21.15.27:0/2574377587 shutdown_connections 2024-04-25T01:31:26.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.677+0000 7f0a252e1640 1 -- 172.21.15.27:0/2574377587 wait complete. 2024-04-25T01:31:26.664 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:31:26.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 -- 172.21.15.27:0/2621493473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c100690 msgr2=0x7f995c100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 --2- 172.21.15.27:0/2621493473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c100690 0x7f995c100b10 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f994c009930 tx=0x7f994c02a420 comp rx=0 tx=0).stop 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 -- 172.21.15.27:0/2621493473 shutdown_connections 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 --2- 172.21.15.27:0/2621493473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f995c101050 0x7f995c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 --2- 172.21.15.27:0/2621493473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c100690 0x7f995c100b10 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 --2- 172.21.15.27:0/2621493473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f995c106650 0x7f995c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 -- 172.21.15.27:0/2621493473 >> 172.21.15.27:0/2621493473 conn(0x7f995c0777a0 msgr2=0x7f995c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 -- 172.21.15.27:0/2621493473 shutdown_connections 2024-04-25T01:31:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.790+0000 7f9962295640 1 -- 172.21.15.27:0/2621493473 wait complete. 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9962295640 1 Processor -- start 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9962295640 1 -- start start 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9962295640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f995c100690 0x7f995c199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9962295640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f995c101050 0x7f995c199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9962295640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c106650 0x7f995c1a4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9962295640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f995c110a60 con 0x7f995c106650 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9962295640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f995c1108e0 con 0x7f995c101050 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9962295640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f995c110be0 con 0x7f995c100690 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9960832640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c106650 0x7f995c1a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9960832640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c106650 0x7f995c1a4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55760/0 (socket says 172.21.15.27:55760) 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f9960832640 1 -- 172.21.15.27:0/1642201600 learned_addr learned my addr 172.21.15.27:0/1642201600 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.791+0000 7f995b7fe640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f995c101050 0x7f995c199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f995bfff640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f995c100690 0x7f995c199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f9960832640 1 -- 172.21.15.27:0/1642201600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f995c100690 msgr2=0x7f995c199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f9960832640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f995c100690 0x7f995c199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f9960832640 1 -- 172.21.15.27:0/1642201600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f995c101050 msgr2=0x7f995c199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f9960832640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f995c101050 0x7f995c199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f9960832640 1 -- 172.21.15.27:0/1642201600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995c1a45c0 con 0x7f995c106650 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f9960832640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c106650 0x7f995c1a4080 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f995400b9e0 tx=0x7f995400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f99597fa640 1 -- 172.21.15.27:0/1642201600 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954019070 con 0x7f995c106650 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f99597fa640 1 -- 172.21.15.27:0/1642201600 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9954014440 con 0x7f995c106650 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f995c1a4790 con 0x7f995c106650 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f99597fa640 1 -- 172.21.15.27:0/1642201600 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540075d0 con 0x7f995c106650 2024-04-25T01:31:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.792+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f995c1a4d00 con 0x7f995c106650 2024-04-25T01:31:26.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.793+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f995c0689b0 con 0x7f995c106650 2024-04-25T01:31:26.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.794+0000 7f99597fa640 1 -- 172.21.15.27:0/1642201600 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f995401c070 con 0x7f995c106650 2024-04-25T01:31:26.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.795+0000 7f99597fa640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f99480a2d80 0x7f99480a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:26.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.795+0000 7f995bfff640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f99480a2d80 0x7f99480a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:26.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.795+0000 7f99597fa640 1 -- 172.21.15.27:0/1642201600 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f99540ced60 con 0x7f995c106650 2024-04-25T01:31:26.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.795+0000 7f995bfff640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f99480a2d80 0x7f99480a5240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9944005fd0 tx=0x7f9944005850 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:26.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:26.799+0000 7f99597fa640 1 -- 172.21.15.27:0/1642201600 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f995409b660 con 0x7f995c106650 2024-04-25T01:31:26.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:26 smithi027 ceph-mon[29197]: pgmap v183: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.026+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f995c108bd0 con 0x7f995c106650 2024-04-25T01:31:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.027+0000 7f99597fa640 1 -- 172.21.15.27:0/1642201600 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7f995409af60 con 0x7f995c106650 2024-04-25T01:31:27.004 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:31:27.004 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:31:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f99480a2d80 msgr2=0x7f99480a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f99480a2d80 0x7f99480a5240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9944005fd0 tx=0x7f9944005850 comp rx=0 tx=0).stop 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c106650 msgr2=0x7f995c1a4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c106650 0x7f995c1a4080 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f995400b9e0 tx=0x7f995400bea0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 shutdown_connections 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f99480a2d80 0x7f99480a5240 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f995c106650 0x7f995c1a4080 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f995c101050 0x7f995c199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 --2- 172.21.15.27:0/1642201600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f995c100690 0x7f995c199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 >> 172.21.15.27:0/1642201600 conn(0x7f995c0777a0 msgr2=0x7f995c103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 shutdown_connections 2024-04-25T01:31:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.029+0000 7f9962295640 1 -- 172.21.15.27:0/1642201600 wait complete. 2024-04-25T01:31:27.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:31:27.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:26 smithi153 ceph-mon[32621]: pgmap v183: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:27.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:26 smithi139 ceph-mon[32760]: pgmap v183: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 -- 172.21.15.27:0/3445815567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd8107730 msgr2=0x7f7fd8107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3445815567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd8107730 0x7f7fd8107b10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009930 tx=0x7f7fc802a420 comp rx=0 tx=0).stop 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 -- 172.21.15.27:0/3445815567 shutdown_connections 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3445815567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fd806a780 0x7f7fd810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3445815567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd8069dc0 0x7f7fd806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3445815567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd8107730 0x7f7fd8107b10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 -- 172.21.15.27:0/3445815567 >> 172.21.15.27:0/3445815567 conn(0x7f7fd80776a0 msgr2=0x7f7fd8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 -- 172.21.15.27:0/3445815567 shutdown_connections 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.144+0000 7f7fddd61640 1 -- 172.21.15.27:0/3445815567 wait complete. 2024-04-25T01:31:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.145+0000 7f7fddd61640 1 Processor -- start 2024-04-25T01:31:27.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.145+0000 7f7fddd61640 1 -- start start 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.145+0000 7f7fddd61640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd8069dc0 0x7f7fd81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.145+0000 7f7fddd61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd806a780 0x7f7fd81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.145+0000 7f7fddd61640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fd8107730 0x7f7fd819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.145+0000 7f7fddd61640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd8111b40 con 0x7f7fd806a780 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.145+0000 7f7fddd61640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd81119c0 con 0x7f7fd8107730 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.145+0000 7f7fddd61640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd8111cc0 con 0x7f7fd8069dc0 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd806a780 0x7f7fd81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd806a780 0x7f7fd81985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55784/0 (socket says 172.21.15.27:55784) 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 -- 172.21.15.27:0/3565751081 learned_addr learned my addr 172.21.15.27:0/3565751081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fd8107730 0x7f7fd819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 -- 172.21.15.27:0/3565751081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd8069dc0 msgr2=0x7f7fd81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd8069dc0 0x7f7fd81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd8069dc0 0x7f7fd81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 -- 172.21.15.27:0/3565751081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fd8107730 msgr2=0x7f7fd819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fd8107730 0x7f7fd819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 -- 172.21.15.27:0/3565751081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd819d090 con 0x7f7fd806a780 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fd8107730 0x7f7fd819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd8069dc0 0x7f7fd81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd6ffd640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd806a780 0x7f7fd81985e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00e9f0 tx=0x7f7fcc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd4ff9640 1 -- 172.21.15.27:0/3565751081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc00cd80 con 0x7f7fd806a780 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd4ff9640 1 -- 172.21.15.27:0/3565751081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fcc005500 con 0x7f7fd806a780 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fddd61640 1 -- 172.21.15.27:0/3565751081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd819d380 con 0x7f7fd806a780 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fd4ff9640 1 -- 172.21.15.27:0/3565751081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc010690 con 0x7f7fd806a780 2024-04-25T01:31:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.146+0000 7f7fddd61640 1 -- 172.21.15.27:0/3565751081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd81a4c10 con 0x7f7fd806a780 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fd4ff9640 1 -- 172.21.15.27:0/3565751081 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7fcc0046c0 con 0x7f7fd806a780 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fe0a6c740 1 Processor -- start 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fe0a6c740 1 -- start start 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d70860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d7f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 0x55c581d81d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c581cd5670 con 0x55c581d70480 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c581c7d970 con 0x55c581d70da0 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.149+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c581cd1630 con 0x55c581d7f980 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d70860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d70860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55800/0 (socket says 172.21.15.27:55800) 2024-04-25T01:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 -- 172.21.15.27:0/4116481707 learned_addr learned my addr 172.21.15.27:0/4116481707 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd4ff9640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fc00a2d00 0x7f7fc00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd6ffd640 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 0x55c581d81d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d7f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fc00a2d00 0x7f7fc00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 -- 172.21.15.27:0/4116481707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 msgr2=0x55c581d81d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 0x55c581d81d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 -- 172.21.15.27:0/4116481707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 msgr2=0x55c581d7f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d7f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 -- 172.21.15.27:0/4116481707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c581ca1400 con 0x55c581d70480 2024-04-25T01:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd4ff9640 1 -- 172.21.15.27:0/3565751081 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7fcc0cfd20 con 0x7f7fd806a780 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d70860 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0014a10 tx=0x7f7fd0014ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea497694d0ee06e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fbdffb640 1 -- 172.21.15.27:0/4116481707 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0014250 con 0x55c581d70480 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fbdffb640 1 -- 172.21.15.27:0/4116481707 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd0023070 con 0x55c581d70480 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.150+0000 7f7fbdffb640 1 -- 172.21.15.27:0/4116481707 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd00139b0 con 0x55c581d70480 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/4116481707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 msgr2=0x55c581d70860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fc00a2d00 0x7f7fc00a51c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0022000 tx=0x7f7fd0020000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d70860 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0014a10 tx=0x7f7fd0014ed0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/4116481707 shutdown_connections 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 0x55c581d81d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d7f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/4116481707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d70860 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/4116481707 >> 172.21.15.27:0/4116481707 conn(0x55c581d6c8a0 msgr2=0x55c581d6cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/4116481707 shutdown_connections 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/4116481707 wait complete. 2024-04-25T01:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.151+0000 7f7fe0a6c740 1 -- start start 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fe0a6c740 1 Processor -- start 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d7f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d7c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 0x55c581d7cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c581d82740 con 0x55c581d70480 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c581c91050 con 0x55c581d70da0 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c581d828c0 con 0x55c581d7f980 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fe0a6c740 10 client.0 _ll_register_callbacks cb 0x55c5819e8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d7f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d7f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55812/0 (socket says 172.21.15.27:55812) 2024-04-25T01:31:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd77fe640 1 -- 172.21.15.27:0/3184402125 learned_addr learned my addr 172.21.15.27:0/3184402125 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd6ffd640 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d7c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd77fe640 1 -- 172.21.15.27:0/3184402125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 msgr2=0x55c581d7cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 0x55c581d7cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd77fe640 1 -- 172.21.15.27:0/3184402125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 msgr2=0x55c581d7c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d7c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.152+0000 7f7fd77fe640 1 -- 172.21.15.27:0/3184402125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c581c62dc0 con 0x55c581d70480 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d7f2b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009930 tx=0x7f7fc80317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cb93584f404f12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3184402125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8040070 con 0x55c581d70480 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3184402125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7fc8004240 con 0x55c581d70480 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3184402125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd0023070 con 0x55c581d70480 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3184402125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7fc8035040 con 0x55c581d70480 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 10 client.16923 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3184402125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c5819e93f0 con 0x55c581d70480 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3184402125 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd0023070 con 0x55c581d70480 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3184402125 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7fc80049c0 con 0x55c581d70480 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 10 client.16923 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 10 client.16923 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 20 client.16923 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c581d8fa10 0x55c581d91ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 4 client.16923 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.153+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3184402125 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c581ce2bb0 con 0x55c581d8fa10 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.154+0000 7f7fd6ffd640 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c581d8fa10 0x55c581d91ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.154+0000 7f7fd6ffd640 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c581d8fa10 0x55c581d91ea0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36bf517039fad77a server_cookie=511cb28a21a993b0 in_seq=0 out_seq=0 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.154+0000 7f7fb6ffd640 10 client.16923 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3184402125 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c581ce2bb0 con 0x55c581d8fa10 2024-04-25T01:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fb6ffd640 10 client.16923 handle_command_reply: tid=0 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 client.16923 shutdown 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3184402125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 msgr2=0x55c581d7f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d7f2b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009930 tx=0x7f7fc80317e0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3184402125 shutdown_connections 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c581d8fa10 0x55c581d91ea0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d7f980 0x55c581d7cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d7c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fb6ffd640 0 client.16923 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3184402125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d70480 0x55c581d7f2b0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3184402125 >> 172.21.15.27:0/3184402125 conn(0x55c581d6c8a0 msgr2=0x55c581d726d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.155+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3184402125 shutdown_connections 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.156+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3184402125 wait complete. 2024-04-25T01:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.156+0000 7f7fe0a6c740 20 client.16923 trim_cache size 0 max 16384 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fe0a6c740 1 Processor -- start 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fe0a6c740 1 -- start start 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70480 0x55c581d8b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d77af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d7ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd81af290 con 0x55c581d7f980 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd81acb60 con 0x55c581d70da0 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd81aa9b0 con 0x55c581d70480 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d7ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d7ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55818/0 (socket says 172.21.15.27:55818) 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fd7fff640 1 -- 172.21.15.27:0/746282629 learned_addr learned my addr 172.21.15.27:0/746282629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fd7fff640 1 -- 172.21.15.27:0/746282629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70480 msgr2=0x55c581d8b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/746282629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70480 0x55c581d8b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fd7fff640 1 -- 172.21.15.27:0/746282629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 msgr2=0x55c581d77af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/746282629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d77af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.169+0000 7f7fd7fff640 1 -- 172.21.15.27:0/746282629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c581d70bd0 con 0x55c581d7f980 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/746282629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d7ede0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0012760 tx=0x7f7fd0005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4c6a239e688f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/746282629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0012070 con 0x55c581d7f980 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/746282629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd0023040 con 0x55c581d7f980 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fbd7fa640 1 -- 172.21.15.27:0/746282629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0014050 con 0x55c581d7f980 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/746282629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 msgr2=0x55c581d7ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/746282629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d7ede0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0012760 tx=0x7f7fd0005f10 comp rx=0 tx=0).stop 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/746282629 shutdown_connections 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/746282629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d7ede0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/746282629 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70da0 0x55c581d77af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/746282629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70480 0x55c581d8b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/746282629 >> 172.21.15.27:0/746282629 conn(0x55c581d6c8a0 msgr2=0x55c581d773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/746282629 shutdown_connections 2024-04-25T01:31:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.170+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/746282629 wait complete. 2024-04-25T01:31:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 1 -- start start 2024-04-25T01:31:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 1 Processor -- start 2024-04-25T01:31:27.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70480 0x55c581d83680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70da0 0x55c581d83bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d84100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd81ab020 con 0x55c581d7f980 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd81ac490 con 0x55c581d70480 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd81ab6f0 con 0x55c581d70da0 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fe0a6c740 10 client.0 _ll_register_callbacks cb 0x7f7fc8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d84100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d84100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55824/0 (socket says 172.21.15.27:55824) 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fd7fff640 1 -- 172.21.15.27:0/3914628193 learned_addr learned my addr 172.21.15.27:0/3914628193 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.171+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70480 0x55c581d83680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fd6ffd640 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70da0 0x55c581d83bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fd7fff640 1 -- 172.21.15.27:0/3914628193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70da0 msgr2=0x55c581d83bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70da0 0x55c581d83bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fd7fff640 1 -- 172.21.15.27:0/3914628193 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70480 msgr2=0x55c581d83680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70480 0x55c581d83680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fd7fff640 1 -- 172.21.15.27:0/3914628193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c581ab37d0 con 0x55c581d7f980 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fd7fff640 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d84100 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0000f80 tx=0x7f7fd0014770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e661fe153d5bef85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3914628193 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd003a060 con 0x55c581d7f980 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3914628193 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7fd0013c10 con 0x55c581d7f980 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3914628193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd0023040 con 0x55c581d7f980 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3914628193 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7fd0041050 con 0x55c581d7f980 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fe0a6c740 10 client.16929 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3914628193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c581df6210 con 0x55c581d7f980 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3914628193 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd0023040 con 0x55c581d7f980 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.172+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3914628193 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7fd000f040 con 0x55c581d7f980 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fe0a6c740 10 client.16929 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fe0a6c740 10 client.16929 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:31:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fe0a6c740 20 client.16929 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c581df66d0 0x55c581df8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fe0a6c740 4 client.16929 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3914628193 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c581cdcf00 con 0x55c581df66d0 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c581df66d0 0x55c581df8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fb6ffd640 10 client.16929 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fd77fe640 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c581df66d0 0x55c581df8ac0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cab71dbccd296e69 server_cookie=e8a7531d5fe9dfcf in_seq=0 out_seq=0 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fb6ffd640 1 -- 172.21.15.27:0/3914628193 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c581cdcf00 con 0x55c581df66d0 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fb6ffd640 10 client.16929 handle_command_reply: tid=0 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.173+0000 7f7fe0a6c740 1 client.16929 shutdown 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3914628193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 msgr2=0x55c581d84100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d84100 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0000f80 tx=0x7f7fd0014770 comp rx=0 tx=0).stop 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3914628193 shutdown_connections 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c581df66d0 0x55c581df8ac0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c581d7f980 0x55c581d84100 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c581d70da0 0x55c581d83bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fb6ffd640 0 client.16929 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:31:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 --2- 172.21.15.27:0/3914628193 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c581d70480 0x55c581d83680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3914628193 >> 172.21.15.27:0/3914628193 conn(0x55c581d6c8a0 msgr2=0x55c581d71e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3914628193 shutdown_connections 2024-04-25T01:31:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 1 -- 172.21.15.27:0/3914628193 wait complete. 2024-04-25T01:31:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fe0a6c740 20 client.16929 trim_cache size 0 max 16384 2024-04-25T01:31:27.157 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:31:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 -- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fc00a2d00 msgr2=0x7f7fc00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fc00a2d00 0x7f7fc00a51c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0022000 tx=0x7f7fd0020000 comp rx=0 tx=0).stop 2024-04-25T01:31:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 -- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd806a780 msgr2=0x7f7fd81985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd806a780 0x7f7fd81985e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00e9f0 tx=0x7f7fcc00edf0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 -- 172.21.15.27:0/3565751081 shutdown_connections 2024-04-25T01:31:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7fd8107730 0x7f7fd819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7fc00a2d00 0x7f7fc00a51c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd806a780 0x7f7fd81985e0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 --2- 172.21.15.27:0/3565751081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fd8069dc0 0x7f7fd81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 -- 172.21.15.27:0/3565751081 >> 172.21.15.27:0/3565751081 conn(0x7f7fd80776a0 msgr2=0x7f7fd80fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 -- 172.21.15.27:0/3565751081 shutdown_connections 2024-04-25T01:31:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.174+0000 7f7fddd61640 1 -- 172.21.15.27:0/3565751081 wait complete. 2024-04-25T01:31:27.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:31:27.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.288+0000 7f5474b89640 1 -- 172.21.15.27:0/381985508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f547010bd00 msgr2=0x7f547010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.288+0000 7f5474b89640 1 --2- 172.21.15.27:0/381985508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f547010bd00 0x7f547010e1a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f5460009ca0 tx=0x7f546002a9e0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 -- 172.21.15.27:0/381985508 shutdown_connections 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 --2- 172.21.15.27:0/381985508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f547010bd00 0x7f547010e1a0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 --2- 172.21.15.27:0/381985508 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54701032e0 0x7f547010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 --2- 172.21.15.27:0/381985508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54701029c0 0x7f5470102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 -- 172.21.15.27:0/381985508 >> 172.21.15.27:0/381985508 conn(0x7f5470077740 msgr2=0x7f5470077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 -- 172.21.15.27:0/381985508 shutdown_connections 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 -- 172.21.15.27:0/381985508 wait complete. 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 Processor -- start 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.289+0000 7f5474b89640 1 -- start start 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f5474b89640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54701029c0 0x7f5470197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f5474b89640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54701032e0 0x7f5470198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f5474b89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f547010bd00 0x7f547019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f5474b89640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5470111b00 con 0x7f54701029c0 2024-04-25T01:31:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f5474b89640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5470111980 con 0x7f54701032e0 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f5474b89640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5470111c80 con 0x7f547010bd00 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54701029c0 0x7f5470197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54701029c0 0x7f5470197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55850/0 (socket says 172.21.15.27:55850) 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 -- 172.21.15.27:0/2219937447 learned_addr learned my addr 172.21.15.27:0/2219937447 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546dd9b640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54701032e0 0x7f5470198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546ed9d640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f547010bd00 0x7f547019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 -- 172.21.15.27:0/2219937447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f547010bd00 msgr2=0x7f547019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f547010bd00 0x7f547019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 -- 172.21.15.27:0/2219937447 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54701032e0 msgr2=0x7f5470198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54701032e0 0x7f5470198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 -- 172.21.15.27:0/2219937447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547019cf10 con 0x7f54701029c0 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546ed9d640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f547010bd00 0x7f547019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f546e59c640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54701029c0 0x7f5470197f20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f546800c920 tx=0x7f546800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f545f7fe640 1 -- 172.21.15.27:0/2219937447 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468007c70 con 0x7f54701029c0 2024-04-25T01:31:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f545f7fe640 1 -- 172.21.15.27:0/2219937447 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5468007e10 con 0x7f54701029c0 2024-04-25T01:31:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f5474b89640 1 -- 172.21.15.27:0/2219937447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547019d200 con 0x7f54701029c0 2024-04-25T01:31:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f545f7fe640 1 -- 172.21.15.27:0/2219937447 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468018890 con 0x7f54701029c0 2024-04-25T01:31:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.290+0000 7f5474b89640 1 -- 172.21.15.27:0/2219937447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54701a4a90 con 0x7f54701029c0 2024-04-25T01:31:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f545f7fe640 1 -- 172.21.15.27:0/2219937447 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5468018a70 con 0x7f54701029c0 2024-04-25T01:31:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f5477972740 1 Processor -- start 2024-04-25T01:31:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f5477972740 1 -- start start 2024-04-25T01:31:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f5477972740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde8c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f5477972740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8cea0 0x556fbde97030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f5477972740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde99960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f5477972740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556fbddf1670 con 0x556fbde8c580 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f5477972740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556fbdd99970 con 0x556fbde97570 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f5477972740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556fbdded630 con 0x556fbde8cea0 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f546ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde8c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f546ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde8c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55858/0 (socket says 172.21.15.27:55858) 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f546ed9d640 1 -- 172.21.15.27:0/1640035435 learned_addr learned my addr 172.21.15.27:0/1640035435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f545f7fe640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f54580a2d00 0x7f54580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.293+0000 7f546dd9b640 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde99960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546e59c640 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8cea0 0x556fbde97030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546ed9d640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f54580a2d00 0x7f54580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546ed9d640 1 -- 172.21.15.27:0/1640035435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8cea0 msgr2=0x556fbde97030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546ed9d640 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8cea0 0x556fbde97030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546ed9d640 1 -- 172.21.15.27:0/1640035435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 msgr2=0x556fbde99960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546ed9d640 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde99960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546ed9d640 1 -- 172.21.15.27:0/1640035435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fbddbd400 con 0x556fbde8c580 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546e59c640 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8cea0 0x556fbde97030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546dd9b640 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde99960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f545f7fe640 1 -- 172.21.15.27:0/2219937447 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f54680cf930 con 0x7f54701029c0 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f546ed9d640 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde8c960 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f546002ef80 tx=0x7f546002efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302b1e287ce625ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f545cff9640 1 -- 172.21.15.27:0/1640035435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460041050 con 0x556fbde8c580 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f545cff9640 1 -- 172.21.15.27:0/1640035435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5460042040 con 0x556fbde8c580 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.294+0000 7f545cff9640 1 -- 172.21.15.27:0/1640035435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546004b020 con 0x556fbde8c580 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 -- 172.21.15.27:0/1640035435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 msgr2=0x556fbde8c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde8c960 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f546002ef80 tx=0x7f546002efb0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f546ed9d640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f54580a2d00 0x7f54580a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f546004c000 tx=0x7f546004a000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 -- 172.21.15.27:0/1640035435 shutdown_connections 2024-04-25T01:31:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde99960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8cea0 0x556fbde97030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 --2- 172.21.15.27:0/1640035435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde8c960 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 -- 172.21.15.27:0/1640035435 >> 172.21.15.27:0/1640035435 conn(0x556fbde888a0 msgr2=0x556fbde88c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 -- 172.21.15.27:0/1640035435 shutdown_connections 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 -- 172.21.15.27:0/1640035435 wait complete. 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 -- start start 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.295+0000 7f5477972740 1 Processor -- start 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f5477972740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8c580 0x556fbde96f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f5477972740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8cea0 0x556fbde941f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f5477972740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde94770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f5477972740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556fbde9a360 con 0x556fbde8cea0 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f5477972740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556fbddad050 con 0x556fbde97570 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f5477972740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556fbde9a4e0 con 0x556fbde8c580 2024-04-25T01:31:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f5477972740 10 client.0 _ll_register_callbacks cb 0x556fbdb04be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8cea0 0x556fbde941f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8cea0 0x556fbde941f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55866/0 (socket says 172.21.15.27:55866) 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 -- 172.21.15.27:0/3511872038 learned_addr learned my addr 172.21.15.27:0/3511872038 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546ed9d640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde94770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546e59c640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8c580 0x556fbde96f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 -- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8c580 msgr2=0x556fbde96f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8c580 0x556fbde96f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 -- 172.21.15.27:0/3511872038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 msgr2=0x556fbde94770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde94770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 -- 172.21.15.27:0/3511872038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fbdd7edc0 con 0x556fbde8cea0 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546ed9d640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde94770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546e59c640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8c580 0x556fbde96f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f546dd9b640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8cea0 0x556fbde941f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f546400b9f0 tx=0x7f546400beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6659c877078a88a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.296+0000 7f5455ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5455ffb640 1 -- 172.21.15.27:0/3511872038 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464003e20 con 0x556fbde8cea0 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 1 -- 172.21.15.27:0/3511872038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5460042040 con 0x556fbde8cea0 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5455ffb640 1 -- 172.21.15.27:0/3511872038 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5464011440 con 0x556fbde8cea0 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5455ffb640 1 -- 172.21.15.27:0/3511872038 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5464011640 con 0x556fbde8cea0 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 10 client.16941 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 1 -- 172.21.15.27:0/3511872038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556fbdb053f0 con 0x556fbde8cea0 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5455ffb640 1 -- 172.21.15.27:0/3511872038 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5460042040 con 0x556fbde8cea0 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5455ffb640 1 -- 172.21.15.27:0/3511872038 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f546401baa0 con 0x556fbde8cea0 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 10 client.16941 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 10 client.16941 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 20 client.16941 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x556fbdea7740 0x556fbdea9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 4 client.16941 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.297+0000 7f5477972740 1 -- 172.21.15.27:0/3511872038 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556fbddfebb0 con 0x556fbdea7740 2024-04-25T01:31:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.298+0000 7f546e59c640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x556fbdea7740 0x556fbdea9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.298+0000 7f546e59c640 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x556fbdea7740 0x556fbdea9bc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=52ff22df0c998d31 server_cookie=c7df2068f4197d8 in_seq=0 out_seq=0 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.298+0000 7f5455ffb640 10 client.16941 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5455ffb640 1 -- 172.21.15.27:0/3511872038 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556fbddfebb0 con 0x556fbdea7740 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5455ffb640 10 client.16941 handle_command_reply: tid=0 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 client.16941 shutdown 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 -- 172.21.15.27:0/3511872038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8cea0 msgr2=0x556fbde941f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8cea0 0x556fbde941f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f546400b9f0 tx=0x7f546400beb0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 -- 172.21.15.27:0/3511872038 shutdown_connections 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x556fbdea7740 0x556fbdea9bc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde97570 0x556fbde94770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8cea0 0x556fbde941f0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5455ffb640 0 client.16941 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 --2- 172.21.15.27:0/3511872038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde8c580 0x556fbde96f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 -- 172.21.15.27:0/3511872038 >> 172.21.15.27:0/3511872038 conn(0x556fbde888a0 msgr2=0x556fbde87b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 -- 172.21.15.27:0/3511872038 shutdown_connections 2024-04-25T01:31:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 1 -- 172.21.15.27:0/3511872038 wait complete. 2024-04-25T01:31:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.299+0000 7f5477972740 20 client.16941 trim_cache size 0 max 16384 2024-04-25T01:31:27.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f5477972740 1 Processor -- start 2024-04-25T01:31:27.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f5477972740 1 -- start start 2024-04-25T01:31:27.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f5477972740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbdea3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f5477972740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbdeaf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f5477972740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 0x556fbdeaf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f5477972740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f54701b1410 con 0x556fbde8c580 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f5477972740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f54701aece0 con 0x556fbde8cea0 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f5477972740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54701acb30 con 0x556fbde888a0 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.312+0000 7f546dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbdeaf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbdeaf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55870/0 (socket says 172.21.15.27:55870) 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546dd9b640 1 -- 172.21.15.27:0/4256424713 learned_addr learned my addr 172.21.15.27:0/4256424713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546e59c640 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbdea3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546ed9d640 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 0x556fbdeaf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546dd9b640 1 -- 172.21.15.27:0/4256424713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 msgr2=0x556fbdea3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546dd9b640 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbdea3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546dd9b640 1 -- 172.21.15.27:0/4256424713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 msgr2=0x556fbdeaf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546dd9b640 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 0x556fbdeaf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546dd9b640 1 -- 172.21.15.27:0/4256424713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fbdbcf7d0 con 0x556fbde8c580 2024-04-25T01:31:27.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546e59c640 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbdea3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f546dd9b640 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbdeaf490 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f546401be90 tx=0x7f5464003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1e34eb69d09197 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f5457fff640 1 -- 172.21.15.27:0/4256424713 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464010560 con 0x556fbde8c580 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f5457fff640 1 -- 172.21.15.27:0/4256424713 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f546402e070 con 0x556fbde8c580 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.313+0000 7f5457fff640 1 -- 172.21.15.27:0/4256424713 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464011b50 con 0x556fbde8c580 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- 172.21.15.27:0/4256424713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 msgr2=0x556fbdeaf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbdeaf490 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f546401be90 tx=0x7f5464003ed0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- 172.21.15.27:0/4256424713 shutdown_connections 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 0x556fbdeaf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbdeaf490 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 --2- 172.21.15.27:0/4256424713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbdea3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- 172.21.15.27:0/4256424713 >> 172.21.15.27:0/4256424713 conn(0x556fbde8fc00 msgr2=0x556fbde93240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- 172.21.15.27:0/4256424713 shutdown_connections 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- 172.21.15.27:0/4256424713 wait complete. 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- start start 2024-04-25T01:31:27.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 Processor -- start 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbde9fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde90d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 0x556fbde91490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f54701ad1a0 con 0x556fbde8c580 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f54701ae610 con 0x556fbde8cea0 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54701ad870 con 0x556fbde888a0 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f5477972740 10 client.0 _ll_register_callbacks cb 0x7f5464004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f546dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde90d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f546dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde90d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55886/0 (socket says 172.21.15.27:55886) 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.314+0000 7f546dd9b640 1 -- 172.21.15.27:0/759368021 learned_addr learned my addr 172.21.15.27:0/759368021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f546e59c640 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbde9fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f546dd9b640 1 -- 172.21.15.27:0/759368021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 msgr2=0x556fbde9fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f546dd9b640 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbde9fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f546dd9b640 1 -- 172.21.15.27:0/759368021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 msgr2=0x556fbde91490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f546dd9b640 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 0x556fbde91490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f546dd9b640 1 -- 172.21.15.27:0/759368021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fbdf0cd00 con 0x556fbde8c580 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f546dd9b640 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde90d50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f546401be10 tx=0x7f5464010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268ec59934631413 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f5455ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f5455ffb640 1 -- 172.21.15.27:0/759368021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464010b20 con 0x556fbde8c580 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f5455ffb640 1 -- 172.21.15.27:0/759368021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5464010cc0 con 0x556fbde8c580 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f5477972740 1 -- 172.21.15.27:0/759368021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f546402e070 con 0x556fbde8c580 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f5455ffb640 1 -- 172.21.15.27:0/759368021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5464011850 con 0x556fbde8c580 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f5477972740 10 client.16947 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f5477972740 1 -- 172.21.15.27:0/759368021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556fbdf0de60 con 0x556fbde8c580 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.315+0000 7f5455ffb640 1 -- 172.21.15.27:0/759368021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f546402e070 con 0x556fbde8c580 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5455ffb640 1 -- 172.21.15.27:0/759368021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5464029480 con 0x556fbde8c580 2024-04-25T01:31:27.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5477972740 10 client.16947 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5477972740 10 client.16947 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5477972740 20 client.16947 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5477972740 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x556fbde97570 0x556fbdf103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5477972740 4 client.16947 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5477972740 1 -- 172.21.15.27:0/759368021 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556fbddf8f00 con 0x556fbde97570 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f546e59c640 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x556fbde97570 0x556fbdf103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5455ffb640 10 client.16947 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f546e59c640 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x556fbde97570 0x556fbdf103a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a791b361ff12b40 server_cookie=61d03e5fd61e92ae in_seq=0 out_seq=0 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5455ffb640 1 -- 172.21.15.27:0/759368021 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556fbddf8f00 con 0x556fbde97570 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5455ffb640 10 client.16947 handle_command_reply: tid=0 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.316+0000 7f5477972740 1 client.16947 shutdown 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 -- 172.21.15.27:0/759368021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 msgr2=0x556fbde90d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde90d50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f546401be10 tx=0x7f5464010740 comp rx=0 tx=0).stop 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 -- 172.21.15.27:0/759368021 shutdown_connections 2024-04-25T01:31:27.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x556fbde97570 0x556fbdf103a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556fbde8cea0 0x556fbde91490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fbde8c580 0x556fbde90d50 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 --2- 172.21.15.27:0/759368021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556fbde888a0 0x556fbde9fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5455ffb640 0 client.16947 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 -- 172.21.15.27:0/759368021 >> 172.21.15.27:0/759368021 conn(0x556fbde8fc00 msgr2=0x556fbde872b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 -- 172.21.15.27:0/759368021 shutdown_connections 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 1 -- 172.21.15.27:0/759368021 wait complete. 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5477972740 20 client.16947 trim_cache size 0 max 16384 2024-04-25T01:31:27.300 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:31:27.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 -- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f54580a2d00 msgr2=0x7f54580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f54580a2d00 0x7f54580a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f546004c000 tx=0x7f546004a000 comp rx=0 tx=0).stop 2024-04-25T01:31:27.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 -- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54701029c0 msgr2=0x7f5470197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54701029c0 0x7f5470197f20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f546800c920 tx=0x7f546800cde0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 -- 172.21.15.27:0/2219937447 shutdown_connections 2024-04-25T01:31:27.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f547010bd00 0x7f547019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f54701032e0 0x7f5470198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f54580a2d00 0x7f54580a51c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 --2- 172.21.15.27:0/2219937447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54701029c0 0x7f5470197f20 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 -- 172.21.15.27:0/2219937447 >> 172.21.15.27:0/2219937447 conn(0x7f5470077740 msgr2=0x7f54700fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 -- 172.21.15.27:0/2219937447 shutdown_connections 2024-04-25T01:31:27.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.317+0000 7f5474b89640 1 -- 172.21.15.27:0/2219937447 wait complete. 2024-04-25T01:31:27.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:31:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- 172.21.15.27:0/934816229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 msgr2=0x7fe638106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/934816229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 0x7fe638106a90 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe630009930 tx=0x7fe63002a420 comp rx=0 tx=0).stop 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- 172.21.15.27:0/934816229 shutdown_connections 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/934816229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe638100fd0 0x7fe63810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/934816229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe638100610 0x7fe638100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/934816229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 0x7fe638106a90 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- 172.21.15.27:0/934816229 >> 172.21.15.27:0/934816229 conn(0x7fe638077740 msgr2=0x7fe638077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- 172.21.15.27:0/934816229 shutdown_connections 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- 172.21.15.27:0/934816229 wait complete. 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 Processor -- start 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- start start 2024-04-25T01:31:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe638100610 0x7fe6381022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe638100fd0 0x7fe6381027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 0x7fe638102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe638110ab0 con 0x7fe6381066b0 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe638110930 con 0x7fe638100fd0 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.432+0000 7fe63f6f7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe638110c30 con 0x7fe638100610 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 0x7fe638102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 0x7fe638102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55894/0 (socket says 172.21.15.27:55894) 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 -- 172.21.15.27:0/752988828 learned_addr learned my addr 172.21.15.27:0/752988828 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63d493640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe638100610 0x7fe6381022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63cc92640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe638100fd0 0x7fe6381027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 -- 172.21.15.27:0/752988828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe638100610 msgr2=0x7fe6381022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe638100610 0x7fe6381022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 -- 172.21.15.27:0/752988828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe638100fd0 msgr2=0x7fe6381027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe638100fd0 0x7fe6381027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 -- 172.21.15.27:0/752988828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6381a6620 con 0x7fe6381066b0 2024-04-25T01:31:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63cc92640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe638100fd0 0x7fe6381027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63dc94640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 0x7fe638102d30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe62800b970 tx=0x7fe62800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe6367fc640 1 -- 172.21.15.27:0/752988828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628019070 con 0x7fe6381066b0 2024-04-25T01:31:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63f6f7640 1 -- 172.21.15.27:0/752988828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6381a6910 con 0x7fe6381066b0 2024-04-25T01:31:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe6367fc640 1 -- 172.21.15.27:0/752988828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62800cb40 con 0x7fe6381066b0 2024-04-25T01:31:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe6367fc640 1 -- 172.21.15.27:0/752988828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628007650 con 0x7fe6381066b0 2024-04-25T01:31:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.433+0000 7fe63f6f7640 1 -- 172.21.15.27:0/752988828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6381a6e60 con 0x7fe6381066b0 2024-04-25T01:31:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6367fc640 1 -- 172.21.15.27:0/752988828 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe62801c070 con 0x7fe6381066b0 2024-04-25T01:31:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6423fd740 1 Processor -- start 2024-04-25T01:31:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6423fd740 1 -- start start 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a349960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a354030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a356960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6423fd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56489a2ae670 con 0x56489a354570 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6423fd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56489a256970 con 0x56489a349ea0 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6423fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56489a2aa630 con 0x56489a349580 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe6367fc640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6200a2d00 0x7fe6200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe63cc92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a356960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe63cc92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a356960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55900/0 (socket says 172.21.15.27:55900) 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe63cc92640 1 -- 172.21.15.27:0/591840868 learned_addr learned my addr 172.21.15.27:0/591840868 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.436+0000 7fe63dc94640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6200a2d00 0x7fe6200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe6367fc640 1 -- 172.21.15.27:0/752988828 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe6280cec30 con 0x7fe6381066b0 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63cc92640 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a349960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63d493640 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a354030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63cc92640 1 -- 172.21.15.27:0/591840868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 msgr2=0x56489a349960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63cc92640 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a349960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63cc92640 1 -- 172.21.15.27:0/591840868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 msgr2=0x56489a354030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63cc92640 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a354030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63cc92640 1 -- 172.21.15.27:0/591840868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489a27a400 con 0x56489a354570 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63cc92640 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a356960 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00adf0 tx=0x7fe62c00a520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baedc3bdb3eaafb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe63dc94640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6200a2d00 0x7fe6200a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe6280cc1b0 tx=0x7fe628004b50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe61f7fe640 1 -- 172.21.15.27:0/591840868 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c01c070 con 0x56489a354570 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe61f7fe640 1 -- 172.21.15.27:0/591840868 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62c00fdf0 con 0x56489a354570 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.437+0000 7fe61f7fe640 1 -- 172.21.15.27:0/591840868 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c014730 con 0x56489a354570 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 -- 172.21.15.27:0/591840868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 msgr2=0x56489a356960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a356960 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00adf0 tx=0x7fe62c00a520 comp rx=0 tx=0).stop 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 -- 172.21.15.27:0/591840868 shutdown_connections 2024-04-25T01:31:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a356960 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a354030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 --2- 172.21.15.27:0/591840868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a349960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 -- 172.21.15.27:0/591840868 >> 172.21.15.27:0/591840868 conn(0x56489a3458a0 msgr2=0x56489a345c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 -- 172.21.15.27:0/591840868 shutdown_connections 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 -- 172.21.15.27:0/591840868 wait complete. 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 -- start start 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.438+0000 7fe6423fd740 1 Processor -- start 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349580 0x56489a353ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349ea0 0x56489a3511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a351720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe6423fd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56489a357360 con 0x56489a354570 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe6423fd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56489a26a050 con 0x56489a349580 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe6423fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56489a3574e0 con 0x56489a349ea0 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe6423fd740 10 client.0 _ll_register_callbacks cb 0x564899fc1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a351720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a351720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55906/0 (socket says 172.21.15.27:55906) 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 -- 172.21.15.27:0/2604555459 learned_addr learned my addr 172.21.15.27:0/2604555459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349580 0x56489a353ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63d493640 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349ea0 0x56489a3511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 -- 172.21.15.27:0/2604555459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349ea0 msgr2=0x56489a3511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349ea0 0x56489a3511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 -- 172.21.15.27:0/2604555459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349580 msgr2=0x56489a353ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349580 0x56489a353ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 -- 172.21.15.27:0/2604555459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56489a23bdc0 con 0x56489a354570 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe63cc92640 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a351720 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00f1b0 tx=0x7fe62c006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918fed8fba2dbc7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe61cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe61cff9640 1 -- 172.21.15.27:0/2604555459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c02b060 con 0x56489a354570 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe6423fd740 1 -- 172.21.15.27:0/2604555459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe62c00fdf0 con 0x56489a354570 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.439+0000 7fe61cff9640 1 -- 172.21.15.27:0/2604555459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe62c014d60 con 0x56489a354570 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe61cff9640 1 -- 172.21.15.27:0/2604555459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe62c01c070 con 0x56489a354570 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe6423fd740 10 client.16959 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe6423fd740 1 -- 172.21.15.27:0/2604555459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564899fc23f0 con 0x56489a354570 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe61cff9640 1 -- 172.21.15.27:0/2604555459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62c00fdf0 con 0x56489a354570 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe61cff9640 1 -- 172.21.15.27:0/2604555459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe62c015040 con 0x56489a354570 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe6423fd740 10 client.16959 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:27.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe6423fd740 10 client.16959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe6423fd740 20 client.16959 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe6423fd740 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56489a3644b0 0x56489a3668a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe6423fd740 4 client.16959 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe6423fd740 1 -- 172.21.15.27:0/2604555459 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56489a2bbbb0 con 0x56489a3644b0 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.440+0000 7fe63d493640 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56489a3644b0 0x56489a3668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.441+0000 7fe63d493640 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56489a3644b0 0x56489a3668a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ce8e1e186e80efd server_cookie=69daa961fde3d127 in_seq=0 out_seq=0 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.441+0000 7fe61cff9640 10 client.16959 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe61cff9640 1 -- 172.21.15.27:0/2604555459 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56489a2bbbb0 con 0x56489a3644b0 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe61cff9640 10 client.16959 handle_command_reply: tid=0 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 client.16959 shutdown 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 -- 172.21.15.27:0/2604555459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 msgr2=0x56489a351720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a351720 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00f1b0 tx=0x7fe62c006f90 comp rx=0 tx=0).stop 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 -- 172.21.15.27:0/2604555459 shutdown_connections 2024-04-25T01:31:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56489a3644b0 0x56489a3668a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a351720 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349ea0 0x56489a3511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe61cff9640 0 client.16959 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 --2- 172.21.15.27:0/2604555459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349580 0x56489a353ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 -- 172.21.15.27:0/2604555459 >> 172.21.15.27:0/2604555459 conn(0x56489a3458a0 msgr2=0x56489a34b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 -- 172.21.15.27:0/2604555459 shutdown_connections 2024-04-25T01:31:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 1 -- 172.21.15.27:0/2604555459 wait complete. 2024-04-25T01:31:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.442+0000 7fe6423fd740 20 client.16959 trim_cache size 0 max 16384 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.455+0000 7fe6423fd740 1 Processor -- start 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.455+0000 7fe6423fd740 1 -- start start 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.455+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a3698d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.455+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a369e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.455+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a34e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.455+0000 7fe6423fd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381b3860 con 0x56489a354570 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.455+0000 7fe6423fd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381b1130 con 0x56489a349ea0 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.455+0000 7fe6423fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381aef80 con 0x56489a349580 2024-04-25T01:31:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a34e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a34e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55912/0 (socket says 172.21.15.27:55912) 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 -- 172.21.15.27:0/513024670 learned_addr learned my addr 172.21.15.27:0/513024670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 -- 172.21.15.27:0/513024670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 msgr2=0x56489a3698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 --2- 172.21.15.27:0/513024670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a3698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 -- 172.21.15.27:0/513024670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 msgr2=0x56489a369e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 --2- 172.21.15.27:0/513024670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a369e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 -- 172.21.15.27:0/513024670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489a08c7d0 con 0x56489a354570 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe63cc92640 1 --2- 172.21.15.27:0/513024670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a34e420 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00a440 tx=0x7fe62c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c157dbf1d8fb651 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe61effd640 1 -- 172.21.15.27:0/513024670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c02b040 con 0x56489a354570 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe61effd640 1 -- 172.21.15.27:0/513024670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62c01d040 con 0x56489a354570 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.456+0000 7fe61effd640 1 -- 172.21.15.27:0/513024670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c017c70 con 0x56489a354570 2024-04-25T01:31:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 -- 172.21.15.27:0/513024670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 msgr2=0x56489a34e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 --2- 172.21.15.27:0/513024670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a34e420 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00a440 tx=0x7fe62c0047d0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 -- 172.21.15.27:0/513024670 shutdown_connections 2024-04-25T01:31:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 --2- 172.21.15.27:0/513024670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a34e420 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 --2- 172.21.15.27:0/513024670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a369e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 --2- 172.21.15.27:0/513024670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a3698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 -- 172.21.15.27:0/513024670 >> 172.21.15.27:0/513024670 conn(0x56489a3458a0 msgr2=0x56489a3501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 -- 172.21.15.27:0/513024670 shutdown_connections 2024-04-25T01:31:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 -- 172.21.15.27:0/513024670 wait complete. 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 -- start start 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 Processor -- start 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.457+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a3c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a3c30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a368d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381af5f0 con 0x56489a354570 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381b0a60 con 0x56489a349ea0 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381afcc0 con 0x56489a349580 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 10 client.0 _ll_register_callbacks cb 0x7fe62c01a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a368d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a368d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55928/0 (socket says 172.21.15.27:55928) 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 -- 172.21.15.27:0/1578981952 learned_addr learned my addr 172.21.15.27:0/1578981952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a3c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 -- 172.21.15.27:0/1578981952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 msgr2=0x56489a3c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a3c2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 -- 172.21.15.27:0/1578981952 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 msgr2=0x56489a3c30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a3c30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 -- 172.21.15.27:0/1578981952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56489a369470 con 0x56489a354570 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe63cc92640 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a368d80 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe62c008040 tx=0x7fe62c006ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3685f282cdd1de18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe61cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe61cff9640 1 -- 172.21.15.27:0/1578981952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c00fa40 con 0x56489a354570 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 1 -- 172.21.15.27:0/1578981952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe62c01d040 con 0x56489a354570 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe61cff9640 1 -- 172.21.15.27:0/1578981952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe62c006170 con 0x56489a354570 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe61cff9640 1 -- 172.21.15.27:0/1578981952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe62c046080 con 0x56489a354570 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 10 client.16965 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.458+0000 7fe6423fd740 1 -- 172.21.15.27:0/1578981952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56489a3ca320 con 0x56489a354570 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe61cff9640 1 -- 172.21.15.27:0/1578981952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62c01d040 con 0x56489a354570 2024-04-25T01:31:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe61cff9640 1 -- 172.21.15.27:0/1578981952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe62c044470 con 0x56489a354570 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe6423fd740 10 client.16965 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe6423fd740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe6423fd740 20 client.16965 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe6423fd740 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56489a3ca800 0x56489a3ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe6423fd740 4 client.16965 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe6423fd740 1 -- 172.21.15.27:0/1578981952 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56489a2b5f00 con 0x56489a3ca800 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe63d493640 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56489a3ca800 0x56489a3ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe63d493640 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56489a3ca800 0x56489a3ccc20 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c79175f21f6a4b26 server_cookie=bafc80c642e599df in_seq=0 out_seq=0 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.459+0000 7fe61cff9640 10 client.16965 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe61cff9640 1 -- 172.21.15.27:0/1578981952 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56489a2b5f00 con 0x56489a3ca800 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe61cff9640 10 client.16965 handle_command_reply: tid=0 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 client.16965 shutdown 2024-04-25T01:31:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 -- 172.21.15.27:0/1578981952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 msgr2=0x56489a368d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a368d80 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe62c008040 tx=0x7fe62c006ad0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 -- 172.21.15.27:0/1578981952 shutdown_connections 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56489a3ca800 0x56489a3ccc20 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56489a354570 0x56489a368d80 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56489a349ea0 0x56489a3c30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe61cff9640 0 client.16965 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 --2- 172.21.15.27:0/1578981952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56489a349580 0x56489a3c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 -- 172.21.15.27:0/1578981952 >> 172.21.15.27:0/1578981952 conn(0x56489a3458a0 msgr2=0x56489a3442b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 -- 172.21.15.27:0/1578981952 shutdown_connections 2024-04-25T01:31:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 1 -- 172.21.15.27:0/1578981952 wait complete. 2024-04-25T01:31:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe6423fd740 20 client.16965 trim_cache size 0 max 16384 2024-04-25T01:31:27.445 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:31:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe63f6f7640 1 -- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6200a2d00 msgr2=0x7fe6200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6200a2d00 0x7fe6200a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe6280cc1b0 tx=0x7fe628004b50 comp rx=0 tx=0).stop 2024-04-25T01:31:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe63f6f7640 1 -- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 msgr2=0x7fe638102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 0x7fe638102d30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe62800b970 tx=0x7fe62800be30 comp rx=0 tx=0).stop 2024-04-25T01:31:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe63f6f7640 1 -- 172.21.15.27:0/752988828 shutdown_connections 2024-04-25T01:31:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe6200a2d00 0x7fe6200a51c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.460+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6381066b0 0x7fe638102d30 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.461+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe638100fd0 0x7fe6381027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.461+0000 7fe63f6f7640 1 --2- 172.21.15.27:0/752988828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe638100610 0x7fe6381022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.461+0000 7fe63f6f7640 1 -- 172.21.15.27:0/752988828 >> 172.21.15.27:0/752988828 conn(0x7fe638077740 msgr2=0x7fe6380ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.461+0000 7fe63f6f7640 1 -- 172.21.15.27:0/752988828 shutdown_connections 2024-04-25T01:31:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:27.461+0000 7fe63f6f7640 1 -- 172.21.15.27:0/752988828 wait complete. 2024-04-25T01:31:27.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:27 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1642201600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:31:28.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:27 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1642201600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:31:28.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:27 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1642201600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:31:28.449 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:31:28.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:31:28.450 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:31:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 -- 172.21.15.27:0/1390835264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 msgr2=0x7f8528106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 --2- 172.21.15.27:0/1390835264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 0x7f8528106a90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f8524009930 tx=0x7f852402a420 comp rx=0 tx=0).stop 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 -- 172.21.15.27:0/1390835264 shutdown_connections 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 --2- 172.21.15.27:0/1390835264 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8528100fd0 0x7f852810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 --2- 172.21.15.27:0/1390835264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8528100610 0x7f8528100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 --2- 172.21.15.27:0/1390835264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 0x7f8528106a90 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 -- 172.21.15.27:0/1390835264 >> 172.21.15.27:0/1390835264 conn(0x7f8528077740 msgr2=0x7f8528077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 -- 172.21.15.27:0/1390835264 shutdown_connections 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.593+0000 7f85303b0640 1 -- 172.21.15.27:0/1390835264 wait complete. 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f85303b0640 1 Processor -- start 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f85303b0640 1 -- start start 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f85303b0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8528100610 0x7f8528198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f85303b0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8528100fd0 0x7f85281985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f85303b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 0x7f852819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f85303b0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8528110a30 con 0x7f85281066b0 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f85303b0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f85281108b0 con 0x7f8528100610 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f85303b0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8528110bb0 con 0x7f8528100fd0 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e94d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 0x7f852819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e94d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 0x7f852819c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52072/0 (socket says 172.21.15.27:52072) 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e94d640 1 -- 172.21.15.27:0/566341813 learned_addr learned my addr 172.21.15.27:0/566341813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e14c640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8528100610 0x7f8528198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852d94b640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8528100fd0 0x7f85281985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e94d640 1 -- 172.21.15.27:0/566341813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8528100fd0 msgr2=0x7f85281985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e94d640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8528100fd0 0x7f85281985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e94d640 1 -- 172.21.15.27:0/566341813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8528100610 msgr2=0x7f8528198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e94d640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8528100610 0x7f8528198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.594+0000 7f852e94d640 1 -- 172.21.15.27:0/566341813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852819d010 con 0x7f85281066b0 2024-04-25T01:31:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.595+0000 7f852e94d640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 0x7f852819c8e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f852000cce0 tx=0x7f852000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.595+0000 7f851f7fe640 1 -- 172.21.15.27:0/566341813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852000f450 con 0x7f85281066b0 2024-04-25T01:31:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.595+0000 7f851f7fe640 1 -- 172.21.15.27:0/566341813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f852000fad0 con 0x7f85281066b0 2024-04-25T01:31:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.595+0000 7f85303b0640 1 -- 172.21.15.27:0/566341813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852819d300 con 0x7f85281066b0 2024-04-25T01:31:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.595+0000 7f851f7fe640 1 -- 172.21.15.27:0/566341813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520018b60 con 0x7f85281066b0 2024-04-25T01:31:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.595+0000 7f85303b0640 1 -- 172.21.15.27:0/566341813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85281a4b90 con 0x7f85281066b0 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.597+0000 7f851f7fe640 1 -- 172.21.15.27:0/566341813 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f852000fc80 con 0x7f85281066b0 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.597+0000 7f853319d740 1 Processor -- start 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f853319d740 1 -- start start 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f853319d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffe5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f853319d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cffe6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f853319d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f853319d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5586cff4a2c0 con 0x5586cfff1010 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f853319d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5586cfef2910 con 0x5586cffe7980 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f853319d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5586cff46280 con 0x5586cffe6290 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f851f7fe640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d80 0x7f85140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f852d94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f852d94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52084/0 (socket says 172.21.15.27:52084) 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f852d94b640 1 -- 172.21.15.27:0/4081493807 learned_addr learned my addr 172.21.15.27:0/4081493807 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f852e14c640 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cffe6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f852e94d640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d80 0x7f85140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f852d94b640 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffe5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.598+0000 7f851f7fe640 1 -- 172.21.15.27:0/566341813 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f85200cfc10 con 0x7f85281066b0 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f852e14c640 1 -- 172.21.15.27:0/4081493807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 msgr2=0x5586cffe5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f852e14c640 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffe5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f852e14c640 1 -- 172.21.15.27:0/4081493807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 msgr2=0x5586cfff3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f852e14c640 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f852e14c640 1 -- 172.21.15.27:0/4081493807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586cff03fb0 con 0x5586cffe6290 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f852e94d640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d80 0x7f85140a5240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8520098200 tx=0x7f852006dd60 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f852e14c640 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cffe6710 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8524009230 tx=0x7f85240316e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fece93eb5d6c1c14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f851cbfe640 1 -- 172.21.15.27:0/4081493807 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524040070 con 0x5586cffe6290 2024-04-25T01:31:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f851cbfe640 1 -- 172.21.15.27:0/4081493807 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8524005500 con 0x5586cffe6290 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.599+0000 7f851cbfe640 1 -- 172.21.15.27:0/4081493807 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524035040 con 0x5586cffe6290 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 -- 172.21.15.27:0/4081493807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 msgr2=0x5586cffe6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cffe6710 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8524009230 tx=0x7f85240316e0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 -- 172.21.15.27:0/4081493807 shutdown_connections 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cffe6710 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 --2- 172.21.15.27:0/4081493807 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffe5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 -- 172.21.15.27:0/4081493807 >> 172.21.15.27:0/4081493807 conn(0x5586cffae950 msgr2=0x5586cffe27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 -- 172.21.15.27:0/4081493807 shutdown_connections 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 -- 172.21.15.27:0/4081493807 wait complete. 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 -- start start 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.600+0000 7f853319d740 1 Processor -- start 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f853319d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cffedfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f853319d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffee500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f853319d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cfff1010 0x5586cffeea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f853319d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5586cfff3e40 con 0x5586cffe6290 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f853319d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5586cff05e90 con 0x5586cffe7980 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f853319d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5586cfff3fc0 con 0x5586cfff1010 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f853319d740 10 client.0 _ll_register_callbacks cb 0x5586cff41970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cffedfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cffedfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52100/0 (socket says 172.21.15.27:52100) 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 -- 172.21.15.27:0/1687987675 learned_addr learned my addr 172.21.15.27:0/1687987675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852e14c640 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 -- 172.21.15.27:0/1687987675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cfff1010 msgr2=0x5586cffeea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cfff1010 0x5586cffeea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 -- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 msgr2=0x5586cffee500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffee500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 -- 172.21.15.27:0/1687987675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586cfd267e0 con 0x5586cffe6290 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852e14c640 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffee500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f852d94b640 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cffedfc0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f85180094e0 tx=0x7f8518017f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45f6f94b9b2ea67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f8511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f8511ffb640 1 -- 172.21.15.27:0/1687987675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851801c070 con 0x5586cffe6290 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f8511ffb640 1 -- 172.21.15.27:0/1687987675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8518016bb0 con 0x5586cffe6290 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.601+0000 7f853319d740 1 -- 172.21.15.27:0/1687987675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8524005500 con 0x5586cffe6290 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f8511ffb640 1 -- 172.21.15.27:0/1687987675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8518028050 con 0x5586cffe6290 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f853319d740 10 client.16977 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f853319d740 1 -- 172.21.15.27:0/1687987675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5586cfd86f70 con 0x5586cffe6290 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f8511ffb640 1 -- 172.21.15.27:0/1687987675 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8524005500 con 0x5586cffe6290 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f8511ffb640 1 -- 172.21.15.27:0/1687987675 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8518020440 con 0x5586cffe6290 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f853319d740 10 client.16977 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f853319d740 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f853319d740 20 client.16977 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f853319d740 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5586d0001140 0x5586d00035a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f853319d740 4 client.16977 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f853319d740 1 -- 172.21.15.27:0/1687987675 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586cff1e5a0 con 0x5586d0001140 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.602+0000 7f852e14c640 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5586d0001140 0x5586d00035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.603+0000 7f852e14c640 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5586d0001140 0x5586d00035a0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a50e5ab20e99322 server_cookie=f49c6bb8405a59e1 in_seq=0 out_seq=0 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.603+0000 7f8511ffb640 10 client.16977 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f8511ffb640 1 -- 172.21.15.27:0/1687987675 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5586cff1e5a0 con 0x5586d0001140 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f8511ffb640 10 client.16977 handle_command_reply: tid=0 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 client.16977 shutdown 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 -- 172.21.15.27:0/1687987675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 msgr2=0x5586cffedfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cffedfc0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f85180094e0 tx=0x7f8518017f70 comp rx=0 tx=0).stop 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 -- 172.21.15.27:0/1687987675 shutdown_connections 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5586d0001140 0x5586d00035a0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cfff1010 0x5586cffeea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cffee500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f8511ffb640 0 client.16977 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:28.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 --2- 172.21.15.27:0/1687987675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cffedfc0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 -- 172.21.15.27:0/1687987675 >> 172.21.15.27:0/1687987675 conn(0x5586cffae950 msgr2=0x5586cffe8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 -- 172.21.15.27:0/1687987675 shutdown_connections 2024-04-25T01:31:28.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 1 -- 172.21.15.27:0/1687987675 wait complete. 2024-04-25T01:31:28.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.604+0000 7f853319d740 20 client.16977 trim_cache size 0 max 16384 2024-04-25T01:31:28.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.622+0000 7f853319d740 1 Processor -- start 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.622+0000 7f853319d740 1 -- start start 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f853319d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cfffcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f853319d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe7980 0x5586d0008e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f853319d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cfff1010 0x5586d0009360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f853319d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f85281b13f0 con 0x5586cffe6290 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f853319d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f85281aecc0 con 0x5586cfff1010 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f853319d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f85281acb10 con 0x5586cffe7980 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cfffcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cfffcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52110/0 (socket says 172.21.15.27:52110) 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 -- 172.21.15.27:0/968827642 learned_addr learned my addr 172.21.15.27:0/968827642 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852e14c640 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe7980 0x5586d0008e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 -- 172.21.15.27:0/968827642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe7980 msgr2=0x5586d0008e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe7980 0x5586d0008e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 -- 172.21.15.27:0/968827642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cfff1010 msgr2=0x5586d0009360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cfff1010 0x5586d0009360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 -- 172.21.15.27:0/968827642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586cfc1e170 con 0x5586cffe6290 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852e14c640 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe7980 0x5586d0008e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f852d94b640 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cfffcd40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f85180093c0 tx=0x7f8518017440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b494311a083f07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f8513fff640 1 -- 172.21.15.27:0/968827642 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518025070 con 0x5586cffe6290 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f8513fff640 1 -- 172.21.15.27:0/968827642 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85180205b0 con 0x5586cffe6290 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.623+0000 7f8513fff640 1 -- 172.21.15.27:0/968827642 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851801c070 con 0x5586cffe6290 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 -- 172.21.15.27:0/968827642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 msgr2=0x5586cfffcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cfffcd40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f85180093c0 tx=0x7f8518017440 comp rx=0 tx=0).stop 2024-04-25T01:31:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 -- 172.21.15.27:0/968827642 shutdown_connections 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cfff1010 0x5586d0009360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe7980 0x5586d0008e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 --2- 172.21.15.27:0/968827642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cffe6290 0x5586cfffcd40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 -- 172.21.15.27:0/968827642 >> 172.21.15.27:0/968827642 conn(0x5586cffae950 msgr2=0x5586cffe1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 -- 172.21.15.27:0/968827642 shutdown_connections 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 -- 172.21.15.27:0/968827642 wait complete. 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 -- start start 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.624+0000 7f853319d740 1 Processor -- start 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cfff9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cfff9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f85281ad180 con 0x5586cfff1010 2024-04-25T01:31:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f85281ae5f0 con 0x5586cffe7980 2024-04-25T01:31:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f85281ad850 con 0x5586cffe6290 2024-04-25T01:31:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 10 client.0 _ll_register_callbacks cb 0x7f85180093f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52122/0 (socket says 172.21.15.27:52122) 2024-04-25T01:31:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 -- 172.21.15.27:0/3354328562 learned_addr learned my addr 172.21.15.27:0/3354328562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852e14c640 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cfff9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cfff9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 -- 172.21.15.27:0/3354328562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 msgr2=0x5586cfff9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cfff9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 -- 172.21.15.27:0/3354328562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 msgr2=0x5586cfff9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cfff9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 -- 172.21.15.27:0/3354328562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586cfff6bd0 con 0x5586cfff1010 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f852d94b640 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff63d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8518009850 tx=0x7f8518016cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bf7c3767ecb09d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f8511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f8511ffb640 1 -- 172.21.15.27:0/3354328562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851801c070 con 0x5586cfff1010 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f8511ffb640 1 -- 172.21.15.27:0/3354328562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f85180203b0 con 0x5586cfff1010 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 1 -- 172.21.15.27:0/3354328562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85180205b0 con 0x5586cfff1010 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f8511ffb640 1 -- 172.21.15.27:0/3354328562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8518025040 con 0x5586cfff1010 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 10 client.16983 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.625+0000 7f853319d740 1 -- 172.21.15.27:0/3354328562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5586d006c700 con 0x5586cfff1010 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f8511ffb640 1 -- 172.21.15.27:0/3354328562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85180205b0 con 0x5586cfff1010 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f8511ffb640 1 -- 172.21.15.27:0/3354328562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8518018450 con 0x5586cfff1010 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f853319d740 10 client.16983 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f853319d740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f853319d740 20 client.16983 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f853319d740 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5586d006cd10 0x5586d006f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f853319d740 4 client.16983 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f853319d740 1 -- 172.21.15.27:0/3354328562 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5586cff57800 con 0x5586d006cd10 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f852e14c640 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5586d006cd10 0x5586d006f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f852e14c640 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5586d006cd10 0x5586d006f100 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6cfa3d3b8ddf20e server_cookie=2ea14ecf85f5a99c in_seq=0 out_seq=0 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.626+0000 7f8511ffb640 10 client.16983 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f8511ffb640 1 -- 172.21.15.27:0/3354328562 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5586cff57800 con 0x5586d006cd10 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f8511ffb640 10 client.16983 handle_command_reply: tid=0 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 client.16983 shutdown 2024-04-25T01:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 -- 172.21.15.27:0/3354328562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 msgr2=0x5586cfff63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff63d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8518009850 tx=0x7f8518016cc0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 -- 172.21.15.27:0/3354328562 shutdown_connections 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5586d006cd10 0x5586d006f100 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586cfff1010 0x5586cfff63d0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f8511ffb640 0 client.16983 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5586cffe7980 0x5586cfff9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 --2- 172.21.15.27:0/3354328562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5586cffe6290 0x5586cfff9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 -- 172.21.15.27:0/3354328562 >> 172.21.15.27:0/3354328562 conn(0x5586cffae950 msgr2=0x5586cffe7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 -- 172.21.15.27:0/3354328562 shutdown_connections 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 1 -- 172.21.15.27:0/3354328562 wait complete. 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.628+0000 7f853319d740 20 client.16983 trim_cache size 0 max 16384 2024-04-25T01:31:28.613 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:31:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:31:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "346406c6-64b3-46cf-bac6-20e0f89c3ffb", 2024-04-25T01:31:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:31:28.614 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 -- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d80 msgr2=0x7f85140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d80 0x7f85140a5240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8520098200 tx=0x7f852006dd60 comp rx=0 tx=0).stop 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 -- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 msgr2=0x7f852819c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 0x7f852819c8e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f852000cce0 tx=0x7f852000a620 comp rx=0 tx=0).stop 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 -- 172.21.15.27:0/566341813 shutdown_connections 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f85140a2d80 0x7f85140a5240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85281066b0 0x7f852819c8e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8528100fd0 0x7f85281985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 --2- 172.21.15.27:0/566341813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8528100610 0x7f8528198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 -- 172.21.15.27:0/566341813 >> 172.21.15.27:0/566341813 conn(0x7f8528077740 msgr2=0x7f8528104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 -- 172.21.15.27:0/566341813 shutdown_connections 2024-04-25T01:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.629+0000 7f85303b0640 1 -- 172.21.15.27:0/566341813 wait complete. 2024-04-25T01:31:28.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:31:28.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:31:28.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 -- 172.21.15.27:0/622590054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930010bd10 msgr2=0x7f930010e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 --2- 172.21.15.27:0/622590054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930010bd10 0x7f930010e1b0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009ca0 tx=0x7f92f802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 -- 172.21.15.27:0/622590054 shutdown_connections 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 --2- 172.21.15.27:0/622590054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930010bd10 0x7f930010e1b0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 --2- 172.21.15.27:0/622590054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93001032f0 0x7f930010b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 --2- 172.21.15.27:0/622590054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f93001029d0 0x7f9300102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 -- 172.21.15.27:0/622590054 >> 172.21.15.27:0/622590054 conn(0x7f93000769c0 msgr2=0x7f9300076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 -- 172.21.15.27:0/622590054 shutdown_connections 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 -- 172.21.15.27:0/622590054 wait complete. 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 Processor -- start 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.745+0000 7f93056a7640 1 -- start start 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f93056a7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93001029d0 0x7f9300197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f93056a7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93001032f0 0x7f9300198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f93056a7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f930010bd10 0x7f930019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f93056a7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9300111b10 con 0x7f93001029d0 2024-04-25T01:31:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f93056a7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9300111990 con 0x7f930010bd10 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f93056a7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9300111c90 con 0x7f93001032f0 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93001029d0 0x7f9300197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93001029d0 0x7f9300197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52144/0 (socket says 172.21.15.27:52144) 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 -- 172.21.15.27:0/3800227062 learned_addr learned my addr 172.21.15.27:0/3800227062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f930010bd10 0x7f930019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93001032f0 0x7f9300198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 -- 172.21.15.27:0/3800227062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93001032f0 msgr2=0x7f9300198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93001032f0 0x7f9300198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 -- 172.21.15.27:0/3800227062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f930010bd10 msgr2=0x7f930019c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f930010bd10 0x7f930019c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 -- 172.21.15.27:0/3800227062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930019cf00 con 0x7f93001029d0 2024-04-25T01:31:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93001032f0 0x7f9300198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92feffd640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93001029d0 0x7f9300197f10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f92f000d950 tx=0x7f92f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.746+0000 7f92e7fff640 1 -- 172.21.15.27:0/3800227062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0017070 con 0x7f93001029d0 2024-04-25T01:31:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.747+0000 7f92e7fff640 1 -- 172.21.15.27:0/3800227062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92f0009c30 con 0x7f93001029d0 2024-04-25T01:31:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.747+0000 7f93056a7640 1 -- 172.21.15.27:0/3800227062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f930019d1f0 con 0x7f93001029d0 2024-04-25T01:31:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.747+0000 7f92e7fff640 1 -- 172.21.15.27:0/3800227062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f000f6b0 con 0x7f93001029d0 2024-04-25T01:31:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.747+0000 7f93056a7640 1 -- 172.21.15.27:0/3800227062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93001a4a80 con 0x7f93001029d0 2024-04-25T01:31:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f93083b0740 1 Processor -- start 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f93083b0740 1 -- start start 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f92e7fff640 1 -- 172.21.15.27:0/3800227062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f92f00109d0 con 0x7f93001029d0 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f93083b0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6630 0x55e763bb6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f93083b0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 0x55e763bc10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f93083b0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bc3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f93083b0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e763b1b6b0 con 0x55e763bc1620 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f93083b0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e763ac3880 con 0x55e763bb6f50 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f93083b0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e763b17670 con 0x55e763bb6630 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f92fe7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bc3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f92fe7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bc3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52156/0 (socket says 172.21.15.27:52156) 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f92fe7fc640 1 -- 172.21.15.27:0/1860863971 learned_addr learned my addr 172.21.15.27:0/1860863971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.749+0000 7f92feffd640 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 0x55e763bc10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6630 0x55e763bb6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92fe7fc640 1 -- 172.21.15.27:0/1860863971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6630 msgr2=0x55e763bb6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6630 0x55e763bb6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92fe7fc640 1 -- 172.21.15.27:0/1860863971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 msgr2=0x55e763bc10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 0x55e763bc10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92fe7fc640 1 -- 172.21.15.27:0/1860863971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7638f97d0 con 0x55e763bc1620 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92e7fff640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f92e80a2d00 0x7f92e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f92e80a2d00 0x7f92e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92e7fff640 1 -- 172.21.15.27:0/3800227062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f92f00cf430 con 0x7f93001029d0 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bc3a10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f92f400bb60 tx=0x7f92f400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fea447b7ac55d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92e57fa640 1 -- 172.21.15.27:0/1860863971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f400be60 con 0x55e763bc1620 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92e57fa640 1 -- 172.21.15.27:0/1860863971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92f4017b00 con 0x55e763bc1620 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.750+0000 7f92e57fa640 1 -- 172.21.15.27:0/1860863971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f400fc90 con 0x55e763bc1620 2024-04-25T01:31:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- 172.21.15.27:0/1860863971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 msgr2=0x55e763bc3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f92e80a2d00 0x7f92e80a51c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92f8005e30 tx=0x7f92f8005c50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bc3a10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f92f400bb60 tx=0x7f92f400d710 comp rx=0 tx=0).stop 2024-04-25T01:31:28.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- 172.21.15.27:0/1860863971 shutdown_connections 2024-04-25T01:31:28.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bc3a10 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 0x55e763bc10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 --2- 172.21.15.27:0/1860863971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6630 0x55e763bb6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- 172.21.15.27:0/1860863971 >> 172.21.15.27:0/1860863971 conn(0x55e763bb2950 msgr2=0x55e763bb2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- 172.21.15.27:0/1860863971 shutdown_connections 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- 172.21.15.27:0/1860863971 wait complete. 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- start start 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 Processor -- start 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6630 0x55e763bbe570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 0x55e763bbeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bbeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e763bc4410 con 0x55e763bc1620 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e763ad6f50 con 0x55e763bb6630 2024-04-25T01:31:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.751+0000 7f93083b0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e763bc4590 con 0x55e763bb6f50 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f93083b0740 10 client.0 _ll_register_callbacks cb 0x55e763b12000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bbeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bbeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52160/0 (socket says 172.21.15.27:52160) 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 -- 172.21.15.27:0/2221580723 learned_addr learned my addr 172.21.15.27:0/2221580723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 0x55e763bbeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92feffd640 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6630 0x55e763bbe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 -- 172.21.15.27:0/2221580723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 msgr2=0x55e763bbeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 0x55e763bbeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 -- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6630 msgr2=0x55e763bbe570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6630 0x55e763bbe570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 -- 172.21.15.27:0/2221580723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e76382f3f0 con 0x55e763bc1620 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92feffd640 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6630 0x55e763bbe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bbeff0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f92f8035000 tx=0x7f92f80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c91dbdc45c2c63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92e2ff5640 1 -- 172.21.15.27:0/2221580723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8054060 con 0x55e763bc1620 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92e2ff5640 1 -- 172.21.15.27:0/2221580723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f92f8004450 con 0x55e763bc1620 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f93083b0740 1 -- 172.21.15.27:0/2221580723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92f4017b00 con 0x55e763bc1620 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f92e2ff5640 1 -- 172.21.15.27:0/2221580723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f92f805b050 con 0x55e763bc1620 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f93083b0740 10 client.16995 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.752+0000 7f93083b0740 1 -- 172.21.15.27:0/2221580723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e7637ef0f0 con 0x55e763bc1620 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f92e2ff5640 1 -- 172.21.15.27:0/2221580723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92f4017b00 con 0x55e763bc1620 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f92e2ff5640 1 -- 172.21.15.27:0/2221580723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f92f80365d0 con 0x55e763bc1620 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f93083b0740 10 client.16995 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f93083b0740 10 client.16995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f93083b0740 20 client.16995 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f93083b0740 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e763bd18c0 0x55e763bd3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f93083b0740 4 client.16995 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f93083b0740 1 -- 172.21.15.27:0/2221580723 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e763b28bf0 con 0x55e763bd18c0 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f92feffd640 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e763bd18c0 0x55e763bd3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f92feffd640 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e763bd18c0 0x55e763bd3d10 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c470153c23473154 server_cookie=91f7496abc0f5b33 in_seq=0 out_seq=0 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.753+0000 7f92e2ff5640 10 client.16995 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f92e2ff5640 1 -- 172.21.15.27:0/2221580723 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e763b28bf0 con 0x55e763bd18c0 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f92e2ff5640 10 client.16995 handle_command_reply: tid=0 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 client.16995 shutdown 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 -- 172.21.15.27:0/2221580723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 msgr2=0x55e763bbeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bbeff0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f92f8035000 tx=0x7f92f80095c0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 -- 172.21.15.27:0/2221580723 shutdown_connections 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e763bd18c0 0x55e763bd3d10 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bc1620 0x55e763bbeff0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 0x55e763bbeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f92e2ff5640 0 client.16995 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:28.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 --2- 172.21.15.27:0/2221580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6630 0x55e763bbe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 -- 172.21.15.27:0/2221580723 >> 172.21.15.27:0/2221580723 conn(0x55e763bb2950 msgr2=0x55e763bb8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 -- 172.21.15.27:0/2221580723 shutdown_connections 2024-04-25T01:31:28.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.754+0000 7f93083b0740 1 -- 172.21.15.27:0/2221580723 wait complete. 2024-04-25T01:31:28.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.755+0000 7f93083b0740 20 client.16995 trim_cache size 0 max 16384 2024-04-25T01:31:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f93083b0740 1 Processor -- start 2024-04-25T01:31:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f93083b0740 1 -- start start 2024-04-25T01:31:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f93083b0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763bcd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f93083b0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 0x55e763bc9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f93083b0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bc1620 0x55e763bb8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f93083b0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93001b1460 con 0x55e763bb6630 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f93083b0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f93001aed30 con 0x55e763bb6f50 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f93083b0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93001acb80 con 0x55e763bc1620 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f92feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763bcd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f92feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763bcd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52176/0 (socket says 172.21.15.27:52176) 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.772+0000 7f92feffd640 1 -- 172.21.15.27:0/3592807410 learned_addr learned my addr 172.21.15.27:0/3592807410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92feffd640 1 -- 172.21.15.27:0/3592807410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bc1620 msgr2=0x55e763bb8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92feffd640 1 --2- 172.21.15.27:0/3592807410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bc1620 0x55e763bb8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92feffd640 1 -- 172.21.15.27:0/3592807410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 msgr2=0x55e763bc9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92feffd640 1 --2- 172.21.15.27:0/3592807410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 0x55e763bc9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92feffd640 1 -- 172.21.15.27:0/3592807410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e763ba4090 con 0x55e763bb6630 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92feffd640 1 --2- 172.21.15.27:0/3592807410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763bcd2d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f92f009b2c0 tx=0x7f92f00d1940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b032fe6c06f19125 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92e4ff9640 1 -- 172.21.15.27:0/3592807410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0020040 con 0x55e763bb6630 2024-04-25T01:31:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92e4ff9640 1 -- 172.21.15.27:0/3592807410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92f009bcc0 con 0x55e763bb6630 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f92e4ff9640 1 -- 172.21.15.27:0/3592807410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f00cf1b0 con 0x55e763bb6630 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 -- 172.21.15.27:0/3592807410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 msgr2=0x55e763bcd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 --2- 172.21.15.27:0/3592807410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763bcd2d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f92f009b2c0 tx=0x7f92f00d1940 comp rx=0 tx=0).stop 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 -- 172.21.15.27:0/3592807410 shutdown_connections 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 --2- 172.21.15.27:0/3592807410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bc1620 0x55e763bb8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 --2- 172.21.15.27:0/3592807410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bb6f50 0x55e763bc9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 --2- 172.21.15.27:0/3592807410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763bcd2d0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 -- 172.21.15.27:0/3592807410 >> 172.21.15.27:0/3592807410 conn(0x55e763bb2950 msgr2=0x55e763bb1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 -- 172.21.15.27:0/3592807410 shutdown_connections 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.773+0000 7f93083b0740 1 -- 172.21.15.27:0/3592807410 wait complete. 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 1 -- start start 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 1 Processor -- start 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763c34640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 0x55e763c34b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bc1620 0x55e763bd8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93001ad1f0 con 0x55e763bb6630 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f93001ae660 con 0x55e763bc1620 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93001ad8c0 con 0x55e763bb6f50 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f93083b0740 10 client.0 _ll_register_callbacks cb 0x7f92f8009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763c34640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763c34640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52182/0 (socket says 172.21.15.27:52182) 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92feffd640 1 -- 172.21.15.27:0/132498934 learned_addr learned my addr 172.21.15.27:0/132498934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bc1620 0x55e763bd8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 0x55e763c34b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92feffd640 1 -- 172.21.15.27:0/132498934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 msgr2=0x55e763c34b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92feffd640 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 0x55e763c34b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92feffd640 1 -- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bc1620 msgr2=0x55e763bd8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92feffd640 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bc1620 0x55e763bd8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92feffd640 1 -- 172.21.15.27:0/132498934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e763bd9330 con 0x55e763bb6630 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.774+0000 7f92ff7fe640 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bc1620 0x55e763bd8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f92feffd640 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763c34640 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f92f009b550 tx=0x7f92f0010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f57ef0deec09c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f92e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f92e2ff5640 1 -- 172.21.15.27:0/132498934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0020070 con 0x55e763bb6630 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f92e2ff5640 1 -- 172.21.15.27:0/132498934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f92f00cca70 con 0x55e763bb6630 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 1 -- 172.21.15.27:0/132498934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92f009bcc0 con 0x55e763bb6630 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f92e2ff5640 1 -- 172.21.15.27:0/132498934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f92f00a4230 con 0x55e763bb6630 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 10 client.17001 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 1 -- 172.21.15.27:0/132498934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e763c3bd60 con 0x55e763bb6630 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f92e2ff5640 1 -- 172.21.15.27:0/132498934 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92f009bcc0 con 0x55e763bb6630 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f92e2ff5640 1 -- 172.21.15.27:0/132498934 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f92f009ab30 con 0x55e763bb6630 2024-04-25T01:31:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 10 client.17001 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 10 client.17001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 20 client.17001 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e763c3c4d0 0x55e763c3e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 4 client.17001 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f93083b0740 1 -- 172.21.15.27:0/132498934 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e763b22f40 con 0x55e763c3c4d0 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.775+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e763c3c4d0 0x55e763c3e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f92e2ff5640 10 client.17001 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f92fe7fc640 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e763c3c4d0 0x55e763c3e8f0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbd69d24be767ba0 server_cookie=7c3be861113cd85d in_seq=0 out_seq=0 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f92e2ff5640 1 -- 172.21.15.27:0/132498934 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e763b22f40 con 0x55e763c3c4d0 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f92e2ff5640 10 client.17001 handle_command_reply: tid=0 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 client.17001 shutdown 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 -- 172.21.15.27:0/132498934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 msgr2=0x55e763c34640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763c34640 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f92f009b550 tx=0x7f92f0010d70 comp rx=0 tx=0).stop 2024-04-25T01:31:28.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 -- 172.21.15.27:0/132498934 shutdown_connections 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e763c3c4d0 0x55e763c3e8f0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e763bc1620 0x55e763bd8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e763bb6f50 0x55e763c34b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 --2- 172.21.15.27:0/132498934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e763bb6630 0x55e763c34640 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f92e2ff5640 0 client.17001 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 -- 172.21.15.27:0/132498934 >> 172.21.15.27:0/132498934 conn(0x55e763bb2950 msgr2=0x55e763bb1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 -- 172.21.15.27:0/132498934 shutdown_connections 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 1 -- 172.21.15.27:0/132498934 wait complete. 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.776+0000 7f93083b0740 20 client.17001 trim_cache size 0 max 16384 2024-04-25T01:31:28.761 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout: "346406c6-64b3-46cf-bac6-20e0f89c3ffb": { 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "346406c6-64b3-46cf-bac6-20e0f89c3ffb", 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:31:28.762 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:31:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 -- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f92e80a2d00 msgr2=0x7f92e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f92e80a2d00 0x7f92e80a51c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92f8005e30 tx=0x7f92f8005c50 comp rx=0 tx=0).stop 2024-04-25T01:31:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 -- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93001029d0 msgr2=0x7f9300197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93001029d0 0x7f9300197f10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f92f000d950 tx=0x7f92f000de10 comp rx=0 tx=0).stop 2024-04-25T01:31:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 -- 172.21.15.27:0/3800227062 shutdown_connections 2024-04-25T01:31:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f930010bd10 0x7f930019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93001032f0 0x7f9300198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f92e80a2d00 0x7f92e80a51c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 --2- 172.21.15.27:0/3800227062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93001029d0 0x7f9300197f10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 -- 172.21.15.27:0/3800227062 >> 172.21.15.27:0/3800227062 conn(0x7f93000769c0 msgr2=0x7f93000fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 -- 172.21.15.27:0/3800227062 shutdown_connections 2024-04-25T01:31:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:28.777+0000 7f93056a7640 1 -- 172.21.15.27:0/3800227062 wait complete. 2024-04-25T01:31:28.764 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:28 smithi027 ceph-mon[29197]: pgmap v184: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:28.765 INFO:tasks.cephfs.filesystem:scrub status for tag:346406c6-64b3-46cf-bac6-20e0f89c3ffb - {'path': '/', 'tag': '346406c6-64b3-46cf-bac6-20e0f89c3ffb', 'options': 'recursive,force'} 2024-04-25T01:31:28.924 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:28 smithi153 ceph-mon[32621]: pgmap v184: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:29.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:28 smithi139 ceph-mon[32760]: pgmap v184: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:29.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:29 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:31:29.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:29 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:29.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:29 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:31:29.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:29 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:31:29.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:29 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:29.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:29 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:30.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:29 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:31:30.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:29 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:30.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:29 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:31:30.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:29 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:31:30.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:29 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:30.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:29 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:30.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:29 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:31:30.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:29 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:30.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:29 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:31:30.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:29 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:31:30.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:29 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:30.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:29 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:31:30.941 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:30 smithi139 ceph-mon[32760]: pgmap v185: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:31:30.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:30 smithi027 ceph-mon[29197]: pgmap v185: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:31:31.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:30 smithi153 ceph-mon[32621]: pgmap v185: 129 pgs: 129 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:31:32.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:32 smithi027 ceph-mon[29197]: pgmap v186: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:33.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:32 smithi153 ceph-mon[32621]: pgmap v186: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:33.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:32 smithi139 ceph-mon[32760]: pgmap v186: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:34 smithi027 ceph-mon[29197]: pgmap v187: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 9 op/s 2024-04-25T01:31:34.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:34 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:35.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:34 smithi153 ceph-mon[32621]: pgmap v187: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 9 op/s 2024-04-25T01:31:35.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:34 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:35.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:34 smithi139 ceph-mon[32760]: pgmap v187: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 9 op/s 2024-04-25T01:31:35.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:34 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:35.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:35 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:31:35.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:35 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:31:35.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:35 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:31:36.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:35 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:31:36.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:35 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:31:36.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:35 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:31:36.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:35 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:31:36.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:35 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:31:36.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:35 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:31:36.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:36 smithi027 ceph-mon[29197]: pgmap v188: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 9 op/s 2024-04-25T01:31:37.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:36 smithi153 ceph-mon[32621]: pgmap v188: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 9 op/s 2024-04-25T01:31:37.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:36 smithi139 ceph-mon[32760]: pgmap v188: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 KiB/s wr, 9 op/s 2024-04-25T01:31:37.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:37 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:31:38.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:37 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:31:38.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:37 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:31:38.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:38 smithi027 ceph-mon[29197]: pgmap v189: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:38.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:38 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:31:39.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:38 smithi153 ceph-mon[32621]: pgmap v189: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:39.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:38 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:31:39.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:38 smithi139 ceph-mon[32760]: pgmap v189: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:39.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:38 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:31:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:40 smithi027 ceph-mon[29197]: pgmap v190: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:41.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:40 smithi153 ceph-mon[32621]: pgmap v190: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:41.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:40 smithi139 ceph-mon[32760]: pgmap v190: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:42 smithi027 ceph-mon[29197]: pgmap v191: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:43.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:42 smithi153 ceph-mon[32621]: pgmap v191: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:43.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:42 smithi139 ceph-mon[32760]: pgmap v191: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:44 smithi027 ceph-mon[29197]: pgmap v192: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:31:45.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:44 smithi153 ceph-mon[32621]: pgmap v192: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:31:45.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:44 smithi139 ceph-mon[32760]: pgmap v192: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:31:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:46 smithi027 ceph-mon[29197]: pgmap v193: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:31:47.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:46 smithi153 ceph-mon[32621]: pgmap v193: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:31:47.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:46 smithi139 ceph-mon[32760]: pgmap v193: 129 pgs: 129 active+clean; 3.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:31:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:48 smithi027 ceph-mon[29197]: pgmap v194: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:31:49.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:48 smithi153 ceph-mon[32621]: pgmap v194: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:31:49.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:48 smithi139 ceph-mon[32760]: pgmap v194: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:31:50.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:50 smithi027 ceph-mon[29197]: pgmap v195: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:51.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:50 smithi153 ceph-mon[32621]: pgmap v195: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:51.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:50 smithi139 ceph-mon[32760]: pgmap v195: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:52 smithi027 ceph-mon[29197]: pgmap v196: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:53.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:52 smithi153 ceph-mon[32621]: pgmap v196: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:53.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:52 smithi139 ceph-mon[32760]: pgmap v196: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:31:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:54 smithi027 ceph-mon[29197]: pgmap v197: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:55.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:54 smithi153 ceph-mon[32621]: pgmap v197: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:55.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:54 smithi139 ceph-mon[32760]: pgmap v197: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:56.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:56 smithi027 ceph-mon[29197]: pgmap v198: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:57.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:56 smithi153 ceph-mon[32621]: pgmap v198: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:57.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:56 smithi139 ceph-mon[32760]: pgmap v198: 129 pgs: 129 active+clean; 3.2 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:31:58.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:31:58.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:31:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 -- 172.21.15.27:0/2559108572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c06a780 msgr2=0x7f6d4c10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/2559108572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c06a780 0x7f6d4c10e5f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c009cd0 tx=0x7f6d3c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 -- 172.21.15.27:0/2559108572 shutdown_connections 2024-04-25T01:31:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/2559108572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c06a780 0x7f6d4c10e5f0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/2559108572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d4c069dc0 0x7f6d4c06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/2559108572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c107730 0x7f6d4c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 -- 172.21.15.27:0/2559108572 >> 172.21.15.27:0/2559108572 conn(0x7f6d4c0776a0 msgr2=0x7f6d4c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 -- 172.21.15.27:0/2559108572 shutdown_connections 2024-04-25T01:31:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.905+0000 7f6d50ad2640 1 -- 172.21.15.27:0/2559108572 wait complete. 2024-04-25T01:31:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.906+0000 7f6d50ad2640 1 Processor -- start 2024-04-25T01:31:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.906+0000 7f6d50ad2640 1 -- start start 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d50ad2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d4c069dc0 0x7f6d4c100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d50ad2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c06a780 0x7f6d4c100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d50ad2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c107730 0x7f6d4c1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d50ad2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c111b40 con 0x7f6d4c107730 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d50ad2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1119c0 con 0x7f6d4c069dc0 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d50ad2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c111cc0 con 0x7f6d4c06a780 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c107730 0x7f6d4c1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c107730 0x7f6d4c1010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50136/0 (socket says 172.21.15.27:50136) 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 -- 172.21.15.27:0/1073004979 learned_addr learned my addr 172.21.15.27:0/1073004979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d4c069dc0 0x7f6d4c100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 -- 172.21.15.27:0/1073004979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c06a780 msgr2=0x7f6d4c100bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c06a780 0x7f6d4c100bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 -- 172.21.15.27:0/1073004979 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d4c069dc0 msgr2=0x7f6d4c100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d4c069dc0 0x7f6d4c100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 -- 172.21.15.27:0/1073004979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4c1a6460 con 0x7f6d4c107730 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d4ad9d640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c107730 0x7f6d4c1010f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c005cc0 tx=0x7f6d3c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d3b7fe640 1 -- 172.21.15.27:0/1073004979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c004040 con 0x7f6d4c107730 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d3b7fe640 1 -- 172.21.15.27:0/1073004979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d3c0046c0 con 0x7f6d4c107730 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d50ad2640 1 -- 172.21.15.27:0/1073004979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4c1a66f0 con 0x7f6d4c107730 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.907+0000 7f6d3b7fe640 1 -- 172.21.15.27:0/1073004979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c046480 con 0x7f6d4c107730 2024-04-25T01:31:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.908+0000 7f6d50ad2640 1 -- 172.21.15.27:0/1073004979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d4c1a6c90 con 0x7f6d4c107730 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.910+0000 7f6d3b7fe640 1 -- 172.21.15.27:0/1073004979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6d3c0305a0 con 0x7f6d4c107730 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.910+0000 7f6d537dd740 1 Processor -- start 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.910+0000 7f6d537dd740 1 -- start start 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d3b7fe640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6d340a2d80 0x7f6d340a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8630 0x55d4138f8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8f50 0x55d4139030e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413905a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d537dd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d41385d6b0 con 0x55d4138f8630 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d537dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d413805880 con 0x55d4138f8f50 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d537dd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d413859670 con 0x55d413903620 2024-04-25T01:31:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8630 0x55d4138f8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6d340a2d80 0x7f6d340a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8630 0x55d4138f8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50152/0 (socket says 172.21.15.27:50152) 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d3b7fe640 1 -- 172.21.15.27:0/1073004979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6d3c02e030 con 0x7f6d4c107730 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 -- 172.21.15.27:0/2622222825 learned_addr learned my addr 172.21.15.27:0/2622222825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4ad9d640 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413905a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8f50 0x55d4139030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 -- 172.21.15.27:0/2622222825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 msgr2=0x55d413905a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413905a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 -- 172.21.15.27:0/2622222825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8f50 msgr2=0x55d4139030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8f50 0x55d4139030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 -- 172.21.15.27:0/2622222825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d41363b7d0 con 0x55d4138f8630 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8f50 0x55d4139030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8630 0x55d4138f8a10 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400bc20 tx=0x7f6d4400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd876422516f976 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.911+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6d340a2d80 0x7f6d340a5240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55d4138f9bf0 tx=0x7f6d4000a270 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d38ff9640 1 -- 172.21.15.27:0/2622222825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44015070 con 0x55d4138f8630 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d38ff9640 1 -- 172.21.15.27:0/2622222825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d44005500 con 0x55d4138f8630 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d38ff9640 1 -- 172.21.15.27:0/2622222825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d440116a0 con 0x55d4138f8630 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d537dd740 1 -- 172.21.15.27:0/2622222825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8630 msgr2=0x55d4138f8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8630 0x55d4138f8a10 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400bc20 tx=0x7f6d4400f730 comp rx=0 tx=0).stop 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d537dd740 1 -- 172.21.15.27:0/2622222825 shutdown_connections 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413905a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8f50 0x55d4139030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2622222825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8630 0x55d4138f8a10 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.912+0000 7f6d537dd740 1 -- 172.21.15.27:0/2622222825 >> 172.21.15.27:0/2622222825 conn(0x55d4138f4950 msgr2=0x55d4138f4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 -- 172.21.15.27:0/2622222825 shutdown_connections 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 -- 172.21.15.27:0/2622222825 wait complete. 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 -- start start 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 Processor -- start 2024-04-25T01:31:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d4139005e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413900b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413901060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d413906410 con 0x55d4138f8f50 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d413818f50 con 0x55d4138f8630 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d413906590 con 0x55d413903620 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d537dd740 10 client.0 _ll_register_callbacks cb 0x55d413854000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413900b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413900b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50162/0 (socket says 172.21.15.27:50162) 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d49d9b640 1 -- 172.21.15.27:0/3186933100 learned_addr learned my addr 172.21.15.27:0/3186933100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.913+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d4139005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d4ad9d640 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413901060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d49d9b640 1 -- 172.21.15.27:0/3186933100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 msgr2=0x55d413901060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413901060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d49d9b640 1 -- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 msgr2=0x55d4139005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d4139005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d49d9b640 1 -- 172.21.15.27:0/3186933100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4135713f0 con 0x55d4138f8f50 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d4139005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413900b20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6d40011e00 tx=0x7f6d40005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdcb1df06d24fc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d31ffb640 1 -- 172.21.15.27:0/3186933100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40022070 con 0x55d4138f8f50 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d31ffb640 1 -- 172.21.15.27:0/3186933100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6d4001e5f0 con 0x55d4138f8f50 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d537dd740 1 -- 172.21.15.27:0/3186933100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d44005500 con 0x55d4138f8f50 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d31ffb640 1 -- 172.21.15.27:0/3186933100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d4002e050 con 0x55d4138f8f50 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d537dd740 10 client.17013 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d537dd740 1 -- 172.21.15.27:0/3186933100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d4135310f0 con 0x55d4138f8f50 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.914+0000 7f6d31ffb640 1 -- 172.21.15.27:0/3186933100 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d44005500 con 0x55d4138f8f50 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d31ffb640 1 -- 172.21.15.27:0/3186933100 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6d4001d070 con 0x55d4138f8f50 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d537dd740 10 client.17013 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d537dd740 10 client.17013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d537dd740 20 client.17013 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d537dd740 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d413913990 0x55d413915e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d537dd740 4 client.17013 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d537dd740 1 -- 172.21.15.27:0/3186933100 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d41386abf0 con 0x55d413913990 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d413913990 0x55d413915e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d413913990 0x55d413915e20 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4ed30e20c967db1 server_cookie=8c4660748fb5da9b in_seq=0 out_seq=0 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.915+0000 7f6d31ffb640 10 client.17013 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d31ffb640 1 -- 172.21.15.27:0/3186933100 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d41386abf0 con 0x55d413913990 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d31ffb640 10 client.17013 handle_command_reply: tid=0 2024-04-25T01:31:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 client.17013 shutdown 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 -- 172.21.15.27:0/3186933100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 msgr2=0x55d413900b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413900b20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6d40011e00 tx=0x7f6d40005c00 comp rx=0 tx=0).stop 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 -- 172.21.15.27:0/3186933100 shutdown_connections 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d413913990 0x55d413915e20 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413901060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413900b20 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 --2- 172.21.15.27:0/3186933100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d4139005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d31ffb640 0 client.17013 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 -- 172.21.15.27:0/3186933100 >> 172.21.15.27:0/3186933100 conn(0x55d4138f4950 msgr2=0x55d4138fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.916+0000 7f6d537dd740 1 -- 172.21.15.27:0/3186933100 shutdown_connections 2024-04-25T01:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.917+0000 7f6d537dd740 1 -- 172.21.15.27:0/3186933100 wait complete. 2024-04-25T01:31:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.917+0000 7f6d537dd740 20 client.17013 trim_cache size 0 max 16384 2024-04-25T01:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d537dd740 1 Processor -- start 2024-04-25T01:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d537dd740 1 -- start start 2024-04-25T01:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d4138f8630 0x55d41390f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d4138faea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d413903620 0x55d413902590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d537dd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1b3620 con 0x55d4138f8f50 2024-04-25T01:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d537dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1b0ef0 con 0x55d413903620 2024-04-25T01:31:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d537dd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1aed40 con 0x55d4138f8630 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d4138faea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d4138faea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50170/0 (socket says 172.21.15.27:50170) 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 -- 172.21.15.27:0/1085416011 learned_addr learned my addr 172.21.15.27:0/1085416011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d4138f8630 0x55d41390f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d4ad9d640 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d413903620 0x55d413902590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 -- 172.21.15.27:0/1085416011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d4138f8630 msgr2=0x55d41390f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d4138f8630 0x55d41390f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 -- 172.21.15.27:0/1085416011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d413903620 msgr2=0x55d413902590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d413903620 0x55d413902590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 -- 172.21.15.27:0/1085416011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4138ff590 con 0x55d4138f8f50 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d4138f8630 0x55d41390f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d4138faea0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6d40005d00 tx=0x7f6d4000e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac84f5d3683d07ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.935+0000 7f6d33fff640 1 -- 172.21.15.27:0/1085416011 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40037070 con 0x55d4138f8f50 2024-04-25T01:31:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d33fff640 1 -- 172.21.15.27:0/1085416011 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d40018070 con 0x55d4138f8f50 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d33fff640 1 -- 172.21.15.27:0/1085416011 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40022040 con 0x55d4138f8f50 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 -- 172.21.15.27:0/1085416011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 msgr2=0x55d4138faea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d4138faea0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6d40005d00 tx=0x7f6d4000e8d0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 -- 172.21.15.27:0/1085416011 shutdown_connections 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d413903620 0x55d413902590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d4138faea0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 --2- 172.21.15.27:0/1085416011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d4138f8630 0x55d41390f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 -- 172.21.15.27:0/1085416011 >> 172.21.15.27:0/1085416011 conn(0x55d4138f4950 msgr2=0x55d4138f4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 -- 172.21.15.27:0/1085416011 shutdown_connections 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 -- 172.21.15.27:0/1085416011 wait complete. 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.936+0000 7f6d537dd740 1 -- start start 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d537dd740 1 Processor -- start 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d41390bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413906f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d537dd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413907460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d537dd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1af3b0 con 0x55d4138f8f50 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d537dd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1b0820 con 0x55d4138f8630 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d537dd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1afa80 con 0x55d413903620 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d537dd740 10 client.0 _ll_register_callbacks cb 0x7f6d40011a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413906f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413906f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50176/0 (socket says 172.21.15.27:50176) 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 -- 172.21.15.27:0/2520438845 learned_addr learned my addr 172.21.15.27:0/2520438845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d41390bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d4ad9d640 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413907460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 -- 172.21.15.27:0/2520438845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 msgr2=0x55d413907460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413907460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 -- 172.21.15.27:0/2520438845 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 msgr2=0x55d41390bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d41390bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 -- 172.21.15.27:0/2520438845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d413907c40 con 0x55d4138f8f50 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d4ad9d640 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413907460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d49d9b640 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413906f20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f6d40005e00 tx=0x7f6d40010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0ce63b758cc730 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.937+0000 7f6d31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d31ffb640 1 -- 172.21.15.27:0/2520438845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40037070 con 0x55d4138f8f50 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d31ffb640 1 -- 172.21.15.27:0/2520438845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6d4001ec00 con 0x55d4138f8f50 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 1 -- 172.21.15.27:0/2520438845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d40018070 con 0x55d4138f8f50 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d31ffb640 1 -- 172.21.15.27:0/2520438845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d40022040 con 0x55d4138f8f50 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 10 client.17019 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 1 -- 172.21.15.27:0/2520438845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d413979f80 con 0x55d4138f8f50 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d31ffb640 1 -- 172.21.15.27:0/2520438845 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d40018070 con 0x55d4138f8f50 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d31ffb640 1 -- 172.21.15.27:0/2520438845 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6d4000f040 con 0x55d4138f8f50 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 10 client.17019 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 10 client.17019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 20 client.17019 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d41397a580 0x55d41397c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 4 client.17019 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d537dd740 1 -- 172.21.15.27:0/2520438845 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d413864f40 con 0x55d41397a580 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.938+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d41397a580 0x55d41397c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d4a59c640 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d41397a580 0x55d41397c9a0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8672c76219371dc8 server_cookie=1dd5590f4c3f0d3c in_seq=0 out_seq=0 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d31ffb640 10 client.17019 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d31ffb640 1 -- 172.21.15.27:0/2520438845 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d413864f40 con 0x55d41397a580 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d31ffb640 10 client.17019 handle_command_reply: tid=0 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 client.17019 shutdown 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 -- 172.21.15.27:0/2520438845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 msgr2=0x55d413906f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413906f20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f6d40005e00 tx=0x7f6d40010d70 comp rx=0 tx=0).stop 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 -- 172.21.15.27:0/2520438845 shutdown_connections 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55d41397a580 0x55d41397c9a0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d413903620 0x55d413907460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d4138f8f50 0x55d413906f20 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d31ffb640 0 client.17019 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 --2- 172.21.15.27:0/2520438845 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d4138f8630 0x55d41390bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 -- 172.21.15.27:0/2520438845 >> 172.21.15.27:0/2520438845 conn(0x55d4138f4950 msgr2=0x55d413900e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 -- 172.21.15.27:0/2520438845 shutdown_connections 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.939+0000 7f6d537dd740 1 -- 172.21.15.27:0/2520438845 wait complete. 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d537dd740 20 client.17019 trim_cache size 0 max 16384 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:31:58.924 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:31:58.925 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:31:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 -- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6d340a2d80 msgr2=0x7f6d340a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6d340a2d80 0x7f6d340a5240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55d4138f9bf0 tx=0x7f6d4000a270 comp rx=0 tx=0).stop 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 -- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c107730 msgr2=0x7f6d4c1010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c107730 0x7f6d4c1010f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c005cc0 tx=0x7f6d3c003970 comp rx=0 tx=0).stop 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 -- 172.21.15.27:0/1073004979 shutdown_connections 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6d340a2d80 0x7f6d340a5240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d4c107730 0x7f6d4c1010f0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c06a780 0x7f6d4c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 --2- 172.21.15.27:0/1073004979 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6d4c069dc0 0x7f6d4c100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 -- 172.21.15.27:0/1073004979 >> 172.21.15.27:0/1073004979 conn(0x7f6d4c0776a0 msgr2=0x7f6d4c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 -- 172.21.15.27:0/1073004979 shutdown_connections 2024-04-25T01:31:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:58.940+0000 7f6d50ad2640 1 -- 172.21.15.27:0/1073004979 wait complete. 2024-04-25T01:31:58.928 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:58 smithi027 ceph-mon[29197]: pgmap v199: 129 pgs: 129 active+clean; 3.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:31:58.929 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:31:58.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:31:59.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:58 smithi153 ceph-mon[32621]: pgmap v199: 129 pgs: 129 active+clean; 3.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 -- 172.21.15.27:0/677939698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c100fe0 msgr2=0x7f470c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 --2- 172.21.15.27:0/677939698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c100fe0 0x7f470c10d4f0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4704009cd0 tx=0x7f470402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 -- 172.21.15.27:0/677939698 shutdown_connections 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 --2- 172.21.15.27:0/677939698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c100fe0 0x7f470c10d4f0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 --2- 172.21.15.27:0/677939698 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f470c100620 0x7f470c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 --2- 172.21.15.27:0/677939698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f470c106630 0x7f470c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 -- 172.21.15.27:0/677939698 >> 172.21.15.27:0/677939698 conn(0x7f470c077740 msgr2=0x7f470c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 -- 172.21.15.27:0/677939698 shutdown_connections 2024-04-25T01:31:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.053+0000 7f47140bc640 1 -- 172.21.15.27:0/677939698 wait complete. 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f47140bc640 1 Processor -- start 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f47140bc640 1 -- start start 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f47140bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f470c100620 0x7f470c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f47140bc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f470c100fe0 0x7f470c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f47140bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c106630 0x7f470c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f47140bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f470c110a40 con 0x7f470c106630 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f47140bc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1108c0 con 0x7f470c100fe0 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f47140bc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f470c110bc0 con 0x7f470c100620 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c106630 0x7f470c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c106630 0x7f470c19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50196/0 (socket says 172.21.15.27:50196) 2024-04-25T01:31:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 -- 172.21.15.27:0/798142621 learned_addr learned my addr 172.21.15.27:0/798142621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4711e58640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f470c100620 0x7f470c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4711657640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f470c100fe0 0x7f470c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 -- 172.21.15.27:0/798142621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f470c100620 msgr2=0x7f470c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f470c100620 0x7f470c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 -- 172.21.15.27:0/798142621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f470c100fe0 msgr2=0x7f470c1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f470c100fe0 0x7f470c1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 -- 172.21.15.27:0/798142621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c19cf50 con 0x7f470c106630 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4711657640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f470c100fe0 0x7f470c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.054+0000 7f4712659640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c106630 0x7f470c19c820 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f4704005cc0 tx=0x7f4704003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.055+0000 7f4702ffd640 1 -- 172.21.15.27:0/798142621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704004040 con 0x7f470c106630 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.055+0000 7f4702ffd640 1 -- 172.21.15.27:0/798142621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47040046c0 con 0x7f470c106630 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.055+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c19d1e0 con 0x7f470c106630 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.055+0000 7f4702ffd640 1 -- 172.21.15.27:0/798142621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704046480 con 0x7f470c106630 2024-04-25T01:31:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.055+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c1a4ad0 con 0x7f470c106630 2024-04-25T01:31:59.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.056+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c068560 con 0x7f470c106630 2024-04-25T01:31:59.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.057+0000 7f4702ffd640 1 -- 172.21.15.27:0/798142621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f47040305a0 con 0x7f470c106630 2024-04-25T01:31:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.058+0000 7f4702ffd640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f46f80a2d00 0x7f46f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.058+0000 7f4711e58640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f46f80a2d00 0x7f46f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.058+0000 7f4702ffd640 1 -- 172.21.15.27:0/798142621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f470402e030 con 0x7f470c106630 2024-04-25T01:31:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.058+0000 7f4711e58640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f46f80a2d00 0x7f46f80a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4708005fd0 tx=0x7f470800b510 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:58 smithi139 ceph-mon[32760]: pgmap v199: 129 pgs: 129 active+clean; 3.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-04-25T01:31:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.062+0000 7f4702ffd640 1 -- 172.21.15.27:0/798142621 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f470404b060 con 0x7f470c106630 2024-04-25T01:31:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.286+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f470c109c00 con 0x7f470c106630 2024-04-25T01:31:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.287+0000 7f4702ffd640 1 -- 172.21.15.27:0/798142621 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7f47040bd960 con 0x7f470c106630 2024-04-25T01:31:59.265 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:31:59.265 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:31:59.265 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f46f80a2d00 msgr2=0x7f46f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f46f80a2d00 0x7f46f80a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4708005fd0 tx=0x7f470800b510 comp rx=0 tx=0).stop 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c106630 msgr2=0x7f470c19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c106630 0x7f470c19c820 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f4704005cc0 tx=0x7f4704003970 comp rx=0 tx=0).stop 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 shutdown_connections 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f46f80a2d00 0x7f46f80a51c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c106630 0x7f470c19c820 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f470c100fe0 0x7f470c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 --2- 172.21.15.27:0/798142621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f470c100620 0x7f470c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 >> 172.21.15.27:0/798142621 conn(0x7f470c077740 msgr2=0x7f470c068b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 shutdown_connections 2024-04-25T01:31:59.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.289+0000 7f47140bc640 1 -- 172.21.15.27:0/798142621 wait complete. 2024-04-25T01:31:59.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:31:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/3370810144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74101710 msgr2=0x7f6c74101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/3370810144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74101710 0x7f6c74101b90 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009930 tx=0x7f6c6402a420 comp rx=0 tx=0).stop 2024-04-25T01:31:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/3370810144 shutdown_connections 2024-04-25T01:31:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/3370810144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c741020d0 0x7f6c7410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/3370810144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74101710 0x7f6c74101b90 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/3370810144 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c74107720 0x7f6c74107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/3370810144 >> 172.21.15.27:0/3370810144 conn(0x7f6c740fd180 msgr2=0x7f6c740ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/3370810144 shutdown_connections 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.403+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/3370810144 wait complete. 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7c0b2640 1 Processor -- start 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7c0b2640 1 -- start start 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7c0b2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c74101710 0x7f6c7419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7c0b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c741020d0 0x7f6c7419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7c0b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74107720 0x7f6c7419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7c0b2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74111b30 con 0x7f6c74107720 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7c0b2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741119b0 con 0x7f6c74101710 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7c0b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74111cb0 con 0x7f6c741020d0 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7a64f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74107720 0x7f6c7419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7a64f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74107720 0x7f6c7419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50208/0 (socket says 172.21.15.27:50208) 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7a64f640 1 -- 172.21.15.27:0/1860986719 learned_addr learned my addr 172.21.15.27:0/1860986719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c74101710 0x7f6c7419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.404+0000 7f6c7964d640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c741020d0 0x7f6c7419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c7a64f640 1 -- 172.21.15.27:0/1860986719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c741020d0 msgr2=0x7f6c7419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c7a64f640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c741020d0 0x7f6c7419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c7a64f640 1 -- 172.21.15.27:0/1860986719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c74101710 msgr2=0x7f6c7419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c7a64f640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c74101710 0x7f6c7419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c7a64f640 1 -- 172.21.15.27:0/1860986719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7419f280 con 0x7f6c74107720 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c7a64f640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74107720 0x7f6c7419eb50 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00eb10 tx=0x7f6c6c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c6affd640 1 -- 172.21.15.27:0/1860986719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c004b20 con 0x7f6c74107720 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/1860986719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7419f570 con 0x7f6c74107720 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c6affd640 1 -- 172.21.15.27:0/1860986719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c004cc0 con 0x7f6c74107720 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c6affd640 1 -- 172.21.15.27:0/1860986719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c010760 con 0x7f6c74107720 2024-04-25T01:31:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.405+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/1860986719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c741a6e00 con 0x7f6c74107720 2024-04-25T01:31:59.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.407+0000 7f6c6affd640 1 -- 172.21.15.27:0/1860986719 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6c6c005020 con 0x7f6c74107720 2024-04-25T01:31:59.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.407+0000 7f6c7edbd740 1 Processor -- start 2024-04-25T01:31:59.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7edbd740 1 -- start start 2024-04-25T01:31:59.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c187722960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c187687670 con 0x55c187722ea0 2024-04-25T01:31:59.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c18762f970 con 0x55c187722580 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c187683630 con 0x55c18772d570 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c79e4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c79e4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50224/0 (socket says 172.21.15.27:50224) 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c79e4e640 1 -- 172.21.15.27:0/177373789 learned_addr learned my addr 172.21.15.27:0/177373789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c6affd640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6c600a2d00 0x7f6c600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7964d640 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c187722960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7a64f640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6c600a2d00 0x7f6c600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c79e4e640 1 -- 172.21.15.27:0/177373789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 msgr2=0x55c18772f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7964d640 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c79e4e640 1 -- 172.21.15.27:0/177373789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 msgr2=0x55c187722960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c187722960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c79e4e640 1 -- 172.21.15.27:0/177373789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c187653400 con 0x55c187722ea0 2024-04-25T01:31:59.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c6affd640 1 -- 172.21.15.27:0/1860986719 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6c6c0cfec0 con 0x7f6c74107720 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.408+0000 7f6c7964d640 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.409+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772d030 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000bc90 tx=0x7f6c700107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ed4c145088ef1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.409+0000 7f6c5ffff640 1 -- 172.21.15.27:0/177373789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7001c480 con 0x55c187722ea0 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.409+0000 7f6c7a64f640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6c600a2d00 0x7f6c600a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c09bd50 tx=0x7f6c6c06e010 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.409+0000 7f6c5ffff640 1 -- 172.21.15.27:0/177373789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c7001cb00 con 0x55c187722ea0 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.409+0000 7f6c5ffff640 1 -- 172.21.15.27:0/177373789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c70011bb0 con 0x55c187722ea0 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.409+0000 7f6c7edbd740 1 -- 172.21.15.27:0/177373789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 msgr2=0x55c18772d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.409+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772d030 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000bc90 tx=0x7f6c700107a0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 -- 172.21.15.27:0/177373789 shutdown_connections 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772d030 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/177373789 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c187722960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 -- 172.21.15.27:0/177373789 >> 172.21.15.27:0/177373789 conn(0x55c18771e8a0 msgr2=0x55c18771ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 -- 172.21.15.27:0/177373789 shutdown_connections 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 -- 172.21.15.27:0/177373789 wait complete. 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 -- start start 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 Processor -- start 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c18772cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.410+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c187730360 con 0x55c187722ea0 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c187643050 con 0x55c187722580 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c1877304e0 con 0x55c18772d570 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7edbd740 10 client.0 _ll_register_callbacks cb 0x55c18739abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50228/0 (socket says 172.21.15.27:50228) 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 -- 172.21.15.27:0/2194925376 learned_addr learned my addr 172.21.15.27:0/2194925376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7964d640 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c18772cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7964d640 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 -- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 msgr2=0x55c18772a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 -- 172.21.15.27:0/2194925376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 msgr2=0x55c18772cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c18772cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 -- 172.21.15.27:0/2194925376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c187614dc0 con 0x55c187722ea0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7964d640 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772a1f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6c70010ca0 tx=0x7f6c7001c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a492adc6a53946 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/2194925376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7001c950 con 0x55c187722ea0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/2194925376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6c70011b40 con 0x55c187722ea0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7edbd740 1 -- 172.21.15.27:0/2194925376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c7001cb00 con 0x55c187722ea0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/2194925376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6c70011d40 con 0x55c187722ea0 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7edbd740 10 client.17037 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.411+0000 7f6c7edbd740 1 -- 172.21.15.27:0/2194925376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c18739b3f0 con 0x55c187722ea0 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/2194925376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c7001cb00 con 0x55c187722ea0 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/2194925376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6c700046c0 con 0x55c187722ea0 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c7edbd740 10 client.17037 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c7edbd740 10 client.17037 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c7edbd740 20 client.17037 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c18773d440 0x55c18773f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c7edbd740 4 client.17037 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c7edbd740 1 -- 172.21.15.27:0/2194925376 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c187694bb0 con 0x55c18773d440 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.412+0000 7f6c7964d640 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c18773d440 0x55c18773f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.413+0000 7f6c7964d640 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c18773d440 0x55c18773f830 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ea791c26c06d9946 server_cookie=b30c7e85a65da7f0 in_seq=0 out_seq=0 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.413+0000 7f6c5d7fa640 10 client.17037 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.413+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/2194925376 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c187694bb0 con 0x55c18773d440 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.413+0000 7f6c5d7fa640 10 client.17037 handle_command_reply: tid=0 2024-04-25T01:31:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.413+0000 7f6c7edbd740 1 client.17037 shutdown 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.413+0000 7f6c7edbd740 1 -- 172.21.15.27:0/2194925376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 msgr2=0x55c18772a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.413+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772a1f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6c70010ca0 tx=0x7f6c7001c730 comp rx=0 tx=0).stop 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 1 -- 172.21.15.27:0/2194925376 shutdown_connections 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c18773d440 0x55c18773f830 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c18772d570 0x55c18772a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c187722ea0 0x55c18772a1f0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c5d7fa640 0 client.17037 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/2194925376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c18772cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 1 -- 172.21.15.27:0/2194925376 >> 172.21.15.27:0/2194925376 conn(0x55c18771e8a0 msgr2=0x55c18771db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 1 -- 172.21.15.27:0/2194925376 shutdown_connections 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 1 -- 172.21.15.27:0/2194925376 wait complete. 2024-04-25T01:31:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.414+0000 7f6c7edbd740 20 client.17037 trim_cache size 0 max 16384 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7edbd740 1 Processor -- start 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7edbd740 1 -- start start 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c187739220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722ea0 0x55c187744ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c187745530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741b3850 con 0x55c18772d570 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741b1120 con 0x55c187722580 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741aef70 con 0x55c187722ea0 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7964d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c187745530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7964d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c187745530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50240/0 (socket says 172.21.15.27:50240) 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.427+0000 7f6c7964d640 1 -- 172.21.15.27:0/3370936060 learned_addr learned my addr 172.21.15.27:0/3370936060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722ea0 0x55c187744ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c7964d640 1 -- 172.21.15.27:0/3370936060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722ea0 msgr2=0x55c187744ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c7964d640 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722ea0 0x55c187744ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c7964d640 1 -- 172.21.15.27:0/3370936060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 msgr2=0x55c187739220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c7964d640 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c187739220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c7964d640 1 -- 172.21.15.27:0/3370936060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1874657d0 con 0x55c18772d570 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722ea0 0x55c187744ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c7964d640 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c187745530 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f6c6402a940 tx=0x7f6c64040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b06d8cfb10dfa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c5f7fe640 1 -- 172.21.15.27:0/3370936060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c64035040 con 0x55c18772d570 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c5f7fe640 1 -- 172.21.15.27:0/3370936060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c64005020 con 0x55c18772d570 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.428+0000 7f6c5f7fe640 1 -- 172.21.15.27:0/3370936060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c64004b20 con 0x55c18772d570 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- 172.21.15.27:0/3370936060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 msgr2=0x55c187745530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c187745530 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f6c6402a940 tx=0x7f6c64040040 comp rx=0 tx=0).stop 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- 172.21.15.27:0/3370936060 shutdown_connections 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c187745530 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722ea0 0x55c187744ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/3370936060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722580 0x55c187739220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- 172.21.15.27:0/3370936060 >> 172.21.15.27:0/3370936060 conn(0x55c18771e8a0 msgr2=0x55c187725740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- 172.21.15.27:0/3370936060 shutdown_connections 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- 172.21.15.27:0/3370936060 wait complete. 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- start start 2024-04-25T01:31:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 Processor -- start 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722580 0x55c187735900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722ea0 0x55c187730c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c1877311b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741af5e0 con 0x55c18772d570 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741b0a50 con 0x55c187722ea0 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.429+0000 7f6c7edbd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741afcb0 con 0x55c187722580 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7edbd740 10 client.0 _ll_register_callbacks cb 0x7f6c64009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c1877311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c1877311b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50256/0 (socket says 172.21.15.27:50256) 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 -- 172.21.15.27:0/293520973 learned_addr learned my addr 172.21.15.27:0/293520973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722580 0x55c187735900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722ea0 0x55c187730c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 -- 172.21.15.27:0/293520973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722580 msgr2=0x55c187735900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722580 0x55c187735900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 -- 172.21.15.27:0/293520973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722ea0 msgr2=0x55c187730c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722ea0 0x55c187730c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 -- 172.21.15.27:0/293520973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1877317a0 con 0x55c18772d570 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7964d640 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c1877311b0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6c64001eb0 tx=0x7f6c64009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206fb8500b9cdbb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/293520973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c64035070 con 0x55c18772d570 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/293520973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6c6404b3f0 con 0x55c18772d570 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7edbd740 1 -- 172.21.15.27:0/293520973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c64005020 con 0x55c18772d570 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/293520973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6c6404b5f0 con 0x55c18772d570 2024-04-25T01:31:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7edbd740 10 client.17043 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.430+0000 7f6c7edbd740 1 -- 172.21.15.27:0/293520973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c1877a35d0 con 0x55c18772d570 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/293520973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c64005020 con 0x55c18772d570 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/293520973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6c6403d5f0 con 0x55c18772d570 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c7edbd740 10 client.17043 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c7edbd740 10 client.17043 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c7edbd740 20 client.17043 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c18773d440 0x55c1877a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c7edbd740 4 client.17043 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c7edbd740 1 -- 172.21.15.27:0/293520973 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c18768ef00 con 0x55c18773d440 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c18773d440 0x55c1877a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c5d7fa640 10 client.17043 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.431+0000 7f6c79e4e640 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c18773d440 0x55c1877a5d30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac655a55b2889b3d server_cookie=b630711573e178c9 in_seq=0 out_seq=0 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c5d7fa640 1 -- 172.21.15.27:0/293520973 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c18768ef00 con 0x55c18773d440 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c5d7fa640 10 client.17043 handle_command_reply: tid=0 2024-04-25T01:31:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 client.17043 shutdown 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 -- 172.21.15.27:0/293520973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 msgr2=0x55c1877311b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c1877311b0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6c64001eb0 tx=0x7f6c64009260 comp rx=0 tx=0).stop 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 -- 172.21.15.27:0/293520973 shutdown_connections 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55c18773d440 0x55c1877a5d30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c18772d570 0x55c1877311b0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c187722ea0 0x55c187730c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c5d7fa640 0 client.17043 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 --2- 172.21.15.27:0/293520973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c187722580 0x55c187735900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 -- 172.21.15.27:0/293520973 >> 172.21.15.27:0/293520973 conn(0x55c18771e8a0 msgr2=0x55c187738210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 -- 172.21.15.27:0/293520973 shutdown_connections 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 1 -- 172.21.15.27:0/293520973 wait complete. 2024-04-25T01:31:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7edbd740 20 client.17043 trim_cache size 0 max 16384 2024-04-25T01:31:59.417 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:31:59.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6c600a2d00 msgr2=0x7f6c600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6c600a2d00 0x7f6c600a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c09bd50 tx=0x7f6c6c06e010 comp rx=0 tx=0).stop 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74107720 msgr2=0x7f6c7419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.432+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74107720 0x7f6c7419eb50 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00eb10 tx=0x7f6c6c00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.433+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/1860986719 shutdown_connections 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.433+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6c600a2d00 0x7f6c600a51c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.433+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c74107720 0x7f6c7419eb50 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.433+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c741020d0 0x7f6c7419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.433+0000 7f6c7c0b2640 1 --2- 172.21.15.27:0/1860986719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c74101710 0x7f6c7419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.433+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/1860986719 >> 172.21.15.27:0/1860986719 conn(0x7f6c740fd180 msgr2=0x7f6c7410ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.433+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/1860986719 shutdown_connections 2024-04-25T01:31:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.433+0000 7f6c7c0b2640 1 -- 172.21.15.27:0/1860986719 wait complete. 2024-04-25T01:31:59.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:31:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1477452160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c40107730 msgr2=0x7f3c40107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1477452160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c40107730 0x7f3c40107b10 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009930 tx=0x7f3c3002a420 comp rx=0 tx=0).stop 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1477452160 shutdown_connections 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1477452160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c4006a780 0x7f3c4010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1477452160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c40069dc0 0x7f3c4006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1477452160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c40107730 0x7f3c40107b10 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1477452160 >> 172.21.15.27:0/1477452160 conn(0x7f3c400776a0 msgr2=0x7f3c40077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1477452160 shutdown_connections 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.546+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1477452160 wait complete. 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c45ebc640 1 Processor -- start 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c45ebc640 1 -- start start 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c45ebc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c40069dc0 0x7f3c401980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c45ebc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4006a780 0x7f3c401985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c45ebc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c40107730 0x7f3c4019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c45ebc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c40111b40 con 0x7f3c4006a780 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c45ebc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3c401119c0 con 0x7f3c40107730 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c45ebc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c40111cc0 con 0x7f3c40069dc0 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4006a780 0x7f3c401985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4006a780 0x7f3c401985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50272/0 (socket says 172.21.15.27:50272) 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.547+0000 7f3c3effd640 1 -- 172.21.15.27:0/1063517233 learned_addr learned my addr 172.21.15.27:0/1063517233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c40069dc0 0x7f3c401980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c40107730 0x7f3c4019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3effd640 1 -- 172.21.15.27:0/1063517233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c40069dc0 msgr2=0x7f3c401980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3effd640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c40069dc0 0x7f3c401980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3effd640 1 -- 172.21.15.27:0/1063517233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c40107730 msgr2=0x7f3c4019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3effd640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c40107730 0x7f3c4019c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3effd640 1 -- 172.21.15.27:0/1063517233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4019d090 con 0x7f3c4006a780 2024-04-25T01:31:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3effd640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4006a780 0x7f3c401985e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400e9f0 tx=0x7f3c3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3cff9640 1 -- 172.21.15.27:0/1063517233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3400cd80 con 0x7f3c4006a780 2024-04-25T01:31:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1063517233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c4019d380 con 0x7f3c4006a780 2024-04-25T01:31:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3cff9640 1 -- 172.21.15.27:0/1063517233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c34005500 con 0x7f3c4006a780 2024-04-25T01:31:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c3cff9640 1 -- 172.21.15.27:0/1063517233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34010690 con 0x7f3c4006a780 2024-04-25T01:31:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.548+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1063517233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c401a4c10 con 0x7f3c4006a780 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.550+0000 7f3c48bc5740 1 Processor -- start 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.550+0000 7f3c3cff9640 1 -- 172.21.15.27:0/1063517233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3c340046c0 con 0x7f3c4006a780 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c48bc5740 1 -- start start 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26580 0x55fcaac26960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26ea0 0x55fcaac31030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac33960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fcaab8b670 con 0x55fcaac26580 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fcaab33970 con 0x55fcaac31570 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fcaab87630 con 0x55fcaac26ea0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26580 0x55fcaac26960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26580 0x55fcaac26960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50284/0 (socket says 172.21.15.27:50284) 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 -- 172.21.15.27:0/2365927651 learned_addr learned my addr 172.21.15.27:0/2365927651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3effd640 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac33960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26ea0 0x55fcaac31030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 -- 172.21.15.27:0/2365927651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26ea0 msgr2=0x55fcaac31030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26ea0 0x55fcaac31030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 -- 172.21.15.27:0/2365927651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 msgr2=0x55fcaac33960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac33960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 -- 172.21.15.27:0/2365927651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcaab57400 con 0x55fcaac26580 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3effd640 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac33960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3cff9640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c280a2d00 0x7f3c280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26580 0x55fcaac26960 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009e60 tx=0x7f3c38010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35d873f15421091 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.551+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c280a2d00 0x7f3c280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c3cff9640 1 -- 172.21.15.27:0/1063517233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3c340cfcb0 con 0x7f3c4006a780 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c25ffb640 1 -- 172.21.15.27:0/2365927651 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38016070 con 0x55fcaac26580 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c25ffb640 1 -- 172.21.15.27:0/2365927651 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c38004d20 con 0x55fcaac26580 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c25ffb640 1 -- 172.21.15.27:0/2365927651 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38012810 con 0x55fcaac26580 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2365927651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26580 msgr2=0x55fcaac26960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26580 0x55fcaac26960 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009e60 tx=0x7f3c38010740 comp rx=0 tx=0).stop 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c280a2d00 0x7f3c280a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009900 tx=0x7f3c300058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2365927651 shutdown_connections 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac33960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26ea0 0x55fcaac31030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/2365927651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26580 0x55fcaac26960 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2365927651 >> 172.21.15.27:0/2365927651 conn(0x55fcaac228a0 msgr2=0x55fcaac22c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2365927651 shutdown_connections 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.552+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2365927651 wait complete. 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 1 -- start start 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 1 Processor -- start 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac30f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac2e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac2e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fcaac34360 con 0x55fcaac26ea0 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fcaab47050 con 0x55fcaac31570 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fcaac344e0 con 0x55fcaac26580 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c48bc5740 10 client.0 _ll_register_callbacks cb 0x55fcaa89ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac2e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac2e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50292/0 (socket says 172.21.15.27:50292) 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c3effd640 1 -- 172.21.15.27:0/3677210519 learned_addr learned my addr 172.21.15.27:0/3677210519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.553+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac2e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac30f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3effd640 1 -- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 msgr2=0x55fcaac30f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3effd640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac30f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3effd640 1 -- 172.21.15.27:0/3677210519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 msgr2=0x55fcaac2e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3effd640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac2e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3effd640 1 -- 172.21.15.27:0/3677210519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcaab18dc0 con 0x55fcaac26ea0 2024-04-25T01:31:59.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac2e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac30f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c3effd640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac2e210 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3c3409bb40 tx=0x7f3c340980e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e1c0f99905c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c1effd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c1effd640 1 -- 172.21.15.27:0/3677210519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3409e070 con 0x55fcaac26ea0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c1effd640 1 -- 172.21.15.27:0/3677210519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3c3409c2a0 con 0x55fcaac26ea0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 1 -- 172.21.15.27:0/3677210519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c38004d20 con 0x55fcaac26ea0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c1effd640 1 -- 172.21.15.27:0/3677210519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c3409b7a0 con 0x55fcaac26ea0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 10 client.17055 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 1 -- 172.21.15.27:0/3677210519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fcaa89f3f0 con 0x55fcaac26ea0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c1effd640 1 -- 172.21.15.27:0/3677210519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c38004d20 con 0x55fcaac26ea0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c1effd640 1 -- 172.21.15.27:0/3677210519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3c340a4620 con 0x55fcaac26ea0 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 10 client.17055 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:59.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 10 client.17055 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 20 client.17055 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55fcaac41b70 0x55fcaac43ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 4 client.17055 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.554+0000 7f3c48bc5740 1 -- 172.21.15.27:0/3677210519 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcaab98bb0 con 0x55fcaac41b70 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.555+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55fcaac41b70 0x55fcaac43ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.555+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55fcaac41b70 0x55fcaac43ff0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8212a26b228204e5 server_cookie=756543a740e60595 in_seq=0 out_seq=0 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.555+0000 7f3c1effd640 10 client.17055 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c1effd640 1 -- 172.21.15.27:0/3677210519 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fcaab98bb0 con 0x55fcaac41b70 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c1effd640 10 client.17055 handle_command_reply: tid=0 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 client.17055 shutdown 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 -- 172.21.15.27:0/3677210519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 msgr2=0x55fcaac2e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac2e210 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3c3409bb40 tx=0x7f3c340980e0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 -- 172.21.15.27:0/3677210519 shutdown_connections 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55fcaac41b70 0x55fcaac43ff0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac2e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c1effd640 0 client.17055 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:31:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac2e210 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/3677210519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac30f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 -- 172.21.15.27:0/3677210519 >> 172.21.15.27:0/3677210519 conn(0x55fcaac228a0 msgr2=0x55fcaac286a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 -- 172.21.15.27:0/3677210519 shutdown_connections 2024-04-25T01:31:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 1 -- 172.21.15.27:0/3677210519 wait complete. 2024-04-25T01:31:59.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.556+0000 7f3c48bc5740 20 client.17055 trim_cache size 0 max 16384 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.569+0000 7f3c48bc5740 1 Processor -- start 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c48bc5740 1 -- start start 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac3d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac39b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac30a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c401b13f0 con 0x55fcaac26ea0 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3c401aecc0 con 0x55fcaac31570 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c401acb10 con 0x55fcaac26580 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac39b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac39b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50304/0 (socket says 172.21.15.27:50304) 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 -- 172.21.15.27:0/2073479361 learned_addr learned my addr 172.21.15.27:0/2073479361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac30a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac3d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 -- 172.21.15.27:0/2073479361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 msgr2=0x55fcaac3d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac3d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 -- 172.21.15.27:0/2073479361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 msgr2=0x55fcaac30a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac30a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 -- 172.21.15.27:0/2073479361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcaac2eb20 con 0x55fcaac26ea0 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac30a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c3effd640 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac39b10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c6a0 tx=0x7f3c3409b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531b22496864e197 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c257fa640 1 -- 172.21.15.27:0/2073479361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3409e070 con 0x55fcaac26ea0 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c257fa640 1 -- 172.21.15.27:0/2073479361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3409c540 con 0x55fcaac26ea0 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.570+0000 7f3c257fa640 1 -- 172.21.15.27:0/2073479361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3409c300 con 0x55fcaac26ea0 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2073479361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 msgr2=0x55fcaac39b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac39b10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c6a0 tx=0x7f3c3409b5c0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2073479361 shutdown_connections 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac31570 0x55fcaac30a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac39b10 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/2073479361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac26580 0x55fcaac3d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2073479361 >> 172.21.15.27:0/2073479361 conn(0x55fcaac228a0 msgr2=0x55fcaac28eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2073479361 shutdown_connections 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 -- 172.21.15.27:0/2073479361 wait complete. 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.571+0000 7f3c48bc5740 1 -- start start 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c48bc5740 1 Processor -- start 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac26580 0x55fcaac2bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac49060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c48bc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac31570 0x55fcaac495a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c401ad180 con 0x55fcaac26ea0 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3c401ae5f0 con 0x55fcaac26580 2024-04-25T01:31:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c48bc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c401ad850 con 0x55fcaac31570 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c48bc5740 10 client.0 _ll_register_callbacks cb 0x7f3c3400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac49060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac49060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50318/0 (socket says 172.21.15.27:50318) 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3effd640 1 -- 172.21.15.27:0/690411853 learned_addr learned my addr 172.21.15.27:0/690411853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac26580 0x55fcaac2bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3ffff640 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac31570 0x55fcaac495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3effd640 1 -- 172.21.15.27:0/690411853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac31570 msgr2=0x55fcaac495a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3effd640 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac31570 0x55fcaac495a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3effd640 1 -- 172.21.15.27:0/690411853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac26580 msgr2=0x55fcaac2bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3effd640 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac26580 0x55fcaac2bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.572+0000 7f3c3effd640 1 -- 172.21.15.27:0/690411853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcaa9697d0 con 0x55fcaac26ea0 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c3effd640 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac49060 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3c340a49c0 tx=0x7f3c340980e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5e78c91720810c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c1effd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c1effd640 1 -- 172.21.15.27:0/690411853 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3409e070 con 0x55fcaac26ea0 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c1effd640 1 -- 172.21.15.27:0/690411853 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3c34097b90 con 0x55fcaac26ea0 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 1 -- 172.21.15.27:0/690411853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c3409c540 con 0x55fcaac26ea0 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c1effd640 1 -- 172.21.15.27:0/690411853 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c3409bc10 con 0x55fcaac26ea0 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 10 client.17061 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 1 -- 172.21.15.27:0/690411853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fcaa9697d0 con 0x55fcaac26ea0 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c1effd640 1 -- 172.21.15.27:0/690411853 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3409c540 con 0x55fcaac26ea0 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c1effd640 1 -- 172.21.15.27:0/690411853 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3c340baa80 con 0x55fcaac26ea0 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 10 client.17061 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 10 client.17061 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 20 client.17061 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55fcaacac340 0x55fcaacae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 4 client.17061 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c48bc5740 1 -- 172.21.15.27:0/690411853 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcaab92f00 con 0x55fcaacac340 2024-04-25T01:31:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.573+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55fcaacac340 0x55fcaacae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c3f7fe640 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55fcaacac340 0x55fcaacae730 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7df30a0b46669343 server_cookie=5ae04756095704ac in_seq=0 out_seq=0 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c1effd640 10 client.17061 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c1effd640 1 -- 172.21.15.27:0/690411853 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcaab92f00 con 0x55fcaacac340 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c1effd640 10 client.17061 handle_command_reply: tid=0 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 client.17061 shutdown 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 -- 172.21.15.27:0/690411853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 msgr2=0x55fcaac49060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac49060 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3c340a49c0 tx=0x7f3c340980e0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 -- 172.21.15.27:0/690411853 shutdown_connections 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55fcaacac340 0x55fcaacae730 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcaac31570 0x55fcaac495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fcaac26ea0 0x55fcaac49060 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c1effd640 0 client.17061 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 --2- 172.21.15.27:0/690411853 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fcaac26580 0x55fcaac2bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 -- 172.21.15.27:0/690411853 >> 172.21.15.27:0/690411853 conn(0x55fcaac228a0 msgr2=0x55fcaac28560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 -- 172.21.15.27:0/690411853 shutdown_connections 2024-04-25T01:31:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 1 -- 172.21.15.27:0/690411853 wait complete. 2024-04-25T01:31:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.574+0000 7f3c48bc5740 20 client.17061 trim_cache size 0 max 16384 2024-04-25T01:31:59.559 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c280a2d00 msgr2=0x7f3c280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c280a2d00 0x7f3c280a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009900 tx=0x7f3c300058d0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4006a780 msgr2=0x7f3c401985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4006a780 0x7f3c401985e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400e9f0 tx=0x7f3c3400edf0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1063517233 shutdown_connections 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c40107730 0x7f3c4019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c280a2d00 0x7f3c280a51c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4006a780 0x7f3c401985e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 --2- 172.21.15.27:0/1063517233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c40069dc0 0x7f3c401980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1063517233 >> 172.21.15.27:0/1063517233 conn(0x7f3c400776a0 msgr2=0x7f3c400fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1063517233 shutdown_connections 2024-04-25T01:31:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.575+0000 7f3c45ebc640 1 -- 172.21.15.27:0/1063517233 wait complete. 2024-04-25T01:31:59.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:31:59.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 -- 172.21.15.27:0/3620106527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 msgr2=0x7fe80010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 --2- 172.21.15.27:0/3620106527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 0x7fe80010e5e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009cd0 tx=0x7fe7f802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 -- 172.21.15.27:0/3620106527 shutdown_connections 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 --2- 172.21.15.27:0/3620106527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 0x7fe80010e5e0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 --2- 172.21.15.27:0/3620106527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe800101710 0x7fe800101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 --2- 172.21.15.27:0/3620106527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe800107720 0x7fe800107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 -- 172.21.15.27:0/3620106527 >> 172.21.15.27:0/3620106527 conn(0x7fe8000fd180 msgr2=0x7fe8000ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 -- 172.21.15.27:0/3620106527 shutdown_connections 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.690+0000 7fe8068d4640 1 -- 172.21.15.27:0/3620106527 wait complete. 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe8068d4640 1 Processor -- start 2024-04-25T01:31:59.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe8068d4640 1 -- start start 2024-04-25T01:31:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe8068d4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe800101710 0x7fe8001980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe8068d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 0x7fe800198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe8068d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe800107720 0x7fe80019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe8068d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe800111b80 con 0x7fe8001020d0 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe8068d4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe800111a00 con 0x7fe800101710 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe8068d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe800111d00 con 0x7fe800107720 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 0x7fe800198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 0x7fe800198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50342/0 (socket says 172.21.15.27:50342) 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7ff7fe640 1 -- 172.21.15.27:0/1389237006 learned_addr learned my addr 172.21.15.27:0/1389237006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7fffff640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe800101710 0x7fe8001980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe804e71640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe800107720 0x7fe80019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7ff7fe640 1 -- 172.21.15.27:0/1389237006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe800107720 msgr2=0x7fe80019c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe800107720 0x7fe80019c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7ff7fe640 1 -- 172.21.15.27:0/1389237006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe800101710 msgr2=0x7fe8001980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe800101710 0x7fe8001980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.691+0000 7fe7ff7fe640 1 -- 172.21.15.27:0/1389237006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80019d0d0 con 0x7fe8001020d0 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.692+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 0x7fe800198620 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400b4a0 tx=0x7fe7f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.692+0000 7fe7fd7fa640 1 -- 172.21.15.27:0/1389237006 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4014070 con 0x7fe8001020d0 2024-04-25T01:31:59.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.692+0000 7fe7fd7fa640 1 -- 172.21.15.27:0/1389237006 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f4010af0 con 0x7fe8001020d0 2024-04-25T01:31:59.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.692+0000 7fe7fd7fa640 1 -- 172.21.15.27:0/1389237006 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4017d10 con 0x7fe8001020d0 2024-04-25T01:31:59.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.692+0000 7fe8068d4640 1 -- 172.21.15.27:0/1389237006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe80019d3c0 con 0x7fe8001020d0 2024-04-25T01:31:59.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.692+0000 7fe8068d4640 1 -- 172.21.15.27:0/1389237006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8001a4c50 con 0x7fe8001020d0 2024-04-25T01:31:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.694+0000 7fe7fd7fa640 1 -- 172.21.15.27:0/1389237006 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe7f4017070 con 0x7fe8001020d0 2024-04-25T01:31:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.694+0000 7fe8095e3740 1 Processor -- start 2024-04-25T01:31:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe8095e3740 1 -- start start 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84eee860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eeeda0 0x55fb84efd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84efd980 0x55fb84effd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe8095e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fb84e53670 con 0x55fb84eee480 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe8095e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fb84dfb970 con 0x55fb84eeeda0 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe8095e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fb84e4f630 con 0x55fb84efd980 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84eee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84eee860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50352/0 (socket says 172.21.15.27:50352) 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 -- 172.21.15.27:0/763145733 learned_addr learned my addr 172.21.15.27:0/763145733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe7fd7fa640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7e80a2d00 0x7fe7e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe7fffff640 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eeeda0 0x55fb84efd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84efd980 0x55fb84effd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 -- 172.21.15.27:0/763145733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84efd980 msgr2=0x55fb84effd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84efd980 0x55fb84effd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 -- 172.21.15.27:0/763145733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eeeda0 msgr2=0x55fb84efd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eeeda0 0x55fb84efd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 -- 172.21.15.27:0/763145733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb84e1f400 con 0x55fb84eee480 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe7fffff640 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eeeda0 0x55fb84efd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84efd980 0x55fb84effd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe7fd7fa640 1 -- 172.21.15.27:0/1389237006 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe7f40ceaa0 con 0x7fe8001020d0 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.695+0000 7fe804e71640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7e80a2d00 0x7fe7e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe804e71640 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84eee860 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009430 tx=0x7fe7f8031660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814c14954e88823d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe7e67fc640 1 -- 172.21.15.27:0/763145733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8031ca0 con 0x55fb84eee480 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe7e67fc640 1 -- 172.21.15.27:0/763145733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f8036040 con 0x55fb84eee480 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe7e67fc640 1 -- 172.21.15.27:0/763145733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80308c0 con 0x55fb84eee480 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe8095e3740 1 -- 172.21.15.27:0/763145733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 msgr2=0x55fb84eee860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe804e71640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7e80a2d00 0x7fe7e80a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe7f804b000 tx=0x7fe7f8048000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe8095e3740 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84eee860 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009430 tx=0x7fe7f8031660 comp rx=0 tx=0).stop 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe8095e3740 1 -- 172.21.15.27:0/763145733 shutdown_connections 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.696+0000 7fe8095e3740 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84efd980 0x55fb84effd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eeeda0 0x55fb84efd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 --2- 172.21.15.27:0/763145733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84eee860 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 -- 172.21.15.27:0/763145733 >> 172.21.15.27:0/763145733 conn(0x55fb84eea8a0 msgr2=0x55fb84eeac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 -- 172.21.15.27:0/763145733 shutdown_connections 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 -- 172.21.15.27:0/763145733 wait complete. 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 -- start start 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 Processor -- start 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eee480 0x55fb84efd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eeeda0 0x55fb84efa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 0x55fb84efaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fb84f00740 con 0x55fb84eeeda0 2024-04-25T01:31:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fb84e0f050 con 0x55fb84efd980 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fb84f008c0 con 0x55fb84eee480 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.697+0000 7fe8095e3740 10 client.0 _ll_register_callbacks cb 0x55fb84b66be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eeeda0 0x55fb84efa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eeeda0 0x55fb84efa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50354/0 (socket says 172.21.15.27:50354) 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 -- 172.21.15.27:0/2000591189 learned_addr learned my addr 172.21.15.27:0/2000591189 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe804e71640 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 0x55fb84efaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7fffff640 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eee480 0x55fb84efd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 -- 172.21.15.27:0/2000591189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eee480 msgr2=0x55fb84efd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eee480 0x55fb84efd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 -- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 msgr2=0x55fb84efaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 0x55fb84efaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 -- 172.21.15.27:0/2000591189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb84de0dc0 con 0x55fb84eeeda0 2024-04-25T01:31:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe804e71640 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 0x55fb84efaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eeeda0 0x55fb84efa5b0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40971a0 tx=0x7fe7f4069c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7507cd3c52490947 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7df7fe640 1 -- 172.21.15.27:0/2000591189 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f401e050 con 0x55fb84eeeda0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7df7fe640 1 -- 172.21.15.27:0/2000591189 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe7f40973b0 con 0x55fb84eeeda0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe8095e3740 1 -- 172.21.15.27:0/2000591189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f8036040 con 0x55fb84eeeda0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7df7fe640 1 -- 172.21.15.27:0/2000591189 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe7f409f0f0 con 0x55fb84eeeda0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe8095e3740 10 client.17073 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe8095e3740 1 -- 172.21.15.27:0/2000591189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fb84b673f0 con 0x55fb84eeeda0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.698+0000 7fe7df7fe640 1 -- 172.21.15.27:0/2000591189 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f8036040 con 0x55fb84eeeda0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.699+0000 7fe7df7fe640 1 -- 172.21.15.27:0/2000591189 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe7f409a220 con 0x55fb84eeeda0 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.699+0000 7fe8095e3740 10 client.17073 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.699+0000 7fe8095e3740 10 client.17073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.699+0000 7fe8095e3740 20 client.17073 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.699+0000 7fe8095e3740 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fb84f0d990 0x55fb84f0fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.699+0000 7fe8095e3740 4 client.17073 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.699+0000 7fe8095e3740 1 -- 172.21.15.27:0/2000591189 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb84e60bb0 con 0x55fb84f0d990 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.699+0000 7fe7fffff640 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fb84f0d990 0x55fb84f0fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.700+0000 7fe7fffff640 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fb84f0d990 0x55fb84f0fda0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a8856248f7f4992 server_cookie=a2aa10a58bb03462 in_seq=0 out_seq=0 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.700+0000 7fe7df7fe640 10 client.17073 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.700+0000 7fe7df7fe640 1 -- 172.21.15.27:0/2000591189 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb84e60bb0 con 0x55fb84f0d990 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.700+0000 7fe7df7fe640 10 client.17073 handle_command_reply: tid=0 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.700+0000 7fe8095e3740 1 client.17073 shutdown 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 -- 172.21.15.27:0/2000591189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eeeda0 msgr2=0x55fb84efa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eeeda0 0x55fb84efa5b0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40971a0 tx=0x7fe7f4069c70 comp rx=0 tx=0).stop 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 -- 172.21.15.27:0/2000591189 shutdown_connections 2024-04-25T01:31:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fb84f0d990 0x55fb84f0fda0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 0x55fb84efaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eeeda0 0x55fb84efa5b0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe7df7fe640 0 client.17073 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 --2- 172.21.15.27:0/2000591189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eee480 0x55fb84efd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 -- 172.21.15.27:0/2000591189 >> 172.21.15.27:0/2000591189 conn(0x55fb84eea8a0 msgr2=0x55fb84ee9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 -- 172.21.15.27:0/2000591189 shutdown_connections 2024-04-25T01:31:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 1 -- 172.21.15.27:0/2000591189 wait complete. 2024-04-25T01:31:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.701+0000 7fe8095e3740 20 client.17073 trim_cache size 0 max 16384 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe8095e3740 1 Processor -- start 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe8095e3740 1 -- start start 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eee480 0x55fb84efcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eeeda0 0x55fb84f15670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84efd980 0x55fb84f15bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe8095e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001af2a0 con 0x55fb84efd980 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe8095e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001acb70 con 0x55fb84eee480 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe8095e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001aa9c0 con 0x55fb84eeeda0 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe804e71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84efd980 0x55fb84f15bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe804e71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84efd980 0x55fb84f15bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50358/0 (socket says 172.21.15.27:50358) 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe804e71640 1 -- 172.21.15.27:0/931318500 learned_addr learned my addr 172.21.15.27:0/931318500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe804e71640 1 -- 172.21.15.27:0/931318500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eeeda0 msgr2=0x55fb84f15670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe804e71640 1 --2- 172.21.15.27:0/931318500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eeeda0 0x55fb84f15670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe804e71640 1 -- 172.21.15.27:0/931318500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eee480 msgr2=0x55fb84efcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe804e71640 1 --2- 172.21.15.27:0/931318500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eee480 0x55fb84efcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.714+0000 7fe804e71640 1 -- 172.21.15.27:0/931318500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb84c317d0 con 0x55fb84efd980 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe804e71640 1 --2- 172.21.15.27:0/931318500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84efd980 0x55fb84f15bb0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe7f802ae00 tx=0x7fe7f803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dc85354be936f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe7e5ffb640 1 -- 172.21.15.27:0/931318500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8031ca0 con 0x55fb84efd980 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe7e5ffb640 1 -- 172.21.15.27:0/931318500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f8036040 con 0x55fb84efd980 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe7e5ffb640 1 -- 172.21.15.27:0/931318500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8003f00 con 0x55fb84efd980 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 -- 172.21.15.27:0/931318500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84efd980 msgr2=0x55fb84f15bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 --2- 172.21.15.27:0/931318500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84efd980 0x55fb84f15bb0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe7f802ae00 tx=0x7fe7f803d000 comp rx=0 tx=0).stop 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 -- 172.21.15.27:0/931318500 shutdown_connections 2024-04-25T01:31:59.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 --2- 172.21.15.27:0/931318500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84efd980 0x55fb84f15bb0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 --2- 172.21.15.27:0/931318500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eeeda0 0x55fb84f15670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 --2- 172.21.15.27:0/931318500 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84eee480 0x55fb84efcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 -- 172.21.15.27:0/931318500 >> 172.21.15.27:0/931318500 conn(0x55fb84eea8a0 msgr2=0x55fb84eefd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 -- 172.21.15.27:0/931318500 shutdown_connections 2024-04-25T01:31:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.715+0000 7fe8095e3740 1 -- 172.21.15.27:0/931318500 wait complete. 2024-04-25T01:31:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 1 -- start start 2024-04-25T01:31:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 1 Processor -- start 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84f05b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eeeda0 0x55fb84f06090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 0x55fb84f02c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001ab030 con 0x55fb84eee480 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001ac4a0 con 0x55fb84efd980 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe8001ab700 con 0x55fb84eeeda0 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe8095e3740 10 client.0 _ll_register_callbacks cb 0x7fe7f4017630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe7fffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84f05b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe7fffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84f05b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50364/0 (socket says 172.21.15.27:50364) 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe7fffff640 1 -- 172.21.15.27:0/1032227620 learned_addr learned my addr 172.21.15.27:0/1032227620 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe7fffff640 1 -- 172.21.15.27:0/1032227620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eeeda0 msgr2=0x55fb84f06090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe7fffff640 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eeeda0 0x55fb84f06090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe7fffff640 1 -- 172.21.15.27:0/1032227620 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 msgr2=0x55fb84f02c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe7fffff640 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 0x55fb84f02c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.716+0000 7fe7fffff640 1 -- 172.21.15.27:0/1032227620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb84f03440 con 0x55fb84eee480 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe7fffff640 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84f05b50 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x55fb84f105e0 tx=0x7fe7f000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41e0bc5f0b87f2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe7df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe7df7fe640 1 -- 172.21.15.27:0/1032227620 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0020030 con 0x55fb84eee480 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe7df7fe640 1 -- 172.21.15.27:0/1032227620 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe7f001c630 con 0x55fb84eee480 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 1 -- 172.21.15.27:0/1032227620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f8036040 con 0x55fb84eee480 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe7df7fe640 1 -- 172.21.15.27:0/1032227620 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe7f002a050 con 0x55fb84eee480 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 10 client.17079 fetch_fsmap learned FSMap version 27 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 1 -- 172.21.15.27:0/1032227620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fb84f74b70 con 0x55fb84eee480 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe7df7fe640 1 -- 172.21.15.27:0/1032227620 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f8036040 con 0x55fb84eee480 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe7df7fe640 1 -- 172.21.15.27:0/1032227620 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe7f001b3a0 con 0x55fb84eee480 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 10 client.17079 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 20 client.17079 populate_metadata read hostname 'smithi027' 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fb84f752f0 0x55fb84f77750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 4 client.17079 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.717+0000 7fe8095e3740 1 -- 172.21.15.27:0/1032227620 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fb84e5af00 con 0x55fb84f752f0 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fb84f752f0 0x55fb84f77750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe7ff7fe640 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fb84f752f0 0x55fb84f77750 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21fa17864c5504e2 server_cookie=196d78f8de8cbeb6 in_seq=0 out_seq=0 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe7df7fe640 10 client.17079 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe7df7fe640 1 -- 172.21.15.27:0/1032227620 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fb84e5af00 con 0x55fb84f752f0 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe7df7fe640 10 client.17079 handle_command_reply: tid=0 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 client.17079 shutdown 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 -- 172.21.15.27:0/1032227620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 msgr2=0x55fb84f05b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84f05b50 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x55fb84f105e0 tx=0x7fe7f000f7f0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 -- 172.21.15.27:0/1032227620 shutdown_connections 2024-04-25T01:31:59.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fb84f752f0 0x55fb84f77750 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fb84efd980 0x55fb84f02c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fb84eeeda0 0x55fb84f06090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 --2- 172.21.15.27:0/1032227620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fb84eee480 0x55fb84f05b50 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe7df7fe640 0 client.17079 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 -- 172.21.15.27:0/1032227620 >> 172.21.15.27:0/1032227620 conn(0x55fb84eea8a0 msgr2=0x55fb84ef0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 -- 172.21.15.27:0/1032227620 shutdown_connections 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.718+0000 7fe8095e3740 1 -- 172.21.15.27:0/1032227620 wait complete. 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8095e3740 20 client.17079 trim_cache size 0 max 16384 2024-04-25T01:31:59.705 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:31:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 -- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7e80a2d00 msgr2=0x7fe7e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7e80a2d00 0x7fe7e80a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe7f804b000 tx=0x7fe7f8048000 comp rx=0 tx=0).stop 2024-04-25T01:31:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 -- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 msgr2=0x7fe800198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 0x7fe800198620 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400b4a0 tx=0x7fe7f400b960 comp rx=0 tx=0).stop 2024-04-25T01:31:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 -- 172.21.15.27:0/1389237006 shutdown_connections 2024-04-25T01:31:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe800107720 0x7fe80019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe7e80a2d00 0x7fe7e80a51c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8001020d0 0x7fe800198620 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 --2- 172.21.15.27:0/1389237006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe800101710 0x7fe8001980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 -- 172.21.15.27:0/1389237006 >> 172.21.15.27:0/1389237006 conn(0x7fe8000fd180 msgr2=0x7fe8000fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 -- 172.21.15.27:0/1389237006 shutdown_connections 2024-04-25T01:31:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:31:59.719+0000 7fe8068d4640 1 -- 172.21.15.27:0/1389237006 wait complete. 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:251/600: open fd = 3 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:252/600: open fd = 3 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:253/600: open fd = 3 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.914 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:254/600: open fd = 3 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:255/600: open fd = 3 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:256/600: open fd = 3 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:257/600: open fd = 3 2024-04-25T01:31:59.915 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:258/600: open fd = 3 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:259/600: open fd = 3 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:260/600: open fd = 3 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.916 INFO:tasks.workunit.client.0.smithi027.stdout:261/600: open fd = 3 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:262/600: open fd = 3 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:263/600: open fd = 3 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:264/600: open fd = 3 2024-04-25T01:31:59.917 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:265/600: open fd = 3 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:266/600: open fd = 3 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:267/600: open fd = 3 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.918 INFO:tasks.workunit.client.0.smithi027.stdout:268/600: open fd = 3 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:269/600: open fd = 3 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:270/600: open fd = 3 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.919 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:271/600: open fd = 3 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:272/600: open fd = 3 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:273/600: open fd = 3 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.920 INFO:tasks.workunit.client.0.smithi027.stdout:274/600: open fd = 3 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:275/600: open fd = 3 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:276/600: open fd = 3 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:277/600: open fd = 3 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.921 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:278/600: open fd = 3 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:279/600: open fd = 3 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:280/600: open fd = 3 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:281/600: open fd = 3 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.922 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:282/600: open fd = 3 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:283/600: open fd = 3 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:284/600: open fd = 3 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:285/600: open fd = 3 2024-04-25T01:31:59.923 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:286/600: open fd = 3 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:287/600: open fd = 3 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:288/600: open fd = 3 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.924 INFO:tasks.workunit.client.0.smithi027.stdout:289/600: open fd = 3 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:290/600: open fd = 3 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:291/600: open fd = 3 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:292/600: open fd = 3 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:293/600: open fd = 3 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:294/600: open fd = 3 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:295/600: open fd = 3 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:296/600: open fd = 3 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:297/600: open fd = 3 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:298/600: open fd = 3 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:299/600: open fd = 3 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:300/600: open fd = 3 2024-04-25T01:31:59.927 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:301/600: open fd = 3 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:302/600: open fd = 3 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:303/600: open fd = 3 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.928 INFO:tasks.workunit.client.0.smithi027.stdout:304/600: open fd = 3 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:305/600: open fd = 3 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:306/600: open fd = 3 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.929 INFO:tasks.workunit.client.0.smithi027.stdout:307/600: open fd = 3 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:308/600: open fd = 3 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:309/600: open fd = 3 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.930 INFO:tasks.workunit.client.0.smithi027.stdout:310/600: open fd = 3 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:311/600: open fd = 3 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:312/600: open fd = 3 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:31:59.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:31:59 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/798142621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:00.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:31:59 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/798142621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:00.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:31:59 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/798142621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:00.708 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:32:00.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:00.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:32:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.840+0000 7f59e8872640 1 -- 172.21.15.27:0/1307708573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100fd0 msgr2=0x7f59e410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.840+0000 7f59e8872640 1 --2- 172.21.15.27:0/1307708573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100fd0 0x7f59e410d4e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009cd0 tx=0x7f59dc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 -- 172.21.15.27:0/1307708573 shutdown_connections 2024-04-25T01:32:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 --2- 172.21.15.27:0/1307708573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100fd0 0x7f59e410d4e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 --2- 172.21.15.27:0/1307708573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59e4100610 0x7f59e4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 --2- 172.21.15.27:0/1307708573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e41066b0 0x7f59e4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 -- 172.21.15.27:0/1307708573 >> 172.21.15.27:0/1307708573 conn(0x7f59e4077740 msgr2=0x7f59e4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 -- 172.21.15.27:0/1307708573 shutdown_connections 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 -- 172.21.15.27:0/1307708573 wait complete. 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 Processor -- start 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.841+0000 7f59e8872640 1 -- start start 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e8872640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100610 0x7f59e419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e8872640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e4100fd0 0x7f59e419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e8872640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59e41066b0 0x7f59e419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e8872640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f59e4111b80 con 0x7f59e4100610 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e8872640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59e4111a00 con 0x7f59e41066b0 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e8872640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59e4111d00 con 0x7f59e4100fd0 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100610 0x7f59e419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100610 0x7f59e419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50386/0 (socket says 172.21.15.27:50386) 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 -- 172.21.15.27:0/1752352978 learned_addr learned my addr 172.21.15.27:0/1752352978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59e41066b0 0x7f59e419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59dbfff640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e4100fd0 0x7f59e419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 -- 172.21.15.27:0/1752352978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e4100fd0 msgr2=0x7f59e419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e4100fd0 0x7f59e419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 -- 172.21.15.27:0/1752352978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59e41066b0 msgr2=0x7f59e419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59e41066b0 0x7f59e419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 -- 172.21.15.27:0/1752352978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e419f080 con 0x7f59e4100610 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59e41066b0 0x7f59e419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e259c640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100610 0x7f59e419a090 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f59d400c970 tx=0x7f59d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59db7fe640 1 -- 172.21.15.27:0/1752352978 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4004220 con 0x7f59e4100610 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59db7fe640 1 -- 172.21.15.27:0/1752352978 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59d40043c0 con 0x7f59e4100610 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e8872640 1 -- 172.21.15.27:0/1752352978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e419f370 con 0x7f59e4100610 2024-04-25T01:32:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59db7fe640 1 -- 172.21.15.27:0/1752352978 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4004a00 con 0x7f59e4100610 2024-04-25T01:32:00.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.842+0000 7f59e8872640 1 -- 172.21.15.27:0/1752352978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e41a6c00 con 0x7f59e4100610 2024-04-25T01:32:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59db7fe640 1 -- 172.21.15.27:0/1752352978 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f59d40079b0 con 0x7f59e4100610 2024-04-25T01:32:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59eb57c740 1 Processor -- start 2024-04-25T01:32:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59eb57c740 1 -- start start 2024-04-25T01:32:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d076cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d077710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d082010 0x55c72d084400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59eb57c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c72cfdb2c0 con 0x55c72d077290 2024-04-25T01:32:00.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59eb57c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c72cf83910 con 0x55c72d082010 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.845+0000 7f59eb57c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c72cfd7280 con 0x55c72d078980 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d077710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d077710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50396/0 (socket says 172.21.15.27:50396) 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 -- 172.21.15.27:0/2467438275 learned_addr learned my addr 172.21.15.27:0/2467438275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59db7fe640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f59d00a2d80 0x7f59d00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d076cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59dbfff640 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d082010 0x55c72d084400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f59d00a2d80 0x7f59d00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59db7fe640 1 -- 172.21.15.27:0/1752352978 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f59d40cf5b0 con 0x7f59e4100610 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 -- 172.21.15.27:0/2467438275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 msgr2=0x55c72d076cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d076cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 -- 172.21.15.27:0/2467438275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d082010 msgr2=0x55c72d084400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d082010 0x55c72d084400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 -- 172.21.15.27:0/2467438275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c72cf94fb0 con 0x55c72d077290 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59dbfff640 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d082010 0x55c72d084400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e259c640 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d077710 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f59d409b440 tx=0x7f59d4004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc35e95b1e026743 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59d8ff9640 1 -- 172.21.15.27:0/2467438275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d401d050 con 0x55c72d077290 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59d8ff9640 1 -- 172.21.15.27:0/2467438275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59d409b740 con 0x55c72d077290 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59d8ff9640 1 -- 172.21.15.27:0/2467438275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d40ccd80 con 0x55c72d077290 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.846+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f59d00a2d80 0x7f59d00a5240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f59dc031860 tx=0x7f59dc0095c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 -- 172.21.15.27:0/2467438275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 msgr2=0x55c72d077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d077710 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f59d409b440 tx=0x7f59d4004570 comp rx=0 tx=0).stop 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 -- 172.21.15.27:0/2467438275 shutdown_connections 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d082010 0x55c72d084400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d077710 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 --2- 172.21.15.27:0/2467438275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d076cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 -- 172.21.15.27:0/2467438275 >> 172.21.15.27:0/2467438275 conn(0x55c72d03f950 msgr2=0x55c72d0737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 -- 172.21.15.27:0/2467438275 shutdown_connections 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.847+0000 7f59eb57c740 1 -- 172.21.15.27:0/2467438275 wait complete. 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 1 -- start start 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 1 Processor -- start 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d077290 0x55c72d07ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d07f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d07f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c72d084e40 con 0x55c72d082010 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c72cf96e90 con 0x55c72d078980 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c72d084fc0 con 0x55c72d077290 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59eb57c740 10 client.0 _ll_register_callbacks cb 0x55c72cfd2970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:00.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d07f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d07f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50406/0 (socket says 172.21.15.27:50406) 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e2d9d640 1 -- 172.21.15.27:0/1414110634 learned_addr learned my addr 172.21.15.27:0/1414110634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59dbfff640 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d07f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e259c640 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d077290 0x55c72d07ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e2d9d640 1 -- 172.21.15.27:0/1414110634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d077290 msgr2=0x55c72d07ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d077290 0x55c72d07ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e2d9d640 1 -- 172.21.15.27:0/1414110634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 msgr2=0x55c72d07f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d07f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.848+0000 7f59e2d9d640 1 -- 172.21.15.27:0/1414110634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c72cdb77e0 con 0x55c72d082010 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d07f9f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009cd0 tx=0x7f59dc02a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312faaa19c1b7f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:00.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1414110634 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc051070 con 0x55c72d082010 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59eb57c740 1 -- 172.21.15.27:0/1414110634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d409b740 con 0x55c72d082010 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1414110634 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f59dc0305a0 con 0x55c72d082010 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1414110634 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f59dc0319e0 con 0x55c72d082010 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59eb57c740 10 client.17091 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59eb57c740 1 -- 172.21.15.27:0/1414110634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c72ce17f70 con 0x55c72d082010 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.849+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1414110634 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59d409b740 con 0x55c72d082010 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1414110634 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f59dc049040 con 0x55c72d082010 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59eb57c740 10 client.17091 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59eb57c740 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59eb57c740 20 client.17091 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c72d0920f0 0x55c72d094530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59eb57c740 4 client.17091 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59eb57c740 1 -- 172.21.15.27:0/1414110634 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c72cfaf5a0 con 0x55c72d0920f0 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59e259c640 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c72d0920f0 0x55c72d094530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59e259c640 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c72d0920f0 0x55c72d094530 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c06cca394af0f558 server_cookie=e65f51cd5e6d4ab2 in_seq=0 out_seq=0 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.850+0000 7f59c9ffb640 10 client.17091 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1414110634 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c72cfaf5a0 con 0x55c72d0920f0 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59c9ffb640 10 client.17091 handle_command_reply: tid=0 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 client.17091 shutdown 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 -- 172.21.15.27:0/1414110634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 msgr2=0x55c72d07f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d07f9f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009cd0 tx=0x7f59dc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 -- 172.21.15.27:0/1414110634 shutdown_connections 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c72d0920f0 0x55c72d094530 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d07f9f0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d07f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59c9ffb640 0 client.17091 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:00.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1414110634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d077290 0x55c72d07ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 -- 172.21.15.27:0/1414110634 >> 172.21.15.27:0/1414110634 conn(0x55c72d03f950 msgr2=0x55c72d0795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 -- 172.21.15.27:0/1414110634 shutdown_connections 2024-04-25T01:32:00.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 1 -- 172.21.15.27:0/1414110634 wait complete. 2024-04-25T01:32:00.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.851+0000 7f59eb57c740 20 client.17091 trim_cache size 0 max 16384 2024-04-25T01:32:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.869+0000 7f59eb57c740 1 Processor -- start 2024-04-25T01:32:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.869+0000 7f59eb57c740 1 -- start start 2024-04-25T01:32:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d077290 0x55c72d08dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d099e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d09a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59eb57c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41b3630 con 0x55c72d082010 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59eb57c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41b0f00 con 0x55c72d077290 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59eb57c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41aed50 con 0x55c72d078980 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d09a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d09a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50418/0 (socket says 172.21.15.27:50418) 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 -- 172.21.15.27:0/4104558866 learned_addr learned my addr 172.21.15.27:0/4104558866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e259c640 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d077290 0x55c72d08dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59dbfff640 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d099e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 -- 172.21.15.27:0/4104558866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 msgr2=0x55c72d099e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d099e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 -- 172.21.15.27:0/4104558866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d077290 msgr2=0x55c72d08dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d077290 0x55c72d08dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 -- 172.21.15.27:0/4104558866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c72ccaf170 con 0x55c72d082010 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e259c640 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d077290 0x55c72d08dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59dbfff640 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d099e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59e2d9d640 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d09a340 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f59dc033040 tx=0x7f59dc02afa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f457272b3dd4e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.870+0000 7f59cbfff640 1 -- 172.21.15.27:0/4104558866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc051040 con 0x55c72d082010 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59cbfff640 1 -- 172.21.15.27:0/4104558866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59dc036040 con 0x55c72d082010 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59cbfff640 1 -- 172.21.15.27:0/4104558866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc030ce0 con 0x55c72d082010 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 -- 172.21.15.27:0/4104558866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 msgr2=0x55c72d09a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d09a340 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f59dc033040 tx=0x7f59dc02afa0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 -- 172.21.15.27:0/4104558866 shutdown_connections 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d082010 0x55c72d09a340 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d078980 0x55c72d099e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 --2- 172.21.15.27:0/4104558866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d077290 0x55c72d08dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 -- 172.21.15.27:0/4104558866 >> 172.21.15.27:0/4104558866 conn(0x55c72d03f950 msgr2=0x55c72d072950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 -- 172.21.15.27:0/4104558866 shutdown_connections 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 -- 172.21.15.27:0/4104558866 wait complete. 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.871+0000 7f59eb57c740 1 -- start start 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59eb57c740 1 Processor -- start 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d08a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d087220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59eb57c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d082010 0x55c72d087760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59eb57c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41af3c0 con 0x55c72d077290 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59eb57c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41b0830 con 0x55c72d078980 2024-04-25T01:32:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59eb57c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41afa90 con 0x55c72d082010 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59eb57c740 10 client.0 _ll_register_callbacks cb 0x7f59dc003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d08a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d08a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50426/0 (socket says 172.21.15.27:50426) 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 -- 172.21.15.27:0/1510490868 learned_addr learned my addr 172.21.15.27:0/1510490868 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59dbfff640 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d087220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 -- 172.21.15.27:0/1510490868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d082010 msgr2=0x55c72d087760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d082010 0x55c72d087760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 -- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 msgr2=0x55c72d087220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d087220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 -- 172.21.15.27:0/1510490868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c72d087ea0 con 0x55c72d077290 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59dbfff640 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d087220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59e259c640 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d08a520 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f59d4098110 tx=0x7f59d40053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cce6aa25455ee2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.872+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1510490868 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d401d040 con 0x55c72d077290 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1510490868 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f59d4099a40 con 0x55c72d077290 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 1 -- 172.21.15.27:0/1510490868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59dc036040 con 0x55c72d077290 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1510490868 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f59d40a43b0 con 0x55c72d077290 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 10 client.17097 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 1 -- 172.21.15.27:0/1510490868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c72d0f9580 con 0x55c72d077290 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1510490868 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59dc036040 con 0x55c72d077290 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1510490868 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f59d4097db0 con 0x55c72d077290 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 10 client.17097 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 20 client.17097 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c72d0f9870 0x55c72d0fbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 4 client.17097 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59eb57c740 1 -- 172.21.15.27:0/1510490868 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c72cfe8800 con 0x55c72d0f9870 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.873+0000 7f59dbfff640 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c72d0f9870 0x55c72d0fbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.874+0000 7f59dbfff640 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c72d0f9870 0x55c72d0fbc60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8927afd6f4763161 server_cookie=ad515a263aa83ff4 in_seq=0 out_seq=0 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.874+0000 7f59c9ffb640 10 client.17097 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59c9ffb640 1 -- 172.21.15.27:0/1510490868 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c72cfe8800 con 0x55c72d0f9870 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59c9ffb640 10 client.17097 handle_command_reply: tid=0 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 client.17097 shutdown 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 -- 172.21.15.27:0/1510490868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 msgr2=0x55c72d08a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d08a520 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f59d4098110 tx=0x7f59d40053f0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 -- 172.21.15.27:0/1510490868 shutdown_connections 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c72d0f9870 0x55c72d0fbc60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c72d082010 0x55c72d087760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c72d078980 0x55c72d087220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 --2- 172.21.15.27:0/1510490868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c72d077290 0x55c72d08a520 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 -- 172.21.15.27:0/1510490868 >> 172.21.15.27:0/1510490868 conn(0x55c72d03f950 msgr2=0x55c72d078d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59c9ffb640 0 client.17097 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 -- 172.21.15.27:0/1510490868 shutdown_connections 2024-04-25T01:32:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 1 -- 172.21.15.27:0/1510490868 wait complete. 2024-04-25T01:32:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.875+0000 7f59eb57c740 20 client.17097 trim_cache size 0 max 16384 2024-04-25T01:32:00.860 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:32:00.861 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:32:00.861 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "2c8b4498-7b72-4f5e-8365-a5a683bf100d", 2024-04-25T01:32:00.861 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:32:00.861 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 -- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f59d00a2d80 msgr2=0x7f59d00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f59d00a2d80 0x7f59d00a5240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f59dc031860 tx=0x7f59dc0095c0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 -- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100610 msgr2=0x7f59e419a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100610 0x7f59e419a090 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f59d400c970 tx=0x7f59d400ce30 comp rx=0 tx=0).stop 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 -- 172.21.15.27:0/1752352978 shutdown_connections 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59e41066b0 0x7f59e419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59e4100fd0 0x7f59e419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f59d00a2d80 0x7f59d00a5240 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 --2- 172.21.15.27:0/1752352978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59e4100610 0x7f59e419a090 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 -- 172.21.15.27:0/1752352978 >> 172.21.15.27:0/1752352978 conn(0x7f59e4077740 msgr2=0x7f59e40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 -- 172.21.15.27:0/1752352978 shutdown_connections 2024-04-25T01:32:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.876+0000 7f59e8872640 1 -- 172.21.15.27:0/1752352978 wait complete. 2024-04-25T01:32:00.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:32:00.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:32:00.885 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:00 smithi027 ceph-mon[29197]: pgmap v200: 129 pgs: 129 active+clean; 3.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:00.941 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:00 smithi139 ceph-mon[32760]: pgmap v200: 129 pgs: 129 active+clean; 3.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 -- 172.21.15.27:0/3234932922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100fe0 msgr2=0x7f012810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 --2- 172.21.15.27:0/3234932922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100fe0 0x7f012810d4f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0120009cd0 tx=0x7f012002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 -- 172.21.15.27:0/3234932922 shutdown_connections 2024-04-25T01:32:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 --2- 172.21.15.27:0/3234932922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100fe0 0x7f012810d4f0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 --2- 172.21.15.27:0/3234932922 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0128100620 0x7f0128100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 --2- 172.21.15.27:0/3234932922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0128106630 0x7f0128106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 -- 172.21.15.27:0/3234932922 >> 172.21.15.27:0/3234932922 conn(0x7f01280776a0 msgr2=0x7f0128077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 -- 172.21.15.27:0/3234932922 shutdown_connections 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.990+0000 7f012e84d640 1 -- 172.21.15.27:0/3234932922 wait complete. 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012e84d640 1 Processor -- start 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012e84d640 1 -- start start 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012e84d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100620 0x7f0128197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012e84d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0128100fe0 0x7f0128198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012e84d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0128106630 0x7f012819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012e84d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0128111b40 con 0x7f0128100620 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012e84d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f01281119c0 con 0x7f0128106630 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012e84d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0128111cc0 con 0x7f0128100fe0 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f0127fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100620 0x7f0128197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f0127fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100620 0x7f0128197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50448/0 (socket says 172.21.15.27:50448) 2024-04-25T01:32:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f0127fff640 1 -- 172.21.15.27:0/539183910 learned_addr learned my addr 172.21.15.27:0/539183910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f012cdea640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0128106630 0x7f012819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f01277fe640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0128100fe0 0x7f0128198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f0127fff640 1 -- 172.21.15.27:0/539183910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0128100fe0 msgr2=0x7f0128198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.991+0000 7f0127fff640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0128100fe0 0x7f0128198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f0127fff640 1 -- 172.21.15.27:0/539183910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0128106630 msgr2=0x7f012819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f0127fff640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0128106630 0x7f012819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f0127fff640 1 -- 172.21.15.27:0/539183910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012819cfb0 con 0x7f0128100620 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f0127fff640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100620 0x7f0128197fc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f011800c890 tx=0x7f011800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f01257fa640 1 -- 172.21.15.27:0/539183910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118004220 con 0x7f0128100620 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f01257fa640 1 -- 172.21.15.27:0/539183910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01180043c0 con 0x7f0128100620 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f012e84d640 1 -- 172.21.15.27:0/539183910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012819d2a0 con 0x7f0128100620 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f01257fa640 1 -- 172.21.15.27:0/539183910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01180049e0 con 0x7f0128100620 2024-04-25T01:32:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.992+0000 7f012e84d640 1 -- 172.21.15.27:0/539183910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01281a4b30 con 0x7f0128100620 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.994+0000 7f01257fa640 1 -- 172.21.15.27:0/539183910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f01180079b0 con 0x7f0128100620 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0131637740 1 Processor -- start 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0131637740 1 -- start start 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0131637740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c0e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0131637740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0131637740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 0x560898c1fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0131637740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560898b736b0 con 0x560898c0e510 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0131637740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560898b1b880 con 0x560898c1da30 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0131637740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560898b6f670 con 0x560898c0ee30 2024-04-25T01:32:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f012cdea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c0e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f012cdea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c0e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50464/0 (socket says 172.21.15.27:50464) 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f012cdea640 1 -- 172.21.15.27:0/3989564668 learned_addr learned my addr 172.21.15.27:0/3989564668 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f01277fe640 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 0x560898c1fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f01257fa640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f01100a2d00 0x7f01100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0127fff640 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f012cdea640 1 -- 172.21.15.27:0/3989564668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 msgr2=0x560898c1d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f012cdea640 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f012cdea640 1 -- 172.21.15.27:0/3989564668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 msgr2=0x560898c1fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f012cdea640 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 0x560898c1fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f012cdea640 1 -- 172.21.15.27:0/3989564668 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608989517d0 con 0x560898c0e510 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f0127fff640 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.995+0000 7f01257fa640 1 -- 172.21.15.27:0/539183910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f01180cf800 con 0x7f0128100620 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.996+0000 7f012cdea640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f01100a2d00 0x7f01100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.996+0000 7f012cdea640 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c0e8f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f0120009cd0 tx=0x7f0120031520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b5634e36409da9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.996+0000 7f010e7fc640 1 -- 172.21.15.27:0/3989564668 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120041040 con 0x560898c0e510 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.996+0000 7f010e7fc640 1 -- 172.21.15.27:0/3989564668 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0120043070 con 0x560898c0e510 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.996+0000 7f012cdea640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f01100a2d00 0x7f01100a51c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f012004c000 tx=0x7f012004d000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.996+0000 7f010e7fc640 1 -- 172.21.15.27:0/3989564668 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120030d30 con 0x560898c0e510 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- 172.21.15.27:0/3989564668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 msgr2=0x560898c0e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c0e8f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f0120009cd0 tx=0x7f0120031520 comp rx=0 tx=0).stop 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- 172.21.15.27:0/3989564668 shutdown_connections 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 0x560898c1fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 --2- 172.21.15.27:0/3989564668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c0e8f0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- 172.21.15.27:0/3989564668 >> 172.21.15.27:0/3989564668 conn(0x560898c0a950 msgr2=0x560898c0ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- 172.21.15.27:0/3989564668 shutdown_connections 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- 172.21.15.27:0/3989564668 wait complete. 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- start start 2024-04-25T01:32:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 Processor -- start 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0e510 0x560898c1a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c1da30 0x560898c1b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560898c20870 con 0x560898c1da30 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560898b2ef50 con 0x560898c0e510 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560898c209f0 con 0x560898c0ee30 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.997+0000 7f0131637740 10 client.0 _ll_register_callbacks cb 0x560898b6a000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c1da30 0x560898c1b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c1da30 0x560898c1b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50468/0 (socket says 172.21.15.27:50468) 2024-04-25T01:32:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 -- 172.21.15.27:0/151187635 learned_addr learned my addr 172.21.15.27:0/151187635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f0127fff640 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0e510 0x560898c1a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f01277fe640 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 -- 172.21.15.27:0/151187635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 msgr2=0x560898c1ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 -- 172.21.15.27:0/151187635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0e510 msgr2=0x560898c1a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0e510 0x560898c1a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 -- 172.21.15.27:0/151187635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608988873f0 con 0x560898c1da30 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f012cdea640 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c1da30 0x560898c1b3d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0120030a10 tx=0x7f0120005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16132c0ab114b47a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f01077fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f01077fe640 1 -- 172.21.15.27:0/151187635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120059020 con 0x560898c1da30 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f01077fe640 1 -- 172.21.15.27:0/151187635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f01200305a0 con 0x560898c1da30 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f0131637740 1 -- 172.21.15.27:0/151187635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0120043070 con 0x560898c1da30 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f01077fe640 1 -- 172.21.15.27:0/151187635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0120064050 con 0x560898c1da30 2024-04-25T01:32:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f0131637740 10 client.17109 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.998+0000 7f0131637740 1 -- 172.21.15.27:0/151187635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5608988470f0 con 0x560898c1da30 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f01077fe640 1 -- 172.21.15.27:0/151187635 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0120043070 con 0x560898c1da30 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f01077fe640 1 -- 172.21.15.27:0/151187635 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0120030070 con 0x560898c1da30 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f0131637740 10 client.17109 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f0131637740 10 client.17109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f0131637740 20 client.17109 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f0131637740 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560898c2db90 0x560898c2ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f0131637740 4 client.17109 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f0131637740 1 -- 172.21.15.27:0/151187635 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560898b80bf0 con 0x560898c2db90 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:00.999+0000 7f0127fff640 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560898c2db90 0x560898c2ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.000+0000 7f01077fe640 10 client.17109 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.000+0000 7f0127fff640 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560898c2db90 0x560898c2ff80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=376aad099b629d63 server_cookie=c9855d8dff70143f in_seq=0 out_seq=0 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.000+0000 7f01077fe640 1 -- 172.21.15.27:0/151187635 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560898b80bf0 con 0x560898c2db90 2024-04-25T01:32:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.000+0000 7f01077fe640 10 client.17109 handle_command_reply: tid=0 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.000+0000 7f0131637740 1 client.17109 shutdown 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.000+0000 7f0131637740 1 -- 172.21.15.27:0/151187635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c1da30 msgr2=0x560898c1b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.000+0000 7f0131637740 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c1da30 0x560898c1b3d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0120030a10 tx=0x7f0120005ef0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 1 -- 172.21.15.27:0/151187635 shutdown_connections 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560898c2db90 0x560898c2ff80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c1da30 0x560898c1b3d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0ee30 0x560898c1ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f01077fe640 0 client.17109 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 1 --2- 172.21.15.27:0/151187635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0e510 0x560898c1a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 1 -- 172.21.15.27:0/151187635 >> 172.21.15.27:0/151187635 conn(0x560898c0a950 msgr2=0x560898c10690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 1 -- 172.21.15.27:0/151187635 shutdown_connections 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 1 -- 172.21.15.27:0/151187635 wait complete. 2024-04-25T01:32:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.001+0000 7f0131637740 20 client.17109 trim_cache size 0 max 16384 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0131637740 1 Processor -- start 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0131637740 1 -- start start 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0131637740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c1d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0131637740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0ee30 0x560898c297d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0131637740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c1da30 0x560898c19ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0131637740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f01281af220 con 0x560898c0e510 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0131637740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f01281acaf0 con 0x560898c0ee30 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0131637740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01281aa940 con 0x560898c1da30 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c1d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c1d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50472/0 (socket says 172.21.15.27:50472) 2024-04-25T01:32:00.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 -- 172.21.15.27:0/214806048 learned_addr learned my addr 172.21.15.27:0/214806048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f01277fe640 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0ee30 0x560898c297d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f012cdea640 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c1da30 0x560898c19ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 -- 172.21.15.27:0/214806048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c1da30 msgr2=0x560898c19ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c1da30 0x560898c19ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 -- 172.21.15.27:0/214806048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0ee30 msgr2=0x560898c297d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0ee30 0x560898c297d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 -- 172.21.15.27:0/214806048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560898bfc090 con 0x560898c0e510 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f01277fe640 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0ee30 0x560898c297d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.019+0000 7f0127fff640 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c1d080 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0118097df0 tx=0x7f011806d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3136ebf954dbcea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.020+0000 7f010dffb640 1 -- 172.21.15.27:0/214806048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011801d040 con 0x560898c0e510 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.020+0000 7f010dffb640 1 -- 172.21.15.27:0/214806048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f011809b990 con 0x560898c0e510 2024-04-25T01:32:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.020+0000 7f010dffb640 1 -- 172.21.15.27:0/214806048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118007ca0 con 0x560898c0e510 2024-04-25T01:32:01.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.020+0000 7f0131637740 1 -- 172.21.15.27:0/214806048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 msgr2=0x560898c1d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.020+0000 7f0131637740 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c1d080 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0118097df0 tx=0x7f011806d950 comp rx=0 tx=0).stop 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 -- 172.21.15.27:0/214806048 shutdown_connections 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c1da30 0x560898c19ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c0ee30 0x560898c297d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 --2- 172.21.15.27:0/214806048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0e510 0x560898c1d080 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 -- 172.21.15.27:0/214806048 >> 172.21.15.27:0/214806048 conn(0x560898c0a950 msgr2=0x560898c0ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 -- 172.21.15.27:0/214806048 shutdown_connections 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 -- 172.21.15.27:0/214806048 wait complete. 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 -- start start 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 Processor -- start 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0e510 0x560898c25f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0ee30 0x560898c21320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 0x560898c21860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:01.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f01281aafb0 con 0x560898c0ee30 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f01281ac420 con 0x560898c1da30 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01281ab680 con 0x560898c0e510 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f0131637740 10 client.0 _ll_register_callbacks cb 0x7f011c009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.021+0000 7f01277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0ee30 0x560898c21320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0ee30 0x560898c21320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50484/0 (socket says 172.21.15.27:50484) 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01277fe640 1 -- 172.21.15.27:0/757796421 learned_addr learned my addr 172.21.15.27:0/757796421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f0127fff640 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0e510 0x560898c25f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f012cdea640 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 0x560898c21860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01277fe640 1 -- 172.21.15.27:0/757796421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0e510 msgr2=0x560898c25f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01277fe640 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0e510 0x560898c25f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01277fe640 1 -- 172.21.15.27:0/757796421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 msgr2=0x560898c21860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01277fe640 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 0x560898c21860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01277fe640 1 -- 172.21.15.27:0/757796421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560898c21ed0 con 0x560898c0ee30 2024-04-25T01:32:01.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01277fe640 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0ee30 0x560898c21320 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f0128105bb0 tx=0x7f011c013b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef56b8158f47fe5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01077fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01077fe640 1 -- 172.21.15.27:0/757796421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011c011e70 con 0x560898c0ee30 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f0131637740 1 -- 172.21.15.27:0/757796421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f011809b990 con 0x560898c0ee30 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01077fe640 1 -- 172.21.15.27:0/757796421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f011c01d410 con 0x560898c0ee30 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f01077fe640 1 -- 172.21.15.27:0/757796421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f011c02b050 con 0x560898c0ee30 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f0131637740 10 client.17115 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.022+0000 7f0131637740 1 -- 172.21.15.27:0/757796421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560898c93d80 con 0x560898c0ee30 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f01077fe640 1 -- 172.21.15.27:0/757796421 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f011809b990 con 0x560898c0ee30 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f01077fe640 1 -- 172.21.15.27:0/757796421 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f011c00f870 con 0x560898c0ee30 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f0131637740 10 client.17115 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f0131637740 10 client.17115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f0131637740 20 client.17115 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f0131637740 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560898c94580 0x560898c969a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f0131637740 4 client.17115 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f0131637740 1 -- 172.21.15.27:0/757796421 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560898b7af40 con 0x560898c94580 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f0127fff640 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560898c94580 0x560898c969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f0127fff640 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560898c94580 0x560898c969a0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da8d355aef226df9 server_cookie=c9454890a22c4a36 in_seq=0 out_seq=0 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f01077fe640 10 client.17115 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f01077fe640 1 -- 172.21.15.27:0/757796421 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560898b7af40 con 0x560898c94580 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.023+0000 7f01077fe640 10 client.17115 handle_command_reply: tid=0 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 client.17115 shutdown 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 -- 172.21.15.27:0/757796421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0ee30 msgr2=0x560898c21320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0ee30 0x560898c21320 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f0128105bb0 tx=0x7f011c013b70 comp rx=0 tx=0).stop 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 -- 172.21.15.27:0/757796421 shutdown_connections 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560898c94580 0x560898c969a0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560898c1da30 0x560898c21860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f01077fe640 0 client.17115 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560898c0ee30 0x560898c21320 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 --2- 172.21.15.27:0/757796421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560898c0e510 0x560898c25f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 -- 172.21.15.27:0/757796421 >> 172.21.15.27:0/757796421 conn(0x560898c0a950 msgr2=0x560898c09260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 -- 172.21.15.27:0/757796421 shutdown_connections 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 1 -- 172.21.15.27:0/757796421 wait complete. 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f0131637740 20 client.17115 trim_cache size 0 max 16384 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:32:01.008 INFO:teuthology.orchestra.run.smithi027.stdout: "2c8b4498-7b72-4f5e-8365-a5a683bf100d": { 2024-04-25T01:32:01.009 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:32:01.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "2c8b4498-7b72-4f5e-8365-a5a683bf100d", 2024-04-25T01:32:01.009 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:32:01.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:32:01.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:32:01.009 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f012e84d640 1 -- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f01100a2d00 msgr2=0x7f01100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f012e84d640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f01100a2d00 0x7f01100a51c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f012004c000 tx=0x7f012004d000 comp rx=0 tx=0).stop 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f012e84d640 1 -- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100620 msgr2=0x7f0128197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f012e84d640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100620 0x7f0128197fc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f011800c890 tx=0x7f011800cd50 comp rx=0 tx=0).stop 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.024+0000 7f012e84d640 1 -- 172.21.15.27:0/539183910 shutdown_connections 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.025+0000 7f012e84d640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0128106630 0x7f012819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.025+0000 7f012e84d640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0128100fe0 0x7f0128198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.025+0000 7f012e84d640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f01100a2d00 0x7f01100a51c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.025+0000 7f012e84d640 1 --2- 172.21.15.27:0/539183910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0128100620 0x7f0128197fc0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.025+0000 7f012e84d640 1 -- 172.21.15.27:0/539183910 >> 172.21.15.27:0/539183910 conn(0x7f01280776a0 msgr2=0x7f01281030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.025+0000 7f012e84d640 1 -- 172.21.15.27:0/539183910 shutdown_connections 2024-04-25T01:32:01.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:01.025+0000 7f012e84d640 1 -- 172.21.15.27:0/539183910 wait complete. 2024-04-25T01:32:01.013 INFO:tasks.cephfs.filesystem:scrub status for tag:2c8b4498-7b72-4f5e-8365-a5a683bf100d - {'path': '/', 'tag': '2c8b4498-7b72-4f5e-8365-a5a683bf100d', 'options': 'recursive,force'} 2024-04-25T01:32:01.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:00 smithi153 ceph-mon[32621]: pgmap v200: 129 pgs: 129 active+clean; 3.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:01.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:01 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:32:01.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:01 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:01.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:01 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:32:01.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:01 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:02.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:01 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:32:02.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:01 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:02.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:01 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:32:02.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:01 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:02.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:01 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:32:02.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:01 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:02.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:01 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:32:02.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:01 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:02.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:02 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:32:02.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:02 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:02.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:02 smithi027 ceph-mon[29197]: pgmap v201: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-25T01:32:03.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:02 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:32:03.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:02 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:03.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:02 smithi153 ceph-mon[32621]: pgmap v201: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-25T01:32:03.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:02 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:32:03.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:02 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:03.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:02 smithi139 ceph-mon[32760]: pgmap v201: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-25T01:32:03.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:03 smithi027 ceph-mon[29197]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T01:32:04.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:03 smithi153 ceph-mon[32621]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T01:32:04.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:03 smithi139 ceph-mon[32760]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T01:32:04.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:04 smithi027 ceph-mon[29197]: pgmap v203: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 66 KiB/s wr, 32 op/s 2024-04-25T01:32:05.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:04 smithi153 ceph-mon[32621]: pgmap v203: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 66 KiB/s wr, 32 op/s 2024-04-25T01:32:05.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:04 smithi139 ceph-mon[32760]: pgmap v203: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 66 KiB/s wr, 32 op/s 2024-04-25T01:32:06.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:06 smithi027 ceph-mon[29197]: pgmap v204: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 66 KiB/s wr, 32 op/s 2024-04-25T01:32:07.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:06 smithi153 ceph-mon[32621]: pgmap v204: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 66 KiB/s wr, 32 op/s 2024-04-25T01:32:07.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:06 smithi139 ceph-mon[32760]: pgmap v204: 129 pgs: 129 active+clean; 3.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 66 KiB/s wr, 32 op/s 2024-04-25T01:32:07.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:07 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:32:08.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:07 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:32:08.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:07 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:32:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:08 smithi027 ceph-mon[29197]: pgmap v205: 129 pgs: 129 active+clean; 3.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:32:08.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:08 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:32:09.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:08 smithi153 ceph-mon[32621]: pgmap v205: 129 pgs: 129 active+clean; 3.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:32:09.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:08 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:32:09.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:08 smithi139 ceph-mon[32760]: pgmap v205: 129 pgs: 129 active+clean; 3.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:32:09.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:08 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:32:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:10 smithi027 ceph-mon[29197]: pgmap v206: 129 pgs: 129 active+clean; 3.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:32:10.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:10 smithi027 ceph-mon[29197]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T01:32:11.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:10 smithi153 ceph-mon[32621]: pgmap v206: 129 pgs: 129 active+clean; 3.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:32:11.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:10 smithi153 ceph-mon[32621]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T01:32:11.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:10 smithi139 ceph-mon[32760]: pgmap v206: 129 pgs: 129 active+clean; 3.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:32:11.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:10 smithi139 ceph-mon[32760]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T01:32:12.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:12 smithi027 ceph-mon[29197]: pgmap v208: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:32:13.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:12 smithi153 ceph-mon[32621]: pgmap v208: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:32:13.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:12 smithi139 ceph-mon[32760]: pgmap v208: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:32:14.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:14 smithi027 ceph-mon[29197]: pgmap v209: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-04-25T01:32:15.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:14 smithi153 ceph-mon[32621]: pgmap v209: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-04-25T01:32:15.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:14 smithi139 ceph-mon[32760]: pgmap v209: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-04-25T01:32:16.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:16 smithi027 ceph-mon[29197]: pgmap v210: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-04-25T01:32:17.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:16 smithi153 ceph-mon[32621]: pgmap v210: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-04-25T01:32:17.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:16 smithi139 ceph-mon[32760]: pgmap v210: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-04-25T01:32:18.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:18 smithi027 ceph-mon[29197]: pgmap v211: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.0 KiB/s wr, 11 op/s 2024-04-25T01:32:19.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:18 smithi153 ceph-mon[32621]: pgmap v211: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.0 KiB/s wr, 11 op/s 2024-04-25T01:32:19.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:18 smithi139 ceph-mon[32760]: pgmap v211: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.0 KiB/s wr, 11 op/s 2024-04-25T01:32:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:20 smithi027 ceph-mon[29197]: pgmap v212: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.0 KiB/s wr, 11 op/s 2024-04-25T01:32:21.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:20 smithi153 ceph-mon[32621]: pgmap v212: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.0 KiB/s wr, 11 op/s 2024-04-25T01:32:21.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:20 smithi139 ceph-mon[32760]: pgmap v212: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.0 KiB/s wr, 11 op/s 2024-04-25T01:32:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:22 smithi027 ceph-mon[29197]: pgmap v213: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-04-25T01:32:23.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:22 smithi153 ceph-mon[32621]: pgmap v213: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-04-25T01:32:23.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:22 smithi139 ceph-mon[32760]: pgmap v213: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-04-25T01:32:24.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:24 smithi027 ceph-mon[29197]: pgmap v214: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:25.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:24 smithi153 ceph-mon[32621]: pgmap v214: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:25.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:24 smithi139 ceph-mon[32760]: pgmap v214: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:26.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:26 smithi027 ceph-mon[29197]: pgmap v215: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:27.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:26 smithi153 ceph-mon[32621]: pgmap v215: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:27.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:26 smithi139 ceph-mon[32760]: pgmap v215: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-25T01:32:28.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:27 smithi153 ceph-mon[32621]: pgmap v216: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-04-25T01:32:28.039 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:27 smithi139 ceph-mon[32760]: pgmap v216: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-04-25T01:32:28.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:27 smithi027 ceph-mon[29197]: pgmap v216: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-04-25T01:32:30.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:30 smithi027 ceph-mon[29197]: pgmap v217: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:32:30.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:30 smithi153 ceph-mon[32621]: pgmap v217: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:32:30.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:30 smithi139 ceph-mon[32760]: pgmap v217: 129 pgs: 129 active+clean; 3.4 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:32:31.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:32:31.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:32:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 -- 172.21.15.27:0/3306475965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a0069d40 msgr2=0x7f63a006a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 --2- 172.21.15.27:0/3306475965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a0069d40 0x7f63a006a180 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f638c009930 tx=0x7f638c02a420 comp rx=0 tx=0).stop 2024-04-25T01:32:31.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 -- 172.21.15.27:0/3306475965 shutdown_connections 2024-04-25T01:32:31.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 --2- 172.21.15.27:0/3306475965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63a006a6c0 0x7f63a010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 --2- 172.21.15.27:0/3306475965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a0069d40 0x7f63a006a180 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 --2- 172.21.15.27:0/3306475965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63a0107720 0x7f63a0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 -- 172.21.15.27:0/3306475965 >> 172.21.15.27:0/3306475965 conn(0x7f63a0076440 msgr2=0x7f63a0076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 -- 172.21.15.27:0/3306475965 shutdown_connections 2024-04-25T01:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 -- 172.21.15.27:0/3306475965 wait complete. 2024-04-25T01:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.158+0000 7f63a71e9640 1 Processor -- start 2024-04-25T01:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a71e9640 1 -- start start 2024-04-25T01:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a71e9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63a0069d40 0x7f63a0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a71e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a006a6c0 0x7f63a01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a71e9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63a0107720 0x7f63a019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a71e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63a0111b20 con 0x7f63a006a6c0 2024-04-25T01:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a71e9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01119a0 con 0x7f63a0069d40 2024-04-25T01:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a71e9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f63a0111ca0 con 0x7f63a0107720 2024-04-25T01:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a006a6c0 0x7f63a01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a006a6c0 0x7f63a01985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48066/0 (socket says 172.21.15.27:48066) 2024-04-25T01:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 -- 172.21.15.27:0/757520064 learned_addr learned my addr 172.21.15.27:0/757520064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a4f85640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63a0069d40 0x7f63a0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a5786640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63a0107720 0x7f63a019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 -- 172.21.15.27:0/757520064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63a0107720 msgr2=0x7f63a019c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63a0107720 0x7f63a019c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 -- 172.21.15.27:0/757520064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63a0069d40 msgr2=0x7f63a0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63a0069d40 0x7f63a0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 -- 172.21.15.27:0/757520064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a019d050 con 0x7f63a006a6c0 2024-04-25T01:32:31.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f63a4f85640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63a0069d40 0x7f63a0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.159+0000 7f639ffff640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a006a6c0 0x7f63a01985a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f638c009230 tx=0x7f638c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.160+0000 7f639dffb640 1 -- 172.21.15.27:0/757520064 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c03f030 con 0x7f63a006a6c0 2024-04-25T01:32:31.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.160+0000 7f63a71e9640 1 -- 172.21.15.27:0/757520064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a019d2e0 con 0x7f63a006a6c0 2024-04-25T01:32:31.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.160+0000 7f639dffb640 1 -- 172.21.15.27:0/757520064 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c003ad0 con 0x7f63a006a6c0 2024-04-25T01:32:31.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.160+0000 7f639dffb640 1 -- 172.21.15.27:0/757520064 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c035070 con 0x7f63a006a6c0 2024-04-25T01:32:31.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.160+0000 7f63a71e9640 1 -- 172.21.15.27:0/757520064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a01a4bd0 con 0x7f63a006a6c0 2024-04-25T01:32:31.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.162+0000 7f639dffb640 1 -- 172.21.15.27:0/757520064 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f638c042030 con 0x7f63a006a6c0 2024-04-25T01:32:31.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a9fcb740 1 Processor -- start 2024-04-25T01:32:31.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a9fcb740 1 -- start start 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 0x55e23557e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e2354e36b0 con 0x55e23558da30 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e23548b880 con 0x55e23557e530 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e2354df670 con 0x55e23557ee50 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639dffb640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f63900a2d00 0x7f63900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a5786640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 0x55e23557e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a4f85640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48078/0 (socket says 172.21.15.27:48078) 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639ffff640 1 -- 172.21.15.27:0/3939284165 learned_addr learned my addr 172.21.15.27:0/3939284165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f63a5786640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f63900a2d00 0x7f63900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639dffb640 1 -- 172.21.15.27:0/757520064 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f638c0efd40 con 0x7f63a006a6c0 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639ffff640 1 -- 172.21.15.27:0/3939284165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 msgr2=0x55e23558d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639ffff640 1 --2- 172.21.15.27:0/3939284165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639ffff640 1 -- 172.21.15.27:0/3939284165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 msgr2=0x55e23557e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639ffff640 1 --2- 172.21.15.27:0/3939284165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 0x55e23557e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.163+0000 7f639ffff640 1 -- 172.21.15.27:0/3939284165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2352c17d0 con 0x55e23558da30 2024-04-25T01:32:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.164+0000 7f639ffff640 1 --2- 172.21.15.27:0/3939284165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558fe20 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f638c0c45b0 tx=0x7f638c0045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72dd6143fcc5a7b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.164+0000 7f63a5786640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f63900a2d00 0x7f63900a51c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6398009fd0 tx=0x7f63980096a0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.164+0000 7f638affd640 1 -- 172.21.15.27:0/3939284165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c049070 con 0x55e23558da30 2024-04-25T01:32:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.164+0000 7f638affd640 1 -- 172.21.15.27:0/3939284165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c0bc570 con 0x55e23558da30 2024-04-25T01:32:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.164+0000 7f638affd640 1 -- 172.21.15.27:0/3939284165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c03b530 con 0x55e23558da30 2024-04-25T01:32:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 -- 172.21.15.27:0/3939284165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 msgr2=0x55e23558fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/3939284165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558fe20 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f638c0c45b0 tx=0x7f638c0045f0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 -- 172.21.15.27:0/3939284165 shutdown_connections 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/3939284165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558fe20 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/3939284165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/3939284165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 0x55e23557e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 -- 172.21.15.27:0/3939284165 >> 172.21.15.27:0/3939284165 conn(0x55e23557a950 msgr2=0x55e23557ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 -- 172.21.15.27:0/3939284165 shutdown_connections 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 -- 172.21.15.27:0/3939284165 wait complete. 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 -- start start 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.165+0000 7f63a9fcb740 1 Processor -- start 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 0x55e23558a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e2355905b0 con 0x55e23558da30 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55e23549ef50 con 0x55e23557e530 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e235590730 con 0x55e23557ee50 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a9fcb740 10 client.0 _ll_register_callbacks cb 0x55e2354da000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48094/0 (socket says 172.21.15.27:48094) 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 -- 172.21.15.27:0/2483702734 learned_addr learned my addr 172.21.15.27:0/2483702734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f639ffff640 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a4f85640 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 0x55e23558a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 -- 172.21.15.27:0/2483702734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 msgr2=0x55e23558aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 -- 172.21.15.27:0/2483702734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 msgr2=0x55e23558a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 0x55e23558a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 -- 172.21.15.27:0/2483702734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2351f73f0 con 0x55e23558da30 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f639ffff640 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f63a5786640 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558b3f0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6398012570 tx=0x7f6398004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e3bb4b90ac728a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.166+0000 7f637ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.169+0000 7f637ffff640 1 -- 172.21.15.27:0/2483702734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398010e20 con 0x55e23558da30 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.169+0000 7f637ffff640 1 -- 172.21.15.27:0/2483702734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f6398004b20 con 0x55e23558da30 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.169+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2483702734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f638c0bc570 con 0x55e23558da30 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.169+0000 7f637ffff640 1 -- 172.21.15.27:0/2483702734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6398031050 con 0x55e23558da30 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.169+0000 7f63a9fcb740 10 client.17127 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.169+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2483702734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e2351b70f0 con 0x55e23558da30 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.169+0000 7f637ffff640 1 -- 172.21.15.27:0/2483702734 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c0bc570 con 0x55e23558da30 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.169+0000 7f637ffff640 1 -- 172.21.15.27:0/2483702734 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f639802b4b0 con 0x55e23558da30 2024-04-25T01:32:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f63a9fcb740 10 client.17127 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f63a9fcb740 10 client.17127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f63a9fcb740 20 client.17127 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e23559d990 0x55e23559fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f63a9fcb740 4 client.17127 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2483702734 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2354f0bf0 con 0x55e23559d990 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f63a4f85640 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e23559d990 0x55e23559fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f63a4f85640 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e23559d990 0x55e23559fdd0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=809fd2c577100b6c server_cookie=3372400155c7fd01 in_seq=0 out_seq=0 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.170+0000 7f637ffff640 10 client.17127 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f637ffff640 1 -- 172.21.15.27:0/2483702734 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e2354f0bf0 con 0x55e23559d990 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f637ffff640 10 client.17127 handle_command_reply: tid=0 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 client.17127 shutdown 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2483702734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 msgr2=0x55e23558b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558b3f0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6398012570 tx=0x7f6398004ab0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2483702734 shutdown_connections 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e23559d990 0x55e23559fdd0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e23558b3f0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557ee50 0x55e23558aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f637ffff640 0 client.17127 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:31.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2483702734 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557e530 0x55e23558a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.171+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2483702734 >> 172.21.15.27:0/2483702734 conn(0x55e23557a950 msgr2=0x55e235580780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.172+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2483702734 shutdown_connections 2024-04-25T01:32:31.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.172+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2483702734 wait complete. 2024-04-25T01:32:31.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.172+0000 7f63a9fcb740 20 client.17127 trim_cache size 0 max 16384 2024-04-25T01:32:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.189+0000 7f63a9fcb740 1 Processor -- start 2024-04-25T01:32:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.189+0000 7f63a9fcb740 1 -- start start 2024-04-25T01:32:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e23558d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557ee50 0x55e2355993e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e235579970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01af390 con 0x55e23558da30 2024-04-25T01:32:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01acc60 con 0x55e23557ee50 2024-04-25T01:32:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01aaad0 con 0x55e23557e530 2024-04-25T01:32:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e235579970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e235579970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48098/0 (socket says 172.21.15.27:48098) 2024-04-25T01:32:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 -- 172.21.15.27:0/631229071 learned_addr learned my addr 172.21.15.27:0/631229071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a4f85640 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e23558d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f639ffff640 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557ee50 0x55e2355993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 -- 172.21.15.27:0/631229071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 msgr2=0x55e23558d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e23558d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 -- 172.21.15.27:0/631229071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557ee50 msgr2=0x55e2355993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557ee50 0x55e2355993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 -- 172.21.15.27:0/631229071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e23557a780 con 0x55e23558da30 2024-04-25T01:32:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f63a5786640 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e235579970 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f6398009790 tx=0x7f639800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f06c94d7110d21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.190+0000 7f638a7fc640 1 -- 172.21.15.27:0/631229071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398005bd0 con 0x55e23558da30 2024-04-25T01:32:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f638a7fc640 1 -- 172.21.15.27:0/631229071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6398010e20 con 0x55e23558da30 2024-04-25T01:32:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f638a7fc640 1 -- 172.21.15.27:0/631229071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639802b340 con 0x55e23558da30 2024-04-25T01:32:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 -- 172.21.15.27:0/631229071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 msgr2=0x55e235579970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e235579970 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f6398009790 tx=0x7f639800bf90 comp rx=0 tx=0).stop 2024-04-25T01:32:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 -- 172.21.15.27:0/631229071 shutdown_connections 2024-04-25T01:32:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23558da30 0x55e235579970 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23557ee50 0x55e2355993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/631229071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e23558d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 -- 172.21.15.27:0/631229071 >> 172.21.15.27:0/631229071 conn(0x55e23557a950 msgr2=0x55e235579260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 -- 172.21.15.27:0/631229071 shutdown_connections 2024-04-25T01:32:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 -- 172.21.15.27:0/631229071 wait complete. 2024-04-25T01:32:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.191+0000 7f63a9fcb740 1 -- start start 2024-04-25T01:32:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a9fcb740 1 Processor -- start 2024-04-25T01:32:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e235595cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23557ee50 0x55e235591040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a9fcb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23558da30 0x55e235591580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01ab120 con 0x55e23557ee50 2024-04-25T01:32:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01ac590 con 0x55e23558da30 2024-04-25T01:32:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a9fcb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01ab7f0 con 0x55e23557e530 2024-04-25T01:32:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a9fcb740 10 client.0 _ll_register_callbacks cb 0x7f6398009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f639ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23557ee50 0x55e235591040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f639ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23557ee50 0x55e235591040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48106/0 (socket says 172.21.15.27:48106) 2024-04-25T01:32:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f639ffff640 1 -- 172.21.15.27:0/2277655231 learned_addr learned my addr 172.21.15.27:0/2277655231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a5786640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23558da30 0x55e235591580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a4f85640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e235595cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f639ffff640 1 -- 172.21.15.27:0/2277655231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 msgr2=0x55e235595cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f639ffff640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e235595cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f639ffff640 1 -- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23558da30 msgr2=0x55e235591580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f639ffff640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23558da30 0x55e235591580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f639ffff640 1 -- 172.21.15.27:0/2277655231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2355856c0 con 0x55e23557ee50 2024-04-25T01:32:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a5786640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23558da30 0x55e235591580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.192+0000 7f63a4f85640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e235595cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f639ffff640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23557ee50 0x55e235591040 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f638c0bbb70 tx=0x7f638c0c14b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bedefee9493c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f637ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f637ffff640 1 -- 172.21.15.27:0/2277655231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c049040 con 0x55e23557ee50 2024-04-25T01:32:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f637ffff640 1 -- 172.21.15.27:0/2277655231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f638c0ed320 con 0x55e23557ee50 2024-04-25T01:32:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2277655231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6398010e20 con 0x55e23557ee50 2024-04-25T01:32:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f637ffff640 1 -- 172.21.15.27:0/2277655231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f638c0c0220 con 0x55e23557ee50 2024-04-25T01:32:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 10 client.17133 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2277655231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e235603d80 con 0x55e23557ee50 2024-04-25T01:32:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f637ffff640 1 -- 172.21.15.27:0/2277655231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6398010e20 con 0x55e23557ee50 2024-04-25T01:32:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f637ffff640 1 -- 172.21.15.27:0/2277655231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f638c031bd0 con 0x55e23557ee50 2024-04-25T01:32:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 10 client.17133 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 10 client.17133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 20 client.17133 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e2356041c0 0x55e2356065e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 4 client.17133 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:32:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.193+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2277655231 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e2354eaf40 con 0x55e2356041c0 2024-04-25T01:32:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.194+0000 7f63a4f85640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e2356041c0 0x55e2356065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.194+0000 7f637ffff640 10 client.17133 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.194+0000 7f63a4f85640 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e2356041c0 0x55e2356065e0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d138431044983b5e server_cookie=3782f2ef8e2b83f8 in_seq=0 out_seq=0 2024-04-25T01:32:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.194+0000 7f637ffff640 1 -- 172.21.15.27:0/2277655231 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e2354eaf40 con 0x55e2356041c0 2024-04-25T01:32:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.194+0000 7f637ffff640 10 client.17133 handle_command_reply: tid=0 2024-04-25T01:32:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.194+0000 7f63a9fcb740 1 client.17133 shutdown 2024-04-25T01:32:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2277655231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23557ee50 msgr2=0x55e235591040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23557ee50 0x55e235591040 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f638c0bbb70 tx=0x7f638c0c14b0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2277655231 shutdown_connections 2024-04-25T01:32:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55e2356041c0 0x55e2356065e0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55e23558da30 0x55e235591580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e23557ee50 0x55e235591040 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 --2- 172.21.15.27:0/2277655231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e23557e530 0x55e235595cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f637ffff640 0 client.17133 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2277655231 >> 172.21.15.27:0/2277655231 conn(0x55e23557a950 msgr2=0x55e23558b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2277655231 shutdown_connections 2024-04-25T01:32:31.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 1 -- 172.21.15.27:0/2277655231 wait complete. 2024-04-25T01:32:31.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a9fcb740 20 client.17133 trim_cache size 0 max 16384 2024-04-25T01:32:31.196 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:32:31.197 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:32:31.197 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:32:31.197 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:32:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a71e9640 1 -- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f63900a2d00 msgr2=0x7f63900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a71e9640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f63900a2d00 0x7f63900a51c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6398009fd0 tx=0x7f63980096a0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a71e9640 1 -- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a006a6c0 msgr2=0x7f63a01985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a71e9640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a006a6c0 0x7f63a01985a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f638c009230 tx=0x7f638c004270 comp rx=0 tx=0).stop 2024-04-25T01:32:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a71e9640 1 -- 172.21.15.27:0/757520064 shutdown_connections 2024-04-25T01:32:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.195+0000 7f63a71e9640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63a0107720 0x7f63a019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.196+0000 7f63a71e9640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f63900a2d00 0x7f63900a51c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.196+0000 7f63a71e9640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a006a6c0 0x7f63a01985a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.196+0000 7f63a71e9640 1 --2- 172.21.15.27:0/757520064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f63a0069d40 0x7f63a0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.196+0000 7f63a71e9640 1 -- 172.21.15.27:0/757520064 >> 172.21.15.27:0/757520064 conn(0x7f63a0076440 msgr2=0x7f63a00fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.196+0000 7f63a71e9640 1 -- 172.21.15.27:0/757520064 shutdown_connections 2024-04-25T01:32:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.196+0000 7f63a71e9640 1 -- 172.21.15.27:0/757520064 wait complete. 2024-04-25T01:32:31.199 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:32:31.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:32:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 -- 172.21.15.27:0/4234230575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 msgr2=0x7fd95c10e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 --2- 172.21.15.27:0/4234230575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 0x7fd95c10e1d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd954009ca0 tx=0x7fd95402a950 comp rx=0 tx=0).stop 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 -- 172.21.15.27:0/4234230575 shutdown_connections 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 --2- 172.21.15.27:0/4234230575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 0x7fd95c10e1d0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 --2- 172.21.15.27:0/4234230575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd95c103340 0x7fd95c10b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 --2- 172.21.15.27:0/4234230575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd95c102a20 0x7fd95c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 -- 172.21.15.27:0/4234230575 >> 172.21.15.27:0/4234230575 conn(0x7fd95c0777a0 msgr2=0x7fd95c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 -- 172.21.15.27:0/4234230575 shutdown_connections 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.324+0000 7fd96444d640 1 -- 172.21.15.27:0/4234230575 wait complete. 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd96444d640 1 Processor -- start 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd96444d640 1 -- start start 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd96444d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd95c102a20 0x7fd95c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd96444d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd95c103340 0x7fd95c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd96444d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 0x7fd95c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd96444d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c111b30 con 0x7fd95c10bd60 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd96444d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c1119b0 con 0x7fd95c102a20 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd96444d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c111cb0 con 0x7fd95c103340 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd9629ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 0x7fd95c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd9629ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 0x7fd95c19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48126/0 (socket says 172.21.15.27:48126) 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd9629ea640 1 -- 172.21.15.27:0/2466280141 learned_addr learned my addr 172.21.15.27:0/2466280141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd9621e9640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd95c102a20 0x7fd95c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd9619e8640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd95c103340 0x7fd95c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd9629ea640 1 -- 172.21.15.27:0/2466280141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd95c103340 msgr2=0x7fd95c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.325+0000 7fd9629ea640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd95c103340 0x7fd95c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd9629ea640 1 -- 172.21.15.27:0/2466280141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd95c102a20 msgr2=0x7fd95c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd9629ea640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd95c102a20 0x7fd95c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd9629ea640 1 -- 172.21.15.27:0/2466280141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd95c19cef0 con 0x7fd95c10bd60 2024-04-25T01:32:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd9619e8640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd95c103340 0x7fd95c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd9629ea640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 0x7fd95c19c7c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd954004af0 tx=0x7fd95402f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd9537fe640 1 -- 172.21.15.27:0/2466280141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954034040 con 0x7fd95c10bd60 2024-04-25T01:32:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd9537fe640 1 -- 172.21.15.27:0/2466280141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd954030920 con 0x7fd95c10bd60 2024-04-25T01:32:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd95c19d180 con 0x7fd95c10bd60 2024-04-25T01:32:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd9537fe640 1 -- 172.21.15.27:0/2466280141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954004740 con 0x7fd95c10bd60 2024-04-25T01:32:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.326+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd95c1a4a70 con 0x7fd95c10bd60 2024-04-25T01:32:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.327+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd95c0689b0 con 0x7fd95c10bd60 2024-04-25T01:32:31.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.328+0000 7fd9537fe640 1 -- 172.21.15.27:0/2466280141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd954037020 con 0x7fd95c10bd60 2024-04-25T01:32:31.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.328+0000 7fd9537fe640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd9480a2d80 0x7fd9480a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.329+0000 7fd9621e9640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd9480a2d80 0x7fd9480a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.329+0000 7fd9537fe640 1 -- 172.21.15.27:0/2466280141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fd9540f0520 con 0x7fd95c10bd60 2024-04-25T01:32:31.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.329+0000 7fd9621e9640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd9480a2d80 0x7fd9480a5240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd95800abe0 tx=0x7fd95800a2d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.334+0000 7fd9537fe640 1 -- 172.21.15.27:0/2466280141 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd9540bce60 con 0x7fd95c10bd60 2024-04-25T01:32:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.558+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd95c1a4c70 con 0x7fd95c10bd60 2024-04-25T01:32:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.559+0000 7fd9537fe640 1 -- 172.21.15.27:0/2466280141 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7fd9540bc760 con 0x7fd95c10bd60 2024-04-25T01:32:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:32:31.537 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:32:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:32:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd9480a2d80 msgr2=0x7fd9480a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd9480a2d80 0x7fd9480a5240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd95800abe0 tx=0x7fd95800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 msgr2=0x7fd95c19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 0x7fd95c19c7c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd954004af0 tx=0x7fd95402f5b0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 shutdown_connections 2024-04-25T01:32:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd9480a2d80 0x7fd9480a5240 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd95c10bd60 0x7fd95c19c7c0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd95c103340 0x7fd95c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 --2- 172.21.15.27:0/2466280141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd95c102a20 0x7fd95c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 >> 172.21.15.27:0/2466280141 conn(0x7fd95c0777a0 msgr2=0x7fd95c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 shutdown_connections 2024-04-25T01:32:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.561+0000 7fd96444d640 1 -- 172.21.15.27:0/2466280141 wait complete. 2024-04-25T01:32:31.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:32:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 -- 172.21.15.27:0/41662577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c106630 msgr2=0x7fba6c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 --2- 172.21.15.27:0/41662577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c106630 0x7fba6c106a10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fba60009930 tx=0x7fba6002a420 comp rx=0 tx=0).stop 2024-04-25T01:32:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 -- 172.21.15.27:0/41662577 shutdown_connections 2024-04-25T01:32:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 --2- 172.21.15.27:0/41662577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba6c100fe0 0x7fba6c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 --2- 172.21.15.27:0/41662577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6c100620 0x7fba6c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 --2- 172.21.15.27:0/41662577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c106630 0x7fba6c106a10 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 -- 172.21.15.27:0/41662577 >> 172.21.15.27:0/41662577 conn(0x7fba6c0776a0 msgr2=0x7fba6c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 -- 172.21.15.27:0/41662577 shutdown_connections 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.676+0000 7fba72dd5640 1 -- 172.21.15.27:0/41662577 wait complete. 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba72dd5640 1 Processor -- start 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba72dd5640 1 -- start start 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba72dd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c100620 0x7fba6c0fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba72dd5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba6c100fe0 0x7fba6c0fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba72dd5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6c106630 0x7fba6c0fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba72dd5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c110a40 con 0x7fba6c100620 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba72dd5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1108c0 con 0x7fba6c100fe0 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba72dd5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c110bc0 con 0x7fba6c106630 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba70b71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c100620 0x7fba6c0fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba70b71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c100620 0x7fba6c0fd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48134/0 (socket says 172.21.15.27:48134) 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba70b71640 1 -- 172.21.15.27:0/347795841 learned_addr learned my addr 172.21.15.27:0/347795841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba6bfff640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba6c100fe0 0x7fba6c0fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba71372640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6c106630 0x7fba6c0fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba70b71640 1 -- 172.21.15.27:0/347795841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6c106630 msgr2=0x7fba6c0fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba70b71640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6c106630 0x7fba6c0fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba70b71640 1 -- 172.21.15.27:0/347795841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba6c100fe0 msgr2=0x7fba6c0fdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba70b71640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba6c100fe0 0x7fba6c0fdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.677+0000 7fba70b71640 1 -- 172.21.15.27:0/347795841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6c1a64d0 con 0x7fba6c100620 2024-04-25T01:32:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.678+0000 7fba70b71640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c100620 0x7fba6c0fd650 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fba60009230 tx=0x7fba60004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.678+0000 7fba69ffb640 1 -- 172.21.15.27:0/347795841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6003f030 con 0x7fba6c100620 2024-04-25T01:32:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.678+0000 7fba72dd5640 1 -- 172.21.15.27:0/347795841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6c1a6760 con 0x7fba6c100620 2024-04-25T01:32:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.678+0000 7fba69ffb640 1 -- 172.21.15.27:0/347795841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba60004440 con 0x7fba6c100620 2024-04-25T01:32:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.678+0000 7fba69ffb640 1 -- 172.21.15.27:0/347795841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60035070 con 0x7fba6c100620 2024-04-25T01:32:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.678+0000 7fba72dd5640 1 -- 172.21.15.27:0/347795841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6c1a6c50 con 0x7fba6c100620 2024-04-25T01:32:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.680+0000 7fba69ffb640 1 -- 172.21.15.27:0/347795841 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fba60042030 con 0x7fba6c100620 2024-04-25T01:32:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba75bb6740 1 Processor -- start 2024-04-25T01:32:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba75bb6740 1 -- start start 2024-04-25T01:32:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x556016904960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x556016911960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba75bb6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556016869670 con 0x556016904580 2024-04-25T01:32:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba75bb6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556016811970 con 0x556016904ea0 2024-04-25T01:32:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba75bb6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556016865630 con 0x55601690f570 2024-04-25T01:32:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x556016904960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba69ffb640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fba5c0a2d80 0x7fba5c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x556016904960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48150/0 (socket says 172.21.15.27:48150) 2024-04-25T01:32:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 -- 172.21.15.27:0/2095125770 learned_addr learned my addr 172.21.15.27:0/2095125770 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba69ffb640 1 -- 172.21.15.27:0/347795841 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fba600efce0 con 0x7fba6c100620 2024-04-25T01:32:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba70b71640 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fba5c0a2d80 0x7fba5c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba6bfff640 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x556016911960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 -- 172.21.15.27:0/2095125770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 msgr2=0x556016911960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x556016911960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 -- 172.21.15.27:0/2095125770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 msgr2=0x55601690f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba71372640 1 -- 172.21.15.27:0/2095125770 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556016835400 con 0x556016904580 2024-04-25T01:32:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.681+0000 7fba70b71640 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba71372640 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x556016904960 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fba64013910 tx=0x7fba64013dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca72c6c540ebbe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba56ffd640 1 -- 172.21.15.27:0/2095125770 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64023020 con 0x556016904580 2024-04-25T01:32:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba56ffd640 1 -- 172.21.15.27:0/2095125770 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba64013250 con 0x556016904580 2024-04-25T01:32:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba56ffd640 1 -- 172.21.15.27:0/2095125770 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6401f690 con 0x556016904580 2024-04-25T01:32:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba71372640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fba5c0a2d80 0x7fba5c0a5240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba64021000 tx=0x7fba64020000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba75bb6740 1 -- 172.21.15.27:0/2095125770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 msgr2=0x556016904960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba75bb6740 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x556016904960 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fba64013910 tx=0x7fba64013dd0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba75bb6740 1 -- 172.21.15.27:0/2095125770 shutdown_connections 2024-04-25T01:32:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba75bb6740 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x556016911960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba75bb6740 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.682+0000 7fba75bb6740 1 --2- 172.21.15.27:0/2095125770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x556016904960 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 -- 172.21.15.27:0/2095125770 >> 172.21.15.27:0/2095125770 conn(0x5560169008a0 msgr2=0x556016900c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 -- 172.21.15.27:0/2095125770 shutdown_connections 2024-04-25T01:32:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 -- 172.21.15.27:0/2095125770 wait complete. 2024-04-25T01:32:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 -- start start 2024-04-25T01:32:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 Processor -- start 2024-04-25T01:32:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904580 0x55601690eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904ea0 0x55601690c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x55601690c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556016912360 con 0x556016904ea0 2024-04-25T01:32:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556016825050 con 0x556016904580 2024-04-25T01:32:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5560169124e0 con 0x55601690f570 2024-04-25T01:32:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba75bb6740 10 client.0 _ll_register_callbacks cb 0x55601657cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba6bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904ea0 0x55601690c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba6bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904ea0 0x55601690c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48154/0 (socket says 172.21.15.27:48154) 2024-04-25T01:32:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.683+0000 7fba6bfff640 1 -- 172.21.15.27:0/4083424210 learned_addr learned my addr 172.21.15.27:0/4083424210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba71372640 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x55601690c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba70b71640 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904580 0x55601690eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba6bfff640 1 -- 172.21.15.27:0/4083424210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 msgr2=0x55601690c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba6bfff640 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x55601690c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba6bfff640 1 -- 172.21.15.27:0/4083424210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904580 msgr2=0x55601690eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba6bfff640 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904580 0x55601690eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba6bfff640 1 -- 172.21.15.27:0/4083424210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560167f6dc0 con 0x556016904ea0 2024-04-25T01:32:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba6bfff640 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904ea0 0x55601690c1d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fba58013710 tx=0x7fba58013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772867459e882d0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:31.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba4bfff640 1 -- 172.21.15.27:0/4083424210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58020060 con 0x556016904ea0 2024-04-25T01:32:31.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba4bfff640 1 -- 172.21.15.27:0/4083424210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fba58011dd0 con 0x556016904ea0 2024-04-25T01:32:31.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba75bb6740 1 -- 172.21.15.27:0/4083424210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba64013250 con 0x556016904ea0 2024-04-25T01:32:31.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba4bfff640 1 -- 172.21.15.27:0/4083424210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fba58027050 con 0x556016904ea0 2024-04-25T01:32:31.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba75bb6740 10 client.17151 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:31.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba75bb6740 1 -- 172.21.15.27:0/4083424210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55601657d3f0 con 0x556016904ea0 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.684+0000 7fba4bfff640 1 -- 172.21.15.27:0/4083424210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba64013250 con 0x556016904ea0 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.685+0000 7fba4bfff640 1 -- 172.21.15.27:0/4083424210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fba5801a030 con 0x556016904ea0 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.685+0000 7fba75bb6740 10 client.17151 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.685+0000 7fba75bb6740 10 client.17151 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.685+0000 7fba75bb6740 20 client.17151 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.685+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55601691f670 0x556016921aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.685+0000 7fba75bb6740 4 client.17151 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.685+0000 7fba75bb6740 1 -- 172.21.15.27:0/4083424210 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556016876bb0 con 0x55601691f670 2024-04-25T01:32:31.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.685+0000 7fba70b71640 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55601691f670 0x556016921aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba4bfff640 10 client.17151 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:32:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba70b71640 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55601691f670 0x556016921aa0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9cc15cb0425588db server_cookie=eead40ffa65b4f50 in_seq=0 out_seq=0 2024-04-25T01:32:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba4bfff640 1 -- 172.21.15.27:0/4083424210 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556016876bb0 con 0x55601691f670 2024-04-25T01:32:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba4bfff640 10 client.17151 handle_command_reply: tid=0 2024-04-25T01:32:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 client.17151 shutdown 2024-04-25T01:32:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 -- 172.21.15.27:0/4083424210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904ea0 msgr2=0x55601690c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904ea0 0x55601690c1d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fba58013710 tx=0x7fba58013bd0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 -- 172.21.15.27:0/4083424210 shutdown_connections 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55601691f670 0x556016921aa0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x55601690c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904ea0 0x55601690c1d0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba4bfff640 0 client.17151 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4083424210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904580 0x55601690eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.686+0000 7fba75bb6740 1 -- 172.21.15.27:0/4083424210 >> 172.21.15.27:0/4083424210 conn(0x5560169008a0 msgr2=0x5560168ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.687+0000 7fba75bb6740 1 -- 172.21.15.27:0/4083424210 shutdown_connections 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.687+0000 7fba75bb6740 1 -- 172.21.15.27:0/4083424210 wait complete. 2024-04-25T01:32:31.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.687+0000 7fba75bb6740 20 client.17151 trim_cache size 0 max 16384 2024-04-25T01:32:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba75bb6740 1 Processor -- start 2024-04-25T01:32:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba75bb6740 1 -- start start 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x55601691b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x55601690ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba75bb6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1b3750 con 0x556016904580 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba75bb6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1b1020 con 0x556016904ea0 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba75bb6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1aee90 con 0x55601690f570 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba70b71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x55601691b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba70b71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x55601691b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48166/0 (socket says 172.21.15.27:48166) 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.700+0000 7fba70b71640 1 -- 172.21.15.27:0/506496497 learned_addr learned my addr 172.21.15.27:0/506496497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba70b71640 1 -- 172.21.15.27:0/506496497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 msgr2=0x55601690ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba6bfff640 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba70b71640 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x55601690ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba70b71640 1 -- 172.21.15.27:0/506496497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 msgr2=0x55601690ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba70b71640 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba70b71640 1 -- 172.21.15.27:0/506496497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556016904cd0 con 0x556016904580 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba6bfff640 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba70b71640 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x55601691b220 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fba600b8210 tx=0x7fba600f20e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638ab70b41c53887 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba567fc640 1 -- 172.21.15.27:0/506496497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60049040 con 0x556016904580 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba567fc640 1 -- 172.21.15.27:0/506496497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba600c0160 con 0x556016904580 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba567fc640 1 -- 172.21.15.27:0/506496497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600ed260 con 0x556016904580 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba75bb6740 1 -- 172.21.15.27:0/506496497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 msgr2=0x55601691b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba75bb6740 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x55601691b220 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fba600b8210 tx=0x7fba600f20e0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba75bb6740 1 -- 172.21.15.27:0/506496497 shutdown_connections 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba75bb6740 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55601690f570 0x55601690ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba75bb6740 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556016904ea0 0x55601690ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba75bb6740 1 --2- 172.21.15.27:0/506496497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x55601691b220 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.701+0000 7fba75bb6740 1 -- 172.21.15.27:0/506496497 >> 172.21.15.27:0/506496497 conn(0x5560169008a0 msgr2=0x556016906560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 -- 172.21.15.27:0/506496497 shutdown_connections 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 -- 172.21.15.27:0/506496497 wait complete. 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 -- start start 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 Processor -- start 2024-04-25T01:32:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x5560169173d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556016904ea0 0x556016917910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55601690f570 0x556016913870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1af4e0 con 0x556016904580 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1b0950 con 0x55601690f570 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1afbb0 con 0x556016904ea0 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba75bb6740 10 client.0 _ll_register_callbacks cb 0x7fba58009690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba70b71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x5560169173d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba70b71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x5560169173d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48172/0 (socket says 172.21.15.27:48172) 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.702+0000 7fba70b71640 1 -- 172.21.15.27:0/4259472609 learned_addr learned my addr 172.21.15.27:0/4259472609 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba71372640 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55601690f570 0x556016913870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba70b71640 1 -- 172.21.15.27:0/4259472609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556016904ea0 msgr2=0x556016917910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba70b71640 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556016904ea0 0x556016917910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba70b71640 1 -- 172.21.15.27:0/4259472609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55601690f570 msgr2=0x556016913870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba70b71640 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55601690f570 0x556016913870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba70b71640 1 -- 172.21.15.27:0/4259472609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560166477d0 con 0x556016904580 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba71372640 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55601690f570 0x556016913870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba70b71640 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x5560169173d0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fba60000c00 tx=0x7fba60009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5d2e39b597ce09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba4bfff640 1 -- 172.21.15.27:0/4259472609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600d9060 con 0x556016904580 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba4bfff640 1 -- 172.21.15.27:0/4259472609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fba600b8420 con 0x556016904580 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba75bb6740 1 -- 172.21.15.27:0/4259472609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba600c0160 con 0x556016904580 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba4bfff640 1 -- 172.21.15.27:0/4259472609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fba60049070 con 0x556016904580 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba75bb6740 10 client.17157 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba75bb6740 1 -- 172.21.15.27:0/4259472609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5560169863c0 con 0x556016904580 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba4bfff640 1 -- 172.21.15.27:0/4259472609 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba600c0160 con 0x556016904580 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba4bfff640 1 -- 172.21.15.27:0/4259472609 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fba600a6040 con 0x556016904580 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba75bb6740 10 client.17157 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.703+0000 7fba75bb6740 10 client.17157 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 20 client.17157 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556016986970 0x556016988d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 4 client.17157 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 -- 172.21.15.27:0/4259472609 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556016870f00 con 0x556016986970 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba6bfff640 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556016986970 0x556016988d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba6bfff640 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556016986970 0x556016988d90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad64261fde78244c server_cookie=52935693c3e61e15 in_seq=0 out_seq=0 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba4bfff640 10 client.17157 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba4bfff640 1 -- 172.21.15.27:0/4259472609 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556016870f00 con 0x556016986970 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba4bfff640 10 client.17157 handle_command_reply: tid=0 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 client.17157 shutdown 2024-04-25T01:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 -- 172.21.15.27:0/4259472609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 msgr2=0x5560169173d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x5560169173d0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fba60000c00 tx=0x7fba60009260 comp rx=0 tx=0).stop 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 -- 172.21.15.27:0/4259472609 shutdown_connections 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556016986970 0x556016988d90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55601690f570 0x556016913870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba4bfff640 0 client.17157 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556016904ea0 0x556016917910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.704+0000 7fba75bb6740 1 --2- 172.21.15.27:0/4259472609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556016904580 0x5560169173d0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba75bb6740 1 -- 172.21.15.27:0/4259472609 >> 172.21.15.27:0/4259472609 conn(0x5560169008a0 msgr2=0x556016906de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba75bb6740 1 -- 172.21.15.27:0/4259472609 shutdown_connections 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba75bb6740 1 -- 172.21.15.27:0/4259472609 wait complete. 2024-04-25T01:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba75bb6740 20 client.17157 trim_cache size 0 max 16384 2024-04-25T01:32:31.692 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:32:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 -- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fba5c0a2d80 msgr2=0x7fba5c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fba5c0a2d80 0x7fba5c0a5240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba64021000 tx=0x7fba64020000 comp rx=0 tx=0).stop 2024-04-25T01:32:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 -- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c100620 msgr2=0x7fba6c0fd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c100620 0x7fba6c0fd650 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fba60009230 tx=0x7fba60004040 comp rx=0 tx=0).stop 2024-04-25T01:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 -- 172.21.15.27:0/347795841 shutdown_connections 2024-04-25T01:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba6c106630 0x7fba6c0fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba6c100fe0 0x7fba6c0fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fba5c0a2d80 0x7fba5c0a5240 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 --2- 172.21.15.27:0/347795841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c100620 0x7fba6c0fd650 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 -- 172.21.15.27:0/347795841 >> 172.21.15.27:0/347795841 conn(0x7fba6c0776a0 msgr2=0x7fba6c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 -- 172.21.15.27:0/347795841 shutdown_connections 2024-04-25T01:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.705+0000 7fba72dd5640 1 -- 172.21.15.27:0/347795841 wait complete. 2024-04-25T01:32:31.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:32:31.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 -- 172.21.15.27:0/3611751106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 msgr2=0x7ff874107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 --2- 172.21.15.27:0/3611751106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 0x7ff874107b00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7ff870009930 tx=0x7ff87002a420 comp rx=0 tx=0).stop 2024-04-25T01:32:31.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 -- 172.21.15.27:0/3611751106 shutdown_connections 2024-04-25T01:32:31.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 --2- 172.21.15.27:0/3611751106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff874100600 0x7ff87410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 --2- 172.21.15.27:0/3611751106 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8740ffc40 0x7ff8741000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 --2- 172.21.15.27:0/3611751106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 0x7ff874107b00 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 -- 172.21.15.27:0/3611751106 >> 172.21.15.27:0/3611751106 conn(0x7ff874076bd0 msgr2=0x7ff8740fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 -- 172.21.15.27:0/3611751106 shutdown_connections 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.820+0000 7ff87b510640 1 -- 172.21.15.27:0/3611751106 wait complete. 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff87b510640 1 Processor -- start 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff87b510640 1 -- start start 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff87b510640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8740ffc40 0x7ff874198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff87b510640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff874100600 0x7ff874198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff87b510640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 0x7ff87419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff87b510640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff874111af0 con 0x7ff874107720 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff87b510640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff874111970 con 0x7ff874100600 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff87b510640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff874111c70 con 0x7ff8740ffc40 2024-04-25T01:32:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff879aad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 0x7ff87419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff879aad640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 0x7ff87419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48184/0 (socket says 172.21.15.27:48184) 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff879aad640 1 -- 172.21.15.27:0/3968894132 learned_addr learned my addr 172.21.15.27:0/3968894132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff8792ac640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8740ffc40 0x7ff874198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.821+0000 7ff878aab640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff874100600 0x7ff874198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff879aad640 1 -- 172.21.15.27:0/3968894132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8740ffc40 msgr2=0x7ff874198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff879aad640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8740ffc40 0x7ff874198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff879aad640 1 -- 172.21.15.27:0/3968894132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff874100600 msgr2=0x7ff874198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff879aad640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff874100600 0x7ff874198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff879aad640 1 -- 172.21.15.27:0/3968894132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87419d040 con 0x7ff874107720 2024-04-25T01:32:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff8792ac640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8740ffc40 0x7ff874198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff878aab640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff874100600 0x7ff874198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff879aad640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 0x7ff87419c910 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009fd0 tx=0x7ff86c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff8667fc640 1 -- 172.21.15.27:0/3968894132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c017070 con 0x7ff874107720 2024-04-25T01:32:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff87b510640 1 -- 172.21.15.27:0/3968894132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87419d330 con 0x7ff874107720 2024-04-25T01:32:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff8667fc640 1 -- 172.21.15.27:0/3968894132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff86c003e50 con 0x7ff874107720 2024-04-25T01:32:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff8667fc640 1 -- 172.21.15.27:0/3968894132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c0106c0 con 0x7ff874107720 2024-04-25T01:32:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.822+0000 7ff87b510640 1 -- 172.21.15.27:0/3968894132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8741a4bc0 con 0x7ff874107720 2024-04-25T01:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.824+0000 7ff8667fc640 1 -- 172.21.15.27:0/3968894132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff86c003970 con 0x7ff874107720 2024-04-25T01:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff87e218740 1 Processor -- start 2024-04-25T01:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff87e218740 1 -- start start 2024-04-25T01:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff87e218740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 0x55ce09895960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff87e218740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 0x55ce098a0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff87e218740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce098a2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff87e218740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce097fa670 con 0x55ce098a0570 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff87e218740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ce097a2970 con 0x55ce09895580 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff87e218740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ce097f6630 con 0x55ce09895ea0 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff8667fc640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8600a2d00 0x7ff8600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff878aab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce098a2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff878aab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce098a2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48190/0 (socket says 172.21.15.27:48190) 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff878aab640 1 -- 172.21.15.27:0/2444345060 learned_addr learned my addr 172.21.15.27:0/2444345060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff8792ac640 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 0x55ce098a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff879aad640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8600a2d00 0x7ff8600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff878aab640 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 0x55ce09895960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.825+0000 7ff8667fc640 1 -- 172.21.15.27:0/3968894132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7ff86c0cf620 con 0x7ff874107720 2024-04-25T01:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff878aab640 1 -- 172.21.15.27:0/2444345060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 msgr2=0x55ce098a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff878aab640 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 0x55ce098a0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff878aab640 1 -- 172.21.15.27:0/2444345060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 msgr2=0x55ce09895960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff878aab640 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 0x55ce09895960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff878aab640 1 -- 172.21.15.27:0/2444345060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce097c6400 con 0x55ce098a0570 2024-04-25T01:32:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff878aab640 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce098a2960 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff868012830 tx=0x7ff868009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f92c1a5d9b65ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff879aad640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8600a2d00 0x7ff8600a51c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff86c097bb0 tx=0x7ff86c086040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff85f7fe640 1 -- 172.21.15.27:0/2444345060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8680107d0 con 0x55ce098a0570 2024-04-25T01:32:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff85f7fe640 1 -- 172.21.15.27:0/2444345060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff868010e50 con 0x55ce098a0570 2024-04-25T01:32:31.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.826+0000 7ff85f7fe640 1 -- 172.21.15.27:0/2444345060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86801ebf0 con 0x55ce098a0570 2024-04-25T01:32:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 -- 172.21.15.27:0/2444345060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 msgr2=0x55ce098a2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce098a2960 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff868012830 tx=0x7ff868009760 comp rx=0 tx=0).stop 2024-04-25T01:32:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 -- 172.21.15.27:0/2444345060 shutdown_connections 2024-04-25T01:32:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce098a2960 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 0x55ce098a0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 --2- 172.21.15.27:0/2444345060 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 0x55ce09895960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 -- 172.21.15.27:0/2444345060 >> 172.21.15.27:0/2444345060 conn(0x55ce098918a0 msgr2=0x55ce09891c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 -- 172.21.15.27:0/2444345060 shutdown_connections 2024-04-25T01:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 -- 172.21.15.27:0/2444345060 wait complete. 2024-04-25T01:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 -- start start 2024-04-25T01:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 Processor -- start 2024-04-25T01:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 0x55ce0989fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce0989d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.827+0000 7ff87e218740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce0989d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff87e218740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce098a3360 con 0x55ce09895ea0 2024-04-25T01:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff87e218740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ce097b6050 con 0x55ce098a0570 2024-04-25T01:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff87e218740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ce098a34e0 con 0x55ce09895580 2024-04-25T01:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff87e218740 10 client.0 _ll_register_callbacks cb 0x55ce0950dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce0989d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce0989d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48194/0 (socket says 172.21.15.27:48194) 2024-04-25T01:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 -- 172.21.15.27:0/2899794117 learned_addr learned my addr 172.21.15.27:0/2899794117 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff878aab640 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce0989d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff878aab640 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 0x55ce0989fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 -- 172.21.15.27:0/2899794117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 msgr2=0x55ce0989fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 0x55ce0989fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 -- 172.21.15.27:0/2899794117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 msgr2=0x55ce0989d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce0989d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 -- 172.21.15.27:0/2899794117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce09787dc0 con 0x55ce09895ea0 2024-04-25T01:32:31.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff878aab640 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce0989d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff8792ac640 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce0989d170 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff870009930 tx=0x7ff870031860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7aa7719524cf54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff85cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:31.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff85cff9640 1 -- 172.21.15.27:0/2899794117 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870035040 con 0x55ce09895ea0 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff85cff9640 1 -- 172.21.15.27:0/2899794117 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7ff870004240 con 0x55ce09895ea0 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff87e218740 1 -- 172.21.15.27:0/2899794117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff868010e50 con 0x55ce09895ea0 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff85cff9640 1 -- 172.21.15.27:0/2899794117 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff87004e050 con 0x55ce09895ea0 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff87e218740 10 client.17169 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.828+0000 7ff87e218740 1 -- 172.21.15.27:0/2899794117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce0950e3f0 con 0x55ce09895ea0 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff85cff9640 1 -- 172.21.15.27:0/2899794117 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff868010e50 con 0x55ce09895ea0 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff85cff9640 1 -- 172.21.15.27:0/2899794117 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff8700049c0 con 0x55ce09895ea0 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff87e218740 10 client.17169 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff87e218740 10 client.17169 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff87e218740 20 client.17169 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff87e218740 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55ce098b0610 0x55ce098b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff87e218740 4 client.17169 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff87e218740 1 -- 172.21.15.27:0/2899794117 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce09807bb0 con 0x55ce098b0610 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.829+0000 7ff878aab640 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55ce098b0610 0x55ce098b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff85cff9640 10 client.17169 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff878aab640 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55ce098b0610 0x55ce098b2a30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ec56e2dec7393460 server_cookie=7c9e36747b9b233f in_seq=0 out_seq=0 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff85cff9640 1 -- 172.21.15.27:0/2899794117 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce09807bb0 con 0x55ce098b0610 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff85cff9640 10 client.17169 handle_command_reply: tid=0 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 client.17169 shutdown 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 -- 172.21.15.27:0/2899794117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 msgr2=0x55ce0989d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce0989d170 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff870009930 tx=0x7ff870031860 comp rx=0 tx=0).stop 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 -- 172.21.15.27:0/2899794117 shutdown_connections 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55ce098b0610 0x55ce098b2a30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce0989d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce0989d170 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff85cff9640 0 client.17169 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:32:31.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 --2- 172.21.15.27:0/2899794117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 0x55ce0989fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 -- 172.21.15.27:0/2899794117 >> 172.21.15.27:0/2899794117 conn(0x55ce098918a0 msgr2=0x55ce098976a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.830+0000 7ff87e218740 1 -- 172.21.15.27:0/2899794117 shutdown_connections 2024-04-25T01:32:31.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.831+0000 7ff87e218740 1 -- 172.21.15.27:0/2899794117 wait complete. 2024-04-25T01:32:31.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.831+0000 7ff87e218740 20 client.17169 trim_cache size 0 max 16384 2024-04-25T01:32:31.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff87e218740 1 Processor -- start 2024-04-25T01:32:31.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff87e218740 1 -- start start 2024-04-25T01:32:31.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff87e218740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 0x55ce098a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff87e218740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 0x55ce09890b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff87e218740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce0989f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff87e218740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff8741b15b0 con 0x55ce098a0570 2024-04-25T01:32:31.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff87e218740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff8741aee80 con 0x55ce09895580 2024-04-25T01:32:31.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff87e218740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8741accf0 con 0x55ce09895ea0 2024-04-25T01:32:31.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff878aab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce0989f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff878aab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce0989f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48196/0 (socket says 172.21.15.27:48196) 2024-04-25T01:32:31.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff878aab640 1 -- 172.21.15.27:0/1687513323 learned_addr learned my addr 172.21.15.27:0/1687513323 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.844+0000 7ff8792ac640 1 --2- 172.21.15.27:0/1687513323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 0x55ce09890b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff878aab640 1 -- 172.21.15.27:0/1687513323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 msgr2=0x55ce09890b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff878aab640 1 --2- 172.21.15.27:0/1687513323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 0x55ce09890b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff878aab640 1 -- 172.21.15.27:0/1687513323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 msgr2=0x55ce098a8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff878aab640 1 --2- 172.21.15.27:0/1687513323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 0x55ce098a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff878aab640 1 -- 172.21.15.27:0/1687513323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce095d87d0 con 0x55ce098a0570 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff878aab640 1 --2- 172.21.15.27:0/1687513323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce0989f9b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff868009730 tx=0x7ff868009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa946887e067a27b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff85effd640 1 -- 172.21.15.27:0/1687513323 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868024040 con 0x55ce098a0570 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff85effd640 1 -- 172.21.15.27:0/1687513323 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff868034070 con 0x55ce098a0570 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff85effd640 1 -- 172.21.15.27:0/1687513323 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8680107a0 con 0x55ce098a0570 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff87e218740 1 -- 172.21.15.27:0/1687513323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 msgr2=0x55ce0989f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.845+0000 7ff87e218740 1 --2- 172.21.15.27:0/1687513323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce0989f9b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff868009730 tx=0x7ff868009760 comp rx=0 tx=0).stop 2024-04-25T01:32:31.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 -- 172.21.15.27:0/1687513323 shutdown_connections 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 --2- 172.21.15.27:0/1687513323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce098a0570 0x55ce0989f9b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 --2- 172.21.15.27:0/1687513323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895ea0 0x55ce09890b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 --2- 172.21.15.27:0/1687513323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce09895580 0x55ce098a8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 -- 172.21.15.27:0/1687513323 >> 172.21.15.27:0/1687513323 conn(0x55ce098918a0 msgr2=0x55ce0989d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 -- 172.21.15.27:0/1687513323 shutdown_connections 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 -- 172.21.15.27:0/1687513323 wait complete. 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 -- start start 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 Processor -- start 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 0x55ce098993e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce09899920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce09899e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff8741ad340 con 0x55ce09895ea0 2024-04-25T01:32:31.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff8741ae7b0 con 0x55ce098a0570 2024-04-25T01:32:31.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8741ada10 con 0x55ce09895580 2024-04-25T01:32:31.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff87e218740 10 client.0 _ll_register_callbacks cb 0x7ff870009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:31.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff8792ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce09899920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff8792ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce09899920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48200/0 (socket says 172.21.15.27:48200) 2024-04-25T01:32:31.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.846+0000 7ff8792ac640 1 -- 172.21.15.27:0/3138695405 learned_addr learned my addr 172.21.15.27:0/3138695405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff878aab640 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 0x55ce098993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff8792ac640 1 -- 172.21.15.27:0/3138695405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 msgr2=0x55ce098993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff8792ac640 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 0x55ce098993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff878aab640 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce09899e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff8792ac640 1 -- 172.21.15.27:0/3138695405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 msgr2=0x55ce09899e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff8792ac640 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce09899e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff8792ac640 1 -- 172.21.15.27:0/3138695405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce0989a690 con 0x55ce09895ea0 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff878aab640 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce09899e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff8792ac640 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce09899920 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x55ce0989c9d0 tx=0x7ff8700098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a70c9bc5bf9030 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff85cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff85cff9640 1 -- 172.21.15.27:0/3138695405 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870035070 con 0x55ce09895ea0 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff85cff9640 1 -- 172.21.15.27:0/3138695405 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7ff8700046c0 con 0x55ce09895ea0 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff87e218740 1 -- 172.21.15.27:0/3138695405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff868034070 con 0x55ce09895ea0 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff85cff9640 1 -- 172.21.15.27:0/3138695405 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff870060050 con 0x55ce09895ea0 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff87e218740 10 client.17175 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff87e218740 1 -- 172.21.15.27:0/3138695405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce0991b120 con 0x55ce09895ea0 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.847+0000 7ff85cff9640 1 -- 172.21.15.27:0/3138695405 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff868034070 con 0x55ce09895ea0 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff85cff9640 1 -- 172.21.15.27:0/3138695405 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff870045500 con 0x55ce09895ea0 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff87e218740 10 client.17175 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:31.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff87e218740 10 client.17175 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff87e218740 20 client.17175 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff87e218740 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55ce0991b890 0x55ce0991dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff87e218740 4 client.17175 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff87e218740 1 -- 172.21.15.27:0/3138695405 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce09801f00 con 0x55ce0991b890 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff878aab640 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55ce0991b890 0x55ce0991dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff878aab640 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55ce0991b890 0x55ce0991dcb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d09c4d3148b20b48 server_cookie=dbee9a2babe086ce in_seq=0 out_seq=0 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.848+0000 7ff85cff9640 10 client.17175 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff85cff9640 1 -- 172.21.15.27:0/3138695405 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce09801f00 con 0x55ce0991b890 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff85cff9640 10 client.17175 handle_command_reply: tid=0 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 client.17175 shutdown 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 -- 172.21.15.27:0/3138695405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 msgr2=0x55ce09899920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce09899920 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x55ce0989c9d0 tx=0x7ff8700098a0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 -- 172.21.15.27:0/3138695405 shutdown_connections 2024-04-25T01:32:31.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55ce0991b890 0x55ce0991dcb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ce098a0570 0x55ce09899e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce09895ea0 0x55ce09899920 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 --2- 172.21.15.27:0/3138695405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ce09895580 0x55ce098993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff85cff9640 0 client.17175 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 -- 172.21.15.27:0/3138695405 >> 172.21.15.27:0/3138695405 conn(0x55ce098918a0 msgr2=0x55ce09890b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 -- 172.21.15.27:0/3138695405 shutdown_connections 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 1 -- 172.21.15.27:0/3138695405 wait complete. 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87e218740 20 client.17175 trim_cache size 0 max 16384 2024-04-25T01:32:31.842 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87b510640 1 -- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8600a2d00 msgr2=0x7ff8600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87b510640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8600a2d00 0x7ff8600a51c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff86c097bb0 tx=0x7ff86c086040 comp rx=0 tx=0).stop 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87b510640 1 -- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 msgr2=0x7ff87419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.849+0000 7ff87b510640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 0x7ff87419c910 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009fd0 tx=0x7ff86c00b260 comp rx=0 tx=0).stop 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.850+0000 7ff87b510640 1 -- 172.21.15.27:0/3968894132 shutdown_connections 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.850+0000 7ff87b510640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8600a2d00 0x7ff8600a51c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.850+0000 7ff87b510640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff874107720 0x7ff87419c910 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.850+0000 7ff87b510640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff874100600 0x7ff874198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.850+0000 7ff87b510640 1 --2- 172.21.15.27:0/3968894132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8740ffc40 0x7ff874198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.850+0000 7ff87b510640 1 -- 172.21.15.27:0/3968894132 >> 172.21.15.27:0/3968894132 conn(0x7ff874076bd0 msgr2=0x7ff8740fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.850+0000 7ff87b510640 1 -- 172.21.15.27:0/3968894132 shutdown_connections 2024-04-25T01:32:31.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.850+0000 7ff87b510640 1 -- 172.21.15.27:0/3968894132 wait complete. 2024-04-25T01:32:31.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:32:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 -- 172.21.15.27:0/760027558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c100e60 msgr2=0x7fa01c10dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 --2- 172.21.15.27:0/760027558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c100e60 0x7fa01c10dbb0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa010009900 tx=0x7fa01002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 -- 172.21.15.27:0/760027558 shutdown_connections 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 --2- 172.21.15.27:0/760027558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa01c10e0f0 0x7fa01c1104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 --2- 172.21.15.27:0/760027558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c100e60 0x7fa01c10dbb0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 --2- 172.21.15.27:0/760027558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c100540 0x7fa01c100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 -- 172.21.15.27:0/760027558 >> 172.21.15.27:0/760027558 conn(0x7fa01c077740 msgr2=0x7fa01c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 -- 172.21.15.27:0/760027558 shutdown_connections 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 -- 172.21.15.27:0/760027558 wait complete. 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 Processor -- start 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.972+0000 7fa02347d640 1 -- start start 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa02347d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa01c100540 0x7fa01c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa02347d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c100e60 0x7fa01c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa02347d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c10e0f0 0x7fa01c19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa02347d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c113d20 con 0x7fa01c10e0f0 2024-04-25T01:32:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa02347d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c113ba0 con 0x7fa01c100540 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa02347d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c113ea0 con 0x7fa01c100e60 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c10e0f0 0x7fa01c19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c10e0f0 0x7fa01c19eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48212/0 (socket says 172.21.15.27:48212) 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 -- 172.21.15.27:0/560605587 learned_addr learned my addr 172.21.15.27:0/560605587 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa020a18640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c100e60 0x7fa01c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021219640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa01c100540 0x7fa01c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 -- 172.21.15.27:0/560605587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c100e60 msgr2=0x7fa01c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c100e60 0x7fa01c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 -- 172.21.15.27:0/560605587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa01c100540 msgr2=0x7fa01c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa01c100540 0x7fa01c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 -- 172.21.15.27:0/560605587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c19f230 con 0x7fa01c10e0f0 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021219640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa01c100540 0x7fa01c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa020a18640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c100e60 0x7fa01c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa021a1a640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c10e0f0 0x7fa01c19eb00 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00d720 tx=0x7fa00c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa01a7fc640 1 -- 172.21.15.27:0/560605587 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c017070 con 0x7fa01c10e0f0 2024-04-25T01:32:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa01a7fc640 1 -- 172.21.15.27:0/560605587 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa00c00de30 con 0x7fa01c10e0f0 2024-04-25T01:32:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa02347d640 1 -- 172.21.15.27:0/560605587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa01c19f520 con 0x7fa01c10e0f0 2024-04-25T01:32:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.973+0000 7fa01a7fc640 1 -- 172.21.15.27:0/560605587 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c004d90 con 0x7fa01c10e0f0 2024-04-25T01:32:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.974+0000 7fa02347d640 1 -- 172.21.15.27:0/560605587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa01c1a6db0 con 0x7fa01c10e0f0 2024-04-25T01:32:31.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa01a7fc640 1 -- 172.21.15.27:0/560605587 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa00c004f30 con 0x7fa01c10e0f0 2024-04-25T01:32:31.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa02618c740 1 Processor -- start 2024-04-25T01:32:31.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa02618c740 1 -- start start 2024-04-25T01:32:31.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa02618c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c26960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa02618c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c31030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa02618c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c33960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa02618c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564ed3b8b670 con 0x564ed3c26ea0 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa02618c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564ed3b33970 con 0x564ed3c26580 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.976+0000 7fa02618c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564ed3b87630 con 0x564ed3c31570 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c31030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c31030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48228/0 (socket says 172.21.15.27:48228) 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 -- 172.21.15.27:0/2226191820 learned_addr learned my addr 172.21.15.27:0/2226191820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa01a7fc640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa0040a2d80 0x7fa0040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa020a18640 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c26960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa020a18640 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c33960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021a1a640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa0040a2d80 0x7fa0040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 -- 172.21.15.27:0/2226191820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 msgr2=0x564ed3c33960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c33960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa01a7fc640 1 -- 172.21.15.27:0/560605587 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fa00c0cf090 con 0x7fa01c10e0f0 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 -- 172.21.15.27:0/2226191820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 msgr2=0x564ed3c26960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c26960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 -- 172.21.15.27:0/2226191820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed3b57400 con 0x564ed3c26ea0 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021219640 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c31030 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fa0140103a0 tx=0x7fa014010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebbc88db9e966b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa021a1a640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa0040a2d80 0x7fa0040a5240 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0a3900 tx=0x7fa00c005ce0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa0037fe640 1 -- 172.21.15.27:0/2226191820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014016070 con 0x564ed3c26ea0 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa0037fe640 1 -- 172.21.15.27:0/2226191820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa014012bf0 con 0x564ed3c26ea0 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.977+0000 7fa0037fe640 1 -- 172.21.15.27:0/2226191820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01401bce0 con 0x564ed3c26ea0 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 -- 172.21.15.27:0/2226191820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 msgr2=0x564ed3c31030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c31030 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fa0140103a0 tx=0x7fa014010860 comp rx=0 tx=0).stop 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 -- 172.21.15.27:0/2226191820 shutdown_connections 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c33960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c31030 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 --2- 172.21.15.27:0/2226191820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c26960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 -- 172.21.15.27:0/2226191820 >> 172.21.15.27:0/2226191820 conn(0x564ed3c228a0 msgr2=0x564ed3c22c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 -- 172.21.15.27:0/2226191820 shutdown_connections 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 -- 172.21.15.27:0/2226191820 wait complete. 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 -- start start 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.978+0000 7fa02618c740 1 Processor -- start 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa02618c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c30f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa02618c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c2e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa02618c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c2e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa02618c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564ed3c34360 con 0x564ed3c26ea0 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa02618c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564ed3b47050 con 0x564ed3c26580 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa02618c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564ed3c344e0 con 0x564ed3c31570 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa02618c740 10 client.0 _ll_register_callbacks cb 0x564ed389ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c2e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c2e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48236/0 (socket says 172.21.15.27:48236) 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 -- 172.21.15.27:0/1901415278 learned_addr learned my addr 172.21.15.27:0/1901415278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa020a18640 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c30f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 -- 172.21.15.27:0/1901415278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 msgr2=0x564ed3c2e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c2e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 -- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 msgr2=0x564ed3c30f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c30f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 -- 172.21.15.27:0/1901415278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed3b18dc0 con 0x564ed3c26ea0 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa020a18640 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c30f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa021219640 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c2e200 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa014010370 tx=0x7fa01401b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4df70779f266fc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.979+0000 7fa000ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa000ff9640 1 -- 172.21.15.27:0/1901415278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014029060 con 0x564ed3c26ea0 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa000ff9640 1 -- 172.21.15.27:0/1901415278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fa01401bc70 con 0x564ed3c26ea0 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 1 -- 172.21.15.27:0/1901415278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa014012bf0 con 0x564ed3c26ea0 2024-04-25T01:32:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa000ff9640 1 -- 172.21.15.27:0/1901415278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa014030050 con 0x564ed3c26ea0 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 10 client.17187 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 1 -- 172.21.15.27:0/1901415278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564ed389f3f0 con 0x564ed3c26ea0 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa000ff9640 1 -- 172.21.15.27:0/1901415278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa014012bf0 con 0x564ed3c26ea0 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa000ff9640 1 -- 172.21.15.27:0/1901415278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa01400f040 con 0x564ed3c26ea0 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 10 client.17187 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 10 client.17187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 20 client.17187 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564ed3c41620 0x564ed3c43a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 4 client.17187 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.980+0000 7fa02618c740 1 -- 172.21.15.27:0/1901415278 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ed3b98bb0 con 0x564ed3c41620 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.981+0000 7fa020a18640 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564ed3c41620 0x564ed3c43a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.981+0000 7fa020a18640 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564ed3c41620 0x564ed3c43a10 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eaa12fd24e2fff3 server_cookie=4d16e23bfe636f04 in_seq=0 out_seq=0 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.981+0000 7fa000ff9640 10 client.17187 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa000ff9640 1 -- 172.21.15.27:0/1901415278 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ed3b98bb0 con 0x564ed3c41620 2024-04-25T01:32:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa000ff9640 10 client.17187 handle_command_reply: tid=0 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 client.17187 shutdown 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 -- 172.21.15.27:0/1901415278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 msgr2=0x564ed3c2e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c2e200 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa014010370 tx=0x7fa01401b930 comp rx=0 tx=0).stop 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 -- 172.21.15.27:0/1901415278 shutdown_connections 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564ed3c41620 0x564ed3c43a10 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c2e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c2e200 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 --2- 172.21.15.27:0/1901415278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c30f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa000ff9640 0 client.17187 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 -- 172.21.15.27:0/1901415278 >> 172.21.15.27:0/1901415278 conn(0x564ed3c228a0 msgr2=0x564ed3c21b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 -- 172.21.15.27:0/1901415278 shutdown_connections 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 1 -- 172.21.15.27:0/1901415278 wait complete. 2024-04-25T01:32:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.982+0000 7fa02618c740 20 client.17187 trim_cache size 0 max 16384 2024-04-25T01:32:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.995+0000 7fa02618c740 1 Processor -- start 2024-04-25T01:32:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa02618c740 1 -- start start 2024-04-25T01:32:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa02618c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c3d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa02618c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c219c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa02618c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c30a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa02618c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1b3830 con 0x564ed3c26ea0 2024-04-25T01:32:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa02618c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1b1100 con 0x564ed3c26580 2024-04-25T01:32:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa02618c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1aef70 con 0x564ed3c31570 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c219c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c219c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48240/0 (socket says 172.21.15.27:48240) 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 -- 172.21.15.27:0/2584916310 learned_addr learned my addr 172.21.15.27:0/2584916310 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa020a18640 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c30a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa020a18640 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c3d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 -- 172.21.15.27:0/2584916310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 msgr2=0x564ed3c30a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c30a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 -- 172.21.15.27:0/2584916310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 msgr2=0x564ed3c3d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c3d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 -- 172.21.15.27:0/2584916310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed39697d0 con 0x564ed3c26ea0 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa021219640 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c219c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa014006010 tx=0x7fa014009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a442132f4c2086 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa002ffd640 1 -- 172.21.15.27:0/2584916310 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014029040 con 0x564ed3c26ea0 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa002ffd640 1 -- 172.21.15.27:0/2584916310 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa014012e50 con 0x564ed3c26ea0 2024-04-25T01:32:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.996+0000 7fa002ffd640 1 -- 172.21.15.27:0/2584916310 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014016070 con 0x564ed3c26ea0 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 -- 172.21.15.27:0/2584916310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 msgr2=0x564ed3c219c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c219c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa014006010 tx=0x7fa014009760 comp rx=0 tx=0).stop 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 -- 172.21.15.27:0/2584916310 shutdown_connections 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c31570 0x564ed3c30a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c26ea0 0x564ed3c219c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 --2- 172.21.15.27:0/2584916310 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c3d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 -- 172.21.15.27:0/2584916310 >> 172.21.15.27:0/2584916310 conn(0x564ed3c228a0 msgr2=0x564ed3c212b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 -- 172.21.15.27:0/2584916310 shutdown_connections 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 -- 172.21.15.27:0/2584916310 wait complete. 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 -- start start 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.997+0000 7fa02618c740 1 Processor -- start 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa02618c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c397e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa02618c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c26ea0 0x564ed3c34d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa02618c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c31570 0x564ed3c35280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa02618c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1af5c0 con 0x564ed3c31570 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa02618c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1b0a30 con 0x564ed3c26580 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa02618c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1afc90 con 0x564ed3c26ea0 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa02618c740 10 client.0 _ll_register_callbacks cb 0x7fa010009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c31570 0x564ed3c35280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c31570 0x564ed3c35280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48250/0 (socket says 172.21.15.27:48250) 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 -- 172.21.15.27:0/4194224226 learned_addr learned my addr 172.21.15.27:0/4194224226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa021219640 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c26ea0 0x564ed3c34d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 -- 172.21.15.27:0/4194224226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c26ea0 msgr2=0x564ed3c34d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c26ea0 0x564ed3c34d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 -- 172.21.15.27:0/4194224226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 msgr2=0x564ed3c397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c397e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 -- 172.21.15.27:0/4194224226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed3c357c0 con 0x564ed3c31570 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa020a18640 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c31570 0x564ed3c35280 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x564ed3c35dc0 tx=0x7fa01003f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c6cfade5be6d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.998+0000 7fa000ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa000ff9640 1 -- 172.21.15.27:0/4194224226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa010035050 con 0x564ed3c31570 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 1 -- 172.21.15.27:0/4194224226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa014012e50 con 0x564ed3c31570 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa000ff9640 1 -- 172.21.15.27:0/4194224226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fa0100042f0 con 0x564ed3c31570 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa000ff9640 1 -- 172.21.15.27:0/4194224226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa010049310 con 0x564ed3c31570 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 10 client.17193 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 1 -- 172.21.15.27:0/4194224226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564ed3ca7650 con 0x564ed3c31570 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa000ff9640 1 -- 172.21.15.27:0/4194224226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa014012e50 con 0x564ed3c31570 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa000ff9640 1 -- 172.21.15.27:0/4194224226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa0100047a0 con 0x564ed3c31570 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 10 client.17193 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 10 client.17193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 20 client.17193 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564ed3ca7d60 0x564ed3caa180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 4 client.17193 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:31.999+0000 7fa02618c740 1 -- 172.21.15.27:0/4194224226 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ed3b92f00 con 0x564ed3ca7d60 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa021219640 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564ed3ca7d60 0x564ed3caa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa021219640 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564ed3ca7d60 0x564ed3caa180 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0572b9114819394 server_cookie=2cf9370c8736008b in_seq=0 out_seq=0 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa000ff9640 10 client.17193 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa000ff9640 1 -- 172.21.15.27:0/4194224226 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ed3b92f00 con 0x564ed3ca7d60 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa000ff9640 10 client.17193 handle_command_reply: tid=0 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 client.17193 shutdown 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 -- 172.21.15.27:0/4194224226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c31570 msgr2=0x564ed3c35280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c31570 0x564ed3c35280 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x564ed3c35dc0 tx=0x7fa01003f040 comp rx=0 tx=0).stop 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 -- 172.21.15.27:0/4194224226 shutdown_connections 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564ed3ca7d60 0x564ed3caa180 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564ed3c31570 0x564ed3c35280 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564ed3c26ea0 0x564ed3c34d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa000ff9640 0 client.17193 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 --2- 172.21.15.27:0/4194224226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564ed3c26580 0x564ed3c397e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 -- 172.21.15.27:0/4194224226 >> 172.21.15.27:0/4194224226 conn(0x564ed3c228a0 msgr2=0x564ed3c296e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 -- 172.21.15.27:0/4194224226 shutdown_connections 2024-04-25T01:32:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 1 -- 172.21.15.27:0/4194224226 wait complete. 2024-04-25T01:32:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.000+0000 7fa02618c740 20 client.17193 trim_cache size 0 max 16384 2024-04-25T01:32:31.985 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:32:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 -- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa0040a2d80 msgr2=0x7fa0040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa0040a2d80 0x7fa0040a5240 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0a3900 tx=0x7fa00c005ce0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 -- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c10e0f0 msgr2=0x7fa01c19eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c10e0f0 0x7fa01c19eb00 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00d720 tx=0x7fa00c00dbe0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 -- 172.21.15.27:0/560605587 shutdown_connections 2024-04-25T01:32:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa0040a2d80 0x7fa0040a5240 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa01c10e0f0 0x7fa01c19eb00 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa01c100e60 0x7fa01c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 --2- 172.21.15.27:0/560605587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa01c100540 0x7fa01c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 -- 172.21.15.27:0/560605587 >> 172.21.15.27:0/560605587 conn(0x7fa01c077740 msgr2=0x7fa01c10f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 -- 172.21.15.27:0/560605587 shutdown_connections 2024-04-25T01:32:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:32.001+0000 7fa02347d640 1 -- 172.21.15.27:0/560605587 wait complete. 2024-04-25T01:32:32.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:32 smithi027 ceph-mon[29197]: pgmap v218: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:32.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:32 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2466280141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:32.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:32 smithi153 ceph-mon[32621]: pgmap v218: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:32.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:32 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2466280141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:32.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:32 smithi139 ceph-mon[32760]: pgmap v218: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:32.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:32 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2466280141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:32.990 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:32:32.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:32.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:32:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 -- 172.21.15.27:0/616236605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c06a780 msgr2=0x7f7f5c10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 --2- 172.21.15.27:0/616236605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c06a780 0x7f7f5c10e5f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009cd0 tx=0x7f7f5402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 -- 172.21.15.27:0/616236605 shutdown_connections 2024-04-25T01:32:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 --2- 172.21.15.27:0/616236605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c06a780 0x7f7f5c10e5f0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 --2- 172.21.15.27:0/616236605 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f5c069dc0 0x7f7f5c06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 --2- 172.21.15.27:0/616236605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f5c107730 0x7f7f5c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 -- 172.21.15.27:0/616236605 >> 172.21.15.27:0/616236605 conn(0x7f7f5c0776a0 msgr2=0x7f7f5c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 -- 172.21.15.27:0/616236605 shutdown_connections 2024-04-25T01:32:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.132+0000 7f7f6124d640 1 -- 172.21.15.27:0/616236605 wait complete. 2024-04-25T01:32:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.133+0000 7f7f6124d640 1 Processor -- start 2024-04-25T01:32:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.133+0000 7f7f6124d640 1 -- start start 2024-04-25T01:32:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.133+0000 7f7f6124d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c069dc0 0x7f7f5c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.133+0000 7f7f6124d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f5c06a780 0x7f7f5c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f6124d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f5c107730 0x7f7f5c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f6124d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c111b60 con 0x7f7f5c069dc0 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f6124d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1119e0 con 0x7f7f5c06a780 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f6124d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c111ce0 con 0x7f7f5c107730 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c069dc0 0x7f7f5c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c069dc0 0x7f7f5c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48274/0 (socket says 172.21.15.27:48274) 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 -- 172.21.15.27:0/338743821 learned_addr learned my addr 172.21.15.27:0/338743821 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f5c107730 0x7f7f5c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f5c06a780 0x7f7f5c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 -- 172.21.15.27:0/338743821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f5c107730 msgr2=0x7f7f5c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f5c107730 0x7f7f5c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 -- 172.21.15.27:0/338743821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f5c06a780 msgr2=0x7f7f5c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f5c06a780 0x7f7f5c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 -- 172.21.15.27:0/338743821 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5c19cff0 con 0x7f7f5c069dc0 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c069dc0 0x7f7f5c198000 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00d770 tx=0x7f7f4c00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f43fff640 1 -- 172.21.15.27:0/338743821 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c017070 con 0x7f7f5c069dc0 2024-04-25T01:32:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f43fff640 1 -- 172.21.15.27:0/338743821 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f4c009820 con 0x7f7f5c069dc0 2024-04-25T01:32:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f6124d640 1 -- 172.21.15.27:0/338743821 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5c19d2e0 con 0x7f7f5c069dc0 2024-04-25T01:32:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.134+0000 7f7f43fff640 1 -- 172.21.15.27:0/338743821 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c004cd0 con 0x7f7f5c069dc0 2024-04-25T01:32:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.135+0000 7f7f6124d640 1 -- 172.21.15.27:0/338743821 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f5c1a4b70 con 0x7f7f5c069dc0 2024-04-25T01:32:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.137+0000 7f7f43fff640 1 -- 172.21.15.27:0/338743821 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7f4c0099d0 con 0x7f7f5c069dc0 2024-04-25T01:32:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.137+0000 7f7f64037740 1 Processor -- start 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.137+0000 7f7f64037740 1 -- start start 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.137+0000 7f7f64037740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8decc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.137+0000 7f7f64037740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8df710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.137+0000 7f7f64037740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ec400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.137+0000 7f7f64037740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55797d8432c0 con 0x55797d8e0980 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f64037740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55797d7eb910 con 0x55797d8df290 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f64037740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55797d83f280 con 0x55797d8ea010 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f43fff640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f440a2d80 0x7f7f440a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8decc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48284/0 (socket says 172.21.15.27:48284) 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 -- 172.21.15.27:0/1664091047 learned_addr learned my addr 172.21.15.27:0/1664091047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f440a2d80 0x7f7f440a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f43fff640 1 -- 172.21.15.27:0/338743821 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f7f4c0cf1b0 con 0x7f7f5c069dc0 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8df710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 -- 172.21.15.27:0/1664091047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 msgr2=0x55797d8ec400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ec400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 -- 172.21.15.27:0/1664091047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 msgr2=0x55797d8df710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8df710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 -- 172.21.15.27:0/1664091047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55797d7fcfb0 con 0x55797d8e0980 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8df710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.138+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8decc0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009cd0 tx=0x7f7f54031c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15ab251ed7ae8f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f417fa640 1 -- 172.21.15.27:0/1664091047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54004b40 con 0x55797d8e0980 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f417fa640 1 -- 172.21.15.27:0/1664091047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f5404f070 con 0x55797d8e0980 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f417fa640 1 -- 172.21.15.27:0/1664091047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54030b80 con 0x55797d8e0980 2024-04-25T01:32:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 -- 172.21.15.27:0/1664091047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 msgr2=0x55797d8decc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f440a2d80 0x7f7f440a5240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7f5404e000 tx=0x7f7f5404c000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8decc0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009cd0 tx=0x7f7f54031c90 comp rx=0 tx=0).stop 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 -- 172.21.15.27:0/1664091047 shutdown_connections 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ec400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8df710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 --2- 172.21.15.27:0/1664091047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8decc0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 -- 172.21.15.27:0/1664091047 >> 172.21.15.27:0/1664091047 conn(0x55797d8a7950 msgr2=0x55797d8db7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 -- 172.21.15.27:0/1664091047 shutdown_connections 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.139+0000 7f7f64037740 1 -- 172.21.15.27:0/1664091047 wait complete. 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 1 -- start start 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 1 Processor -- start 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8df290 0x55797d8e6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8e0980 0x55797d8e7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8ea010 0x55797d8e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55797d8ecea0 con 0x55797d8ea010 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55797d7fee90 con 0x55797d8e0980 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55797d8ed020 con 0x55797d8df290 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f64037740 10 client.0 _ll_register_callbacks cb 0x55797d83a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8ea010 0x55797d8e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8ea010 0x55797d8e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48296/0 (socket says 172.21.15.27:48296) 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5b59e640 1 -- 172.21.15.27:0/1996853317 learned_addr learned my addr 172.21.15.27:0/1996853317 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8df290 0x55797d8e6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8e0980 0x55797d8e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5b59e640 1 -- 172.21.15.27:0/1996853317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8df290 msgr2=0x55797d8e6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8df290 0x55797d8e6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5b59e640 1 -- 172.21.15.27:0/1996853317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8e0980 msgr2=0x55797d8e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.140+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8e0980 0x55797d8e7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f5b59e640 1 -- 172.21.15.27:0/1996853317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55797d61f7e0 con 0x55797d8ea010 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8ea010 0x55797d8e7970 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7f54004a80 tx=0x7f7f5402ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9a770ddb9449a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f3eff5640 1 -- 172.21.15.27:0/1996853317 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54003e00 con 0x55797d8ea010 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f3eff5640 1 -- 172.21.15.27:0/1996853317 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f7f54004d50 con 0x55797d8ea010 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f64037740 1 -- 172.21.15.27:0/1996853317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f5404f070 con 0x55797d8ea010 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f3eff5640 1 -- 172.21.15.27:0/1996853317 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7f54066050 con 0x55797d8ea010 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f64037740 10 client.17205 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f64037740 1 -- 172.21.15.27:0/1996853317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55797d67ff70 con 0x55797d8ea010 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f3eff5640 1 -- 172.21.15.27:0/1996853317 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f5404f070 con 0x55797d8ea010 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f3eff5640 1 -- 172.21.15.27:0/1996853317 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7f54030070 con 0x55797d8ea010 2024-04-25T01:32:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f64037740 10 client.17205 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f64037740 10 client.17205 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.141+0000 7f7f64037740 20 client.17205 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.142+0000 7f7f64037740 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55797d8f9ec0 0x55797d8fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.142+0000 7f7f64037740 4 client.17205 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.142+0000 7f7f64037740 1 -- 172.21.15.27:0/1996853317 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55797d8175a0 con 0x55797d8f9ec0 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.142+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55797d8f9ec0 0x55797d8fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.142+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55797d8f9ec0 0x55797d8fc2b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15ae8e50977e86e server_cookie=dcdeaebccbccacae in_seq=0 out_seq=0 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.142+0000 7f7f3eff5640 10 client.17205 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f3eff5640 1 -- 172.21.15.27:0/1996853317 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55797d8175a0 con 0x55797d8f9ec0 2024-04-25T01:32:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f3eff5640 10 client.17205 handle_command_reply: tid=0 2024-04-25T01:32:33.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 client.17205 shutdown 2024-04-25T01:32:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 -- 172.21.15.27:0/1996853317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8ea010 msgr2=0x55797d8e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8ea010 0x55797d8e7970 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7f54004a80 tx=0x7f7f5402ae80 comp rx=0 tx=0).stop 2024-04-25T01:32:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 -- 172.21.15.27:0/1996853317 shutdown_connections 2024-04-25T01:32:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55797d8f9ec0 0x55797d8fc2b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8ea010 0x55797d8e7970 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8e0980 0x55797d8e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 --2- 172.21.15.27:0/1996853317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8df290 0x55797d8e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f3eff5640 0 client.17205 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 -- 172.21.15.27:0/1996853317 >> 172.21.15.27:0/1996853317 conn(0x55797d8a7950 msgr2=0x55797d8e15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 -- 172.21.15.27:0/1996853317 shutdown_connections 2024-04-25T01:32:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.143+0000 7f7f64037740 1 -- 172.21.15.27:0/1996853317 wait complete. 2024-04-25T01:32:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.144+0000 7f7f64037740 20 client.17205 trim_cache size 0 max 16384 2024-04-25T01:32:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.161+0000 7f7f64037740 1 Processor -- start 2024-04-25T01:32:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.161+0000 7f7f64037740 1 -- start start 2024-04-25T01:32:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f64037740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f64037740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8e7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f64037740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d901960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f64037740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1b1560 con 0x55797d8e0980 2024-04-25T01:32:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f64037740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1aee30 con 0x55797d8df290 2024-04-25T01:32:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f64037740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1acca0 con 0x55797d8ea010 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8e7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8e7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48310/0 (socket says 172.21.15.27:48310) 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 -- 172.21.15.27:0/3556978697 learned_addr learned my addr 172.21.15.27:0/3556978697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d901960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 -- 172.21.15.27:0/3556978697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 msgr2=0x55797d901960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d901960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 -- 172.21.15.27:0/3556978697 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 msgr2=0x55797d8f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 -- 172.21.15.27:0/3556978697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55797d517170 con 0x55797d8e0980 2024-04-25T01:32:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8e7790 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009a30 tx=0x7f7f50009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2299220bdcb0609c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f40ff9640 1 -- 172.21.15.27:0/3556978697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50014be0 con 0x55797d8e0980 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f40ff9640 1 -- 172.21.15.27:0/3556978697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f50014d80 con 0x55797d8e0980 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.162+0000 7f7f40ff9640 1 -- 172.21.15.27:0/3556978697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50016970 con 0x55797d8e0980 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 -- 172.21.15.27:0/3556978697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 msgr2=0x55797d8e7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8e7790 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009a30 tx=0x7f7f50009110 comp rx=0 tx=0).stop 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 -- 172.21.15.27:0/3556978697 shutdown_connections 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d901960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8e7790 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 --2- 172.21.15.27:0/3556978697 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 -- 172.21.15.27:0/3556978697 >> 172.21.15.27:0/3556978697 conn(0x55797d8a7950 msgr2=0x55797d8da950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 -- 172.21.15.27:0/3556978697 shutdown_connections 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 -- 172.21.15.27:0/3556978697 wait complete. 2024-04-25T01:32:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 -- start start 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.163+0000 7f7f64037740 1 Processor -- start 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f64037740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f64037740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f64037740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ef280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f64037740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1ad2f0 con 0x55797d8e0980 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f64037740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1ae760 con 0x55797d8df290 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f64037740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1ad9c0 con 0x55797d8ea010 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f64037740 10 client.0 _ll_register_callbacks cb 0x7f7f500093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8f27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48320/0 (socket says 172.21.15.27:48320) 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 -- 172.21.15.27:0/211134608 learned_addr learned my addr 172.21.15.27:0/211134608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 -- 172.21.15.27:0/211134608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 msgr2=0x55797d8ef280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ef280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 -- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 msgr2=0x55797d8f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 -- 172.21.15.27:0/211134608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55797d8efa90 con 0x55797d8e0980 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5b59e640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ef280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f5a59c640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8f27f0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f7f50000c00 tx=0x7f7f5001fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07f98802fef4d6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f3eff5640 1 -- 172.21.15.27:0/211134608 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50014be0 con 0x55797d8e0980 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f3eff5640 1 -- 172.21.15.27:0/211134608 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f7f50016980 con 0x55797d8e0980 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.164+0000 7f7f64037740 1 -- 172.21.15.27:0/211134608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f50014d80 con 0x55797d8e0980 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f3eff5640 1 -- 172.21.15.27:0/211134608 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7f50016b80 con 0x55797d8e0980 2024-04-25T01:32:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f64037740 10 client.17211 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f64037740 1 -- 172.21.15.27:0/211134608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55797d9650a0 con 0x55797d8e0980 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f3eff5640 1 -- 172.21.15.27:0/211134608 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f50014d80 con 0x55797d8e0980 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f3eff5640 1 -- 172.21.15.27:0/211134608 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7f5002abe0 con 0x55797d8e0980 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f64037740 10 client.17211 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f64037740 10 client.17211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f64037740 20 client.17211 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f64037740 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55797d9656b0 0x55797d967ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f64037740 4 client.17211 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f64037740 1 -- 172.21.15.27:0/211134608 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55797d850800 con 0x55797d9656b0 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.165+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55797d9656b0 0x55797d967ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.166+0000 7f7f3eff5640 10 client.17211 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.166+0000 7f7f5ad9d640 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55797d9656b0 0x55797d967ad0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8033fb3ba8bfdea server_cookie=48cf833f3c267e7e in_seq=0 out_seq=0 2024-04-25T01:32:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f3eff5640 1 -- 172.21.15.27:0/211134608 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55797d850800 con 0x55797d9656b0 2024-04-25T01:32:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f3eff5640 10 client.17211 handle_command_reply: tid=0 2024-04-25T01:32:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 client.17211 shutdown 2024-04-25T01:32:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 -- 172.21.15.27:0/211134608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 msgr2=0x55797d8f27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8f27f0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f7f50000c00 tx=0x7f7f5001fcc0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 -- 172.21.15.27:0/211134608 shutdown_connections 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55797d9656b0 0x55797d967ad0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55797d8ea010 0x55797d8ef280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55797d8e0980 0x55797d8f27f0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 --2- 172.21.15.27:0/211134608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55797d8df290 0x55797d8f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f3eff5640 0 client.17211 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.167+0000 7f7f64037740 1 -- 172.21.15.27:0/211134608 >> 172.21.15.27:0/211134608 conn(0x55797d8a7950 msgr2=0x55797d8e8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f64037740 1 -- 172.21.15.27:0/211134608 shutdown_connections 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f64037740 1 -- 172.21.15.27:0/211134608 wait complete. 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f64037740 20 client.17211 trim_cache size 0 max 16384 2024-04-25T01:32:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:32:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:32:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "86f02222-7a51-4a6b-8872-1230149cc588", 2024-04-25T01:32:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:32:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:32:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 -- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f440a2d80 msgr2=0x7f7f440a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f440a2d80 0x7f7f440a5240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7f5404e000 tx=0x7f7f5404c000 comp rx=0 tx=0).stop 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 -- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c069dc0 msgr2=0x7f7f5c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c069dc0 0x7f7f5c198000 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00d770 tx=0x7f7f4c00db70 comp rx=0 tx=0).stop 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 -- 172.21.15.27:0/338743821 shutdown_connections 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f5c107730 0x7f7f5c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f5c06a780 0x7f7f5c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f7f440a2d80 0x7f7f440a5240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 --2- 172.21.15.27:0/338743821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c069dc0 0x7f7f5c198000 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 -- 172.21.15.27:0/338743821 >> 172.21.15.27:0/338743821 conn(0x7f7f5c0776a0 msgr2=0x7f7f5c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 -- 172.21.15.27:0/338743821 shutdown_connections 2024-04-25T01:32:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.168+0000 7f7f6124d640 1 -- 172.21.15.27:0/338743821 wait complete. 2024-04-25T01:32:33.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:32:33.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:32:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 -- 172.21.15.27:0/1823810654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 msgr2=0x7f4a0c1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 --2- 172.21.15.27:0/1823810654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 0x7f4a0c1006c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009900 tx=0x7f4a0002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 -- 172.21.15.27:0/1823810654 shutdown_connections 2024-04-25T01:32:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 --2- 172.21.15.27:0/1823810654 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a0c106aa0 0x7f4a0c10e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 --2- 172.21.15.27:0/1823810654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a0c100c90 0x7f4a0c1063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 --2- 172.21.15.27:0/1823810654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 0x7f4a0c1006c0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 -- 172.21.15.27:0/1823810654 >> 172.21.15.27:0/1823810654 conn(0x7f4a0c0758d0 msgr2=0x7f4a0c075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 -- 172.21.15.27:0/1823810654 shutdown_connections 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.284+0000 7f4a143e3640 1 -- 172.21.15.27:0/1823810654 wait complete. 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a143e3640 1 Processor -- start 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a143e3640 1 -- start start 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a143e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 0x7f4a0c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a143e3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a0c106aa0 0x7f4a0c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a143e3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a0c19ca50 0x7f4a0c19cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a143e3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c111b90 con 0x7f4a0c1002e0 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a143e3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c111a10 con 0x7f4a0c19ca50 2024-04-25T01:32:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a143e3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c111d10 con 0x7f4a0c106aa0 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 0x7f4a0c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 0x7f4a0c198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48342/0 (socket says 172.21.15.27:48342) 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 -- 172.21.15.27:0/206501822 learned_addr learned my addr 172.21.15.27:0/206501822 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a12980640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a0c19ca50 0x7f4a0c19cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1197e640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a0c106aa0 0x7f4a0c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 -- 172.21.15.27:0/206501822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a0c106aa0 msgr2=0x7f4a0c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a0c106aa0 0x7f4a0c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 -- 172.21.15.27:0/206501822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a0c19ca50 msgr2=0x7f4a0c19cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a0c19ca50 0x7f4a0c19cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 -- 172.21.15.27:0/206501822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c19d540 con 0x7f4a0c1002e0 2024-04-25T01:32:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1197e640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a0c106aa0 0x7f4a0c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.285+0000 7f4a1217f640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 0x7f4a0c198190 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4a00006010 tx=0x7f4a000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.286+0000 7f49ff7fe640 1 -- 172.21.15.27:0/206501822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0003f030 con 0x7f4a0c1002e0 2024-04-25T01:32:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.286+0000 7f4a143e3640 1 -- 172.21.15.27:0/206501822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c19d7d0 con 0x7f4a0c1002e0 2024-04-25T01:32:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.286+0000 7f49ff7fe640 1 -- 172.21.15.27:0/206501822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a00003ce0 con 0x7f4a0c1002e0 2024-04-25T01:32:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.286+0000 7f49ff7fe640 1 -- 172.21.15.27:0/206501822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00049060 con 0x7f4a0c1002e0 2024-04-25T01:32:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.286+0000 7f4a143e3640 1 -- 172.21.15.27:0/206501822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0c1a4ff0 con 0x7f4a0c1002e0 2024-04-25T01:32:33.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f49ff7fe640 1 -- 172.21.15.27:0/206501822 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4a00042030 con 0x7f4a0c1002e0 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a171cd740 1 Processor -- start 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a171cd740 1 -- start start 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b147e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1524f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b154e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a171cd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56352b0a86b0 con 0x56352b147a40 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a171cd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56352b050880 con 0x56352b152a30 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a171cd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56352b0a4670 con 0x56352b148360 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.289+0000 7f4a12980640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b147e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48348/0 (socket says 172.21.15.27:48348) 2024-04-25T01:32:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 -- 172.21.15.27:0/2263400871 learned_addr learned my addr 172.21.15.27:0/2263400871 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f49ff7fe640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f49f80a2d00 0x7f49f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a1197e640 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f49f80a2d00 0x7f49f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f49ff7fe640 1 -- 172.21.15.27:0/206501822 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f4a000efe10 con 0x7f4a0c1002e0 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a1217f640 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 -- 172.21.15.27:0/2263400871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 msgr2=0x56352b1524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 -- 172.21.15.27:0/2263400871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 msgr2=0x56352b154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 -- 172.21.15.27:0/2263400871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352ae867d0 con 0x56352b147a40 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a1217f640 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a1197e640 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f4a12980640 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b147e20 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400bce0 tx=0x7f4a040039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3396c2d712d0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f49fcbfe640 1 -- 172.21.15.27:0/2263400871 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0401d600 con 0x56352b147a40 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f49fcbfe640 1 -- 172.21.15.27:0/2263400871 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a04022070 con 0x56352b147a40 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.290+0000 7f49fcbfe640 1 -- 172.21.15.27:0/2263400871 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0401c710 con 0x56352b147a40 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 -- 172.21.15.27:0/2263400871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 msgr2=0x56352b147e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a12980640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f49f80a2d00 0x7f49f80a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4a04021000 tx=0x7f4a0401f000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b147e20 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400bce0 tx=0x7f4a040039c0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 -- 172.21.15.27:0/2263400871 shutdown_connections 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 --2- 172.21.15.27:0/2263400871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b147e20 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 -- 172.21.15.27:0/2263400871 >> 172.21.15.27:0/2263400871 conn(0x56352b143d60 msgr2=0x56352b144140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 -- 172.21.15.27:0/2263400871 shutdown_connections 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 -- 172.21.15.27:0/2263400871 wait complete. 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.291+0000 7f4a171cd740 1 -- start start 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a171cd740 1 Processor -- start 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b147a40 0x56352b14f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b148360 0x56352b14fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b150430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a171cd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56352b155820 con 0x56352b148360 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a171cd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56352b063f50 con 0x56352b152a30 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a171cd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56352b1559a0 con 0x56352b147a40 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a171cd740 10 client.0 _ll_register_callbacks cb 0x56352b09f000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b148360 0x56352b14fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b148360 0x56352b14fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48358/0 (socket says 172.21.15.27:48358) 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 -- 172.21.15.27:0/261258303 learned_addr learned my addr 172.21.15.27:0/261258303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1217f640 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b147a40 0x56352b14f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a12980640 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 -- 172.21.15.27:0/261258303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b147a40 msgr2=0x56352b14f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b147a40 0x56352b14f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 -- 172.21.15.27:0/261258303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 msgr2=0x56352b150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 -- 172.21.15.27:0/261258303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56352adbc3f0 con 0x56352b148360 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f4a1197e640 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b148360 0x56352b14fef0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4a08012720 tx=0x7f4a08012b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a490aa0418d3e726 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f49f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.292+0000 7f49f5ffb640 1 -- 172.21.15.27:0/261258303 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0801f060 con 0x56352b148360 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f49f5ffb640 1 -- 172.21.15.27:0/261258303 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f4a08010d60 con 0x56352b148360 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 1 -- 172.21.15.27:0/261258303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a04022070 con 0x56352b148360 2024-04-25T01:32:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f49f5ffb640 1 -- 172.21.15.27:0/261258303 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a08026050 con 0x56352b148360 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 10 client.17223 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 1 -- 172.21.15.27:0/261258303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56352ad7c0f0 con 0x56352b148360 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f49f5ffb640 1 -- 172.21.15.27:0/261258303 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a04022070 con 0x56352b148360 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f49f5ffb640 1 -- 172.21.15.27:0/261258303 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4a08019030 con 0x56352b148360 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 10 client.17223 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 10 client.17223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 20 client.17223 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56352b162dd0 0x56352b1651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 4 client.17223 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a171cd740 1 -- 172.21.15.27:0/261258303 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56352b0b5bf0 con 0x56352b162dd0 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.293+0000 7f4a1217f640 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56352b162dd0 0x56352b1651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.294+0000 7f49f5ffb640 10 client.17223 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.294+0000 7f4a1217f640 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56352b162dd0 0x56352b1651e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa3e646ba36ec564 server_cookie=8e910b856ed502df in_seq=0 out_seq=0 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f49f5ffb640 1 -- 172.21.15.27:0/261258303 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56352b0b5bf0 con 0x56352b162dd0 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f49f5ffb640 10 client.17223 handle_command_reply: tid=0 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 client.17223 shutdown 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 -- 172.21.15.27:0/261258303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b148360 msgr2=0x56352b14fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b148360 0x56352b14fef0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4a08012720 tx=0x7f4a08012b20 comp rx=0 tx=0).stop 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 -- 172.21.15.27:0/261258303 shutdown_connections 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56352b162dd0 0x56352b1651e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b148360 0x56352b14fef0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f49f5ffb640 0 client.17223 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 --2- 172.21.15.27:0/261258303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b147a40 0x56352b14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 -- 172.21.15.27:0/261258303 >> 172.21.15.27:0/261258303 conn(0x56352b143d60 msgr2=0x56352b13eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 -- 172.21.15.27:0/261258303 shutdown_connections 2024-04-25T01:32:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 1 -- 172.21.15.27:0/261258303 wait complete. 2024-04-25T01:32:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.295+0000 7f4a171cd740 20 client.17223 trim_cache size 0 max 16384 2024-04-25T01:32:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a171cd740 1 Processor -- start 2024-04-25T01:32:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a171cd740 1 -- start start 2024-04-25T01:32:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b1520e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b15e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b13e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a171cd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c1af410 con 0x56352b147a40 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a171cd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c1acce0 con 0x56352b152a30 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a171cd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c1aab50 con 0x56352b148360 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b1520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b1520e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48364/0 (socket says 172.21.15.27:48364) 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 -- 172.21.15.27:0/2988381185 learned_addr learned my addr 172.21.15.27:0/2988381185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1197e640 1 --2- 172.21.15.27:0/2988381185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 -- 172.21.15.27:0/2988381185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 msgr2=0x56352b15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 --2- 172.21.15.27:0/2988381185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b15e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 -- 172.21.15.27:0/2988381185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 msgr2=0x56352b13e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 --2- 172.21.15.27:0/2988381185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b13e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 -- 172.21.15.27:0/2988381185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352b1488b0 con 0x56352b147a40 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.313+0000 7f4a1217f640 1 --2- 172.21.15.27:0/2988381185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b1520e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f4a000b89d0 tx=0x7f4a0003b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbf9e819382c2e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f49f7fff640 1 -- 172.21.15.27:0/2988381185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00035040 con 0x56352b147a40 2024-04-25T01:32:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f49f7fff640 1 -- 172.21.15.27:0/2988381185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a000bc700 con 0x56352b147a40 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f49f7fff640 1 -- 172.21.15.27:0/2988381185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a000ba390 con 0x56352b147a40 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 -- 172.21.15.27:0/2988381185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 msgr2=0x56352b1520e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 --2- 172.21.15.27:0/2988381185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b1520e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f4a000b89d0 tx=0x7f4a0003b460 comp rx=0 tx=0).stop 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 -- 172.21.15.27:0/2988381185 shutdown_connections 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 --2- 172.21.15.27:0/2988381185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b13e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 --2- 172.21.15.27:0/2988381185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b15e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 --2- 172.21.15.27:0/2988381185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b1520e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 -- 172.21.15.27:0/2988381185 >> 172.21.15.27:0/2988381185 conn(0x56352b143d60 msgr2=0x56352b13e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 -- 172.21.15.27:0/2988381185 shutdown_connections 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.314+0000 7f4a171cd740 1 -- 172.21.15.27:0/2988381185 wait complete. 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 1 -- start start 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 1 Processor -- start 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b156f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b1c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c1ab1a0 con 0x56352b147a40 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c1ac610 con 0x56352b152a30 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c1ab870 con 0x56352b148360 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a171cd740 10 client.0 _ll_register_callbacks cb 0x7f4a080096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48380/0 (socket says 172.21.15.27:48380) 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 -- 172.21.15.27:0/1288207296 learned_addr learned my addr 172.21.15.27:0/1288207296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a12980640 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b1c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1197e640 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 -- 172.21.15.27:0/1288207296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 msgr2=0x56352b1574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 -- 172.21.15.27:0/1288207296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 msgr2=0x56352b1c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b1c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 -- 172.21.15.27:0/1288207296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56352b1c29e0 con 0x56352b147a40 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f4a1217f640 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b156f60 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f4a00005ab0 tx=0x7f4a00005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b67c413e58880e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.315+0000 7f49f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f49f5ffb640 1 -- 172.21.15.27:0/1288207296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00035070 con 0x56352b147a40 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f49f5ffb640 1 -- 172.21.15.27:0/1288207296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f4a000ed3f0 con 0x56352b147a40 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 1 -- 172.21.15.27:0/1288207296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a000bc700 con 0x56352b147a40 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f49f5ffb640 1 -- 172.21.15.27:0/1288207296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a000c03b0 con 0x56352b147a40 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 10 client.17229 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 1 -- 172.21.15.27:0/1288207296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56352b1c9d60 con 0x56352b147a40 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f49f5ffb640 1 -- 172.21.15.27:0/1288207296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a000bc700 con 0x56352b147a40 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f49f5ffb640 1 -- 172.21.15.27:0/1288207296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4a0003bc50 con 0x56352b147a40 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 10 client.17229 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 10 client.17229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 20 client.17229 populate_metadata read hostname 'smithi027' 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56352b1ca280 0x56352b1cc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 4 client.17229 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a171cd740 1 -- 172.21.15.27:0/1288207296 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56352b0aff40 con 0x56352b1ca280 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.316+0000 7f4a1197e640 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56352b1ca280 0x56352b1cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a1197e640 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56352b1ca280 0x56352b1cc670 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8ee2553d1f497c2 server_cookie=975b2fa6e9c0a212 in_seq=0 out_seq=0 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f49f5ffb640 10 client.17229 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f49f5ffb640 1 -- 172.21.15.27:0/1288207296 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56352b0aff40 con 0x56352b1ca280 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f49f5ffb640 10 client.17229 handle_command_reply: tid=0 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 client.17229 shutdown 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 -- 172.21.15.27:0/1288207296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 msgr2=0x56352b156f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b156f60 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f4a00005ab0 tx=0x7f4a00005750 comp rx=0 tx=0).stop 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 -- 172.21.15.27:0/1288207296 shutdown_connections 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56352b1ca280 0x56352b1cc670 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56352b152a30 0x56352b1c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56352b148360 0x56352b1574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 --2- 172.21.15.27:0/1288207296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56352b147a40 0x56352b156f60 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f49f5ffb640 0 client.17229 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:32:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 -- 172.21.15.27:0/1288207296 >> 172.21.15.27:0/1288207296 conn(0x56352b143d60 msgr2=0x56352b1501e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 -- 172.21.15.27:0/1288207296 shutdown_connections 2024-04-25T01:32:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 1 -- 172.21.15.27:0/1288207296 wait complete. 2024-04-25T01:32:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.317+0000 7f4a171cd740 20 client.17229 trim_cache size 0 max 16384 2024-04-25T01:32:33.302 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:32:33.302 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:32:33.302 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:32:33.302 INFO:teuthology.orchestra.run.smithi027.stdout: "86f02222-7a51-4a6b-8872-1230149cc588": { 2024-04-25T01:32:33.303 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:32:33.303 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "86f02222-7a51-4a6b-8872-1230149cc588", 2024-04-25T01:32:33.303 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:32:33.303 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:32:33.303 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:32:33.303 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 -- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f49f80a2d00 msgr2=0x7f49f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f49f80a2d00 0x7f49f80a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4a04021000 tx=0x7f4a0401f000 comp rx=0 tx=0).stop 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 -- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 msgr2=0x7f4a0c198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 0x7f4a0c198190 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4a00006010 tx=0x7f4a000042f0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 -- 172.21.15.27:0/206501822 shutdown_connections 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a0c19ca50 0x7f4a0c19cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a0c106aa0 0x7f4a0c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f49f80a2d00 0x7f49f80a51c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 --2- 172.21.15.27:0/206501822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0c1002e0 0x7f4a0c198190 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 -- 172.21.15.27:0/206501822 >> 172.21.15.27:0/206501822 conn(0x7f4a0c0758d0 msgr2=0x7f4a0c107c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 -- 172.21.15.27:0/206501822 shutdown_connections 2024-04-25T01:32:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:32:33.318+0000 7f4a143e3640 1 -- 172.21.15.27:0/206501822 wait complete. 2024-04-25T01:32:33.306 INFO:tasks.cephfs.filesystem:scrub status for tag:86f02222-7a51-4a6b-8872-1230149cc588 - {'path': '/', 'tag': '86f02222-7a51-4a6b-8872-1230149cc588', 'options': 'recursive,force'} 2024-04-25T01:32:34.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:34 smithi027 ceph-mon[29197]: pgmap v219: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:32:34.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:34 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:32:34.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:34 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:34.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:34 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:32:34.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:34 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:32:34.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:34 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:34.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:34 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:34.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:34 smithi153 ceph-mon[32621]: pgmap v219: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:32:34.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:34 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:32:34.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:34 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:34.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:34 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:32:34.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:34 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:32:34.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:34 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:34.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:34 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:34.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:34 smithi139 ceph-mon[32760]: pgmap v219: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:32:34.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:34 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:32:34.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:34 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:34.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:34 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:32:34.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:34 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:32:34.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:34 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:34.540 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:34 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:32:36.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:36 smithi027 ceph-mon[29197]: pgmap v220: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:32:36.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:36 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:32:36.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:36 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:36.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:36 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:36.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:36 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:32:36.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:36 smithi153 ceph-mon[32621]: pgmap v220: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:32:36.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:36 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:32:36.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:36 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:36.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:36 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:36.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:36 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:32:36.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:36 smithi139 ceph-mon[32760]: pgmap v220: 129 pgs: 129 active+clean; 3.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-04-25T01:32:36.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:36 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:32:36.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:36 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:36.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:36 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:36.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:36 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:32:38.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:38 smithi027 ceph-mon[29197]: pgmap v221: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:38.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:38 smithi153 ceph-mon[32621]: pgmap v221: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:38.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:38 smithi139 ceph-mon[32760]: pgmap v221: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:40.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:40 smithi027 ceph-mon[29197]: pgmap v222: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-04-25T01:32:40.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:40 smithi153 ceph-mon[32621]: pgmap v222: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-04-25T01:32:40.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:40 smithi139 ceph-mon[32760]: pgmap v222: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-04-25T01:32:42.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:42 smithi027 ceph-mon[29197]: pgmap v223: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:42.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:42 smithi153 ceph-mon[32621]: pgmap v223: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:42.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:42 smithi139 ceph-mon[32760]: pgmap v223: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:43.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:43 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:32:43.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:43 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:32:43.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:43 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:32:43.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:43 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:32:43.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:43 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:32:43.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:43 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:32:44.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:44 smithi027 ceph-mon[29197]: pgmap v224: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:32:44.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:44 smithi153 ceph-mon[32621]: pgmap v224: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:32:44.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:44 smithi139 ceph-mon[32760]: pgmap v224: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:32:46.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:46 smithi027 ceph-mon[29197]: pgmap v225: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:32:46.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:46 smithi153 ceph-mon[32621]: pgmap v225: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:32:46.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:46 smithi139 ceph-mon[32760]: pgmap v225: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:32:48.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:48 smithi027 ceph-mon[29197]: pgmap v226: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:48.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:48 smithi153 ceph-mon[32621]: pgmap v226: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:48.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:48 smithi139 ceph-mon[32760]: pgmap v226: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:50.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:50 smithi027 ceph-mon[29197]: pgmap v227: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:32:50.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:50 smithi153 ceph-mon[32621]: pgmap v227: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:32:50.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:50 smithi139 ceph-mon[32760]: pgmap v227: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:32:52.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:52 smithi027 ceph-mon[29197]: pgmap v228: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:32:52.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:52 smithi153 ceph-mon[32621]: pgmap v228: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:32:52.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:52 smithi139 ceph-mon[32760]: pgmap v228: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:32:54.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:54 smithi027 ceph-mon[29197]: pgmap v229: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:32:54.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:54 smithi153 ceph-mon[32621]: pgmap v229: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:32:54.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:54 smithi139 ceph-mon[32760]: pgmap v229: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:32:56.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:56 smithi027 ceph-mon[29197]: pgmap v230: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:32:56.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:56 smithi153 ceph-mon[32621]: pgmap v230: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:32:56.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:56 smithi139 ceph-mon[32760]: pgmap v230: 129 pgs: 129 active+clean; 3.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:32:58.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:32:58 smithi027 ceph-mon[29197]: pgmap v231: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:58.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:32:58 smithi153 ceph-mon[32621]: pgmap v231: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:32:58.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:32:58 smithi139 ceph-mon[32760]: pgmap v231: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:33:00.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:00 smithi027 ceph-mon[29197]: pgmap v232: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:33:00.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:00 smithi153 ceph-mon[32621]: pgmap v232: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:33:00.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:00 smithi139 ceph-mon[32760]: pgmap v232: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-04-25T01:33:02.361 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.362 INFO:tasks.workunit.client.0.smithi027.stdout:313/600: open fd = 3 2024-04-25T01:33:02.362 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.362 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.362 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.362 INFO:tasks.workunit.client.0.smithi027.stdout:314/600: open fd = 3 2024-04-25T01:33:02.363 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.363 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.363 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.363 INFO:tasks.workunit.client.0.smithi027.stdout:315/600: open fd = 3 2024-04-25T01:33:02.363 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.363 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.363 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.364 INFO:tasks.workunit.client.0.smithi027.stdout:316/600: open fd = 3 2024-04-25T01:33:02.364 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.364 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.364 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.364 INFO:tasks.workunit.client.0.smithi027.stdout:317/600: open fd = 3 2024-04-25T01:33:02.364 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.365 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.365 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.365 INFO:tasks.workunit.client.0.smithi027.stdout:318/600: open fd = 3 2024-04-25T01:33:02.365 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.365 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.365 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.365 INFO:tasks.workunit.client.0.smithi027.stdout:319/600: open fd = 3 2024-04-25T01:33:02.366 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.366 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.366 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.366 INFO:tasks.workunit.client.0.smithi027.stdout:320/600: open fd = 3 2024-04-25T01:33:02.366 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.366 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:321/600: open fd = 3 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:322/600: open fd = 3 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.367 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.368 INFO:tasks.workunit.client.0.smithi027.stdout:323/600: open fd = 3 2024-04-25T01:33:02.368 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.368 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.369 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.369 INFO:tasks.workunit.client.0.smithi027.stdout:324/600: open fd = 3 2024-04-25T01:33:02.369 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.369 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.369 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.370 INFO:tasks.workunit.client.0.smithi027.stdout:325/600: open fd = 3 2024-04-25T01:33:02.370 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.370 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.370 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.370 INFO:tasks.workunit.client.0.smithi027.stdout:326/600: open fd = 3 2024-04-25T01:33:02.371 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.371 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.371 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.371 INFO:tasks.workunit.client.0.smithi027.stdout:327/600: open fd = 3 2024-04-25T01:33:02.371 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.371 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.372 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.372 INFO:tasks.workunit.client.0.smithi027.stdout:328/600: open fd = 3 2024-04-25T01:33:02.372 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.372 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.372 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.372 INFO:tasks.workunit.client.0.smithi027.stdout:329/600: open fd = 3 2024-04-25T01:33:02.373 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.373 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.373 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.373 INFO:tasks.workunit.client.0.smithi027.stdout:330/600: open fd = 3 2024-04-25T01:33:02.373 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.373 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.374 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.374 INFO:tasks.workunit.client.0.smithi027.stdout:331/600: open fd = 3 2024-04-25T01:33:02.374 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.374 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.374 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.374 INFO:tasks.workunit.client.0.smithi027.stdout:332/600: open fd = 3 2024-04-25T01:33:02.375 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.375 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.375 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.375 INFO:tasks.workunit.client.0.smithi027.stdout:333/600: open fd = 3 2024-04-25T01:33:02.375 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.375 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.376 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.376 INFO:tasks.workunit.client.0.smithi027.stdout:334/600: open fd = 3 2024-04-25T01:33:02.376 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.376 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.376 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.376 INFO:tasks.workunit.client.0.smithi027.stdout:335/600: open fd = 3 2024-04-25T01:33:02.377 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.377 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:336/600: open fd = 3 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:337/600: open fd = 3 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:338/600: open fd = 3 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.378 INFO:tasks.workunit.client.0.smithi027.stdout:339/600: open fd = 3 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:340/600: open fd = 3 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:341/600: open fd = 3 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:342/600: open fd = 3 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.379 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:343/600: open fd = 3 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:344/600: open fd = 3 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:345/600: open fd = 3 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:346/600: open fd = 3 2024-04-25T01:33:02.380 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:347/600: open fd = 3 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:348/600: open fd = 3 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:349/600: open fd = 3 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.381 INFO:tasks.workunit.client.0.smithi027.stdout:350/600: open fd = 3 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:351/600: open fd = 3 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:352/600: open fd = 3 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:353/600: open fd = 3 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.382 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:354/600: open fd = 3 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:355/600: open fd = 3 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:356/600: open fd = 3 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:357/600: open fd = 3 2024-04-25T01:33:02.383 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:358/600: open fd = 3 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:359/600: open fd = 3 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:360/600: open fd = 3 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.384 INFO:tasks.workunit.client.0.smithi027.stdout:361/600: open fd = 3 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:362/600: open fd = 3 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:363/600: open fd = 3 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.385 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.386 INFO:tasks.workunit.client.0.smithi027.stdout:364/600: open fd = 3 2024-04-25T01:33:02.386 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.386 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.386 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.386 INFO:tasks.workunit.client.0.smithi027.stdout:365/600: open fd = 3 2024-04-25T01:33:02.386 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:366/600: open fd = 3 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:367/600: open fd = 3 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:368/600: open fd = 3 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.387 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:369/600: open fd = 3 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:370/600: open fd = 3 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:371/600: open fd = 3 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:372/600: open fd = 3 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.388 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:373/600: open fd = 3 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:374/600: open fd = 3 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:33:02.389 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:33:02.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:02 smithi153 ceph-mon[32621]: pgmap v233: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-04-25T01:33:02.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:02 smithi139 ceph-mon[32760]: pgmap v233: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-04-25T01:33:02.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:02 smithi027 ceph-mon[29197]: pgmap v233: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-04-25T01:33:03.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:33:03.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 -- 172.21.15.27:0/1501799048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698100fe0 msgr2=0x7f269810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 --2- 172.21.15.27:0/1501799048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698100fe0 0x7f269810d4f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2690009cd0 tx=0x7f269002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 -- 172.21.15.27:0/1501799048 shutdown_connections 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 --2- 172.21.15.27:0/1501799048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698100fe0 0x7f269810d4f0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 --2- 172.21.15.27:0/1501799048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2698100620 0x7f2698100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 --2- 172.21.15.27:0/1501799048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2698106630 0x7f2698106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 -- 172.21.15.27:0/1501799048 >> 172.21.15.27:0/1501799048 conn(0x7f26980776a0 msgr2=0x7f2698077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 -- 172.21.15.27:0/1501799048 shutdown_connections 2024-04-25T01:33:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.448+0000 7f269f34f640 1 -- 172.21.15.27:0/1501799048 wait complete. 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269f34f640 1 Processor -- start 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269f34f640 1 -- start start 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269f34f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2698100620 0x7f2698198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269f34f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2698100fe0 0x7f2698198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269f34f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698106630 0x7f269819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269f34f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2698110a90 con 0x7f2698106630 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269f34f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2698110910 con 0x7f2698100fe0 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269f34f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2698110c10 con 0x7f2698100620 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269d8ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698106630 0x7f269819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269d8ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698106630 0x7f269819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33562/0 (socket says 172.21.15.27:33562) 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269d8ec640 1 -- 172.21.15.27:0/2905296296 learned_addr learned my addr 172.21.15.27:0/2905296296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269d0eb640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2698100620 0x7f2698198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269c8ea640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2698100fe0 0x7f2698198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269d8ec640 1 -- 172.21.15.27:0/2905296296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2698100620 msgr2=0x7f2698198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269d8ec640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2698100620 0x7f2698198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269d8ec640 1 -- 172.21.15.27:0/2905296296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2698100fe0 msgr2=0x7f2698198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.449+0000 7f269d8ec640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2698100fe0 0x7f2698198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f269d8ec640 1 -- 172.21.15.27:0/2905296296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269819d030 con 0x7f2698106630 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f269c8ea640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2698100fe0 0x7f2698198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f269d0eb640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2698100620 0x7f2698198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f269d8ec640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698106630 0x7f269819c900 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2690033040 tx=0x7f2690031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f268a7fc640 1 -- 172.21.15.27:0/2905296296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690004040 con 0x7f2698106630 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f269f34f640 1 -- 172.21.15.27:0/2905296296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269819d2c0 con 0x7f2698106630 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f268a7fc640 1 -- 172.21.15.27:0/2905296296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26900046c0 con 0x7f2698106630 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f268a7fc640 1 -- 172.21.15.27:0/2905296296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690046480 con 0x7f2698106630 2024-04-25T01:33:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.450+0000 7f269f34f640 1 -- 172.21.15.27:0/2905296296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26981a4bb0 con 0x7f2698106630 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.452+0000 7f268a7fc640 1 -- 172.21.15.27:0/2905296296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f26900305a0 con 0x7f2698106630 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f26a2057740 1 Processor -- start 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f26a2057740 1 -- start start 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f26a2057740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c9294a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f26a2057740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c929f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f26a2057740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c92a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f26a2057740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b2c91f96b0 con 0x55b2c9294630 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f26a2057740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b2c91a1880 con 0x55b2c929f620 2024-04-25T01:33:03.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f26a2057740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b2c91f5670 con 0x55b2c9294f50 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f268a7fc640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f26840a2d00 0x7f26840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f269c8ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c9294a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f269c8ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c9294a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33578/0 (socket says 172.21.15.27:33578) 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f269c8ea640 1 -- 172.21.15.27:0/344769288 learned_addr learned my addr 172.21.15.27:0/344769288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.453+0000 7f269d8ec640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f26840a2d00 0x7f26840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269c8ea640 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c92a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269d0eb640 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c929f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f268a7fc640 1 -- 172.21.15.27:0/2905296296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f269002e030 con 0x7f2698106630 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269c8ea640 1 -- 172.21.15.27:0/344769288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 msgr2=0x55b2c929f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269c8ea640 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c929f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269c8ea640 1 -- 172.21.15.27:0/344769288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 msgr2=0x55b2c92a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269c8ea640 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c92a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269c8ea640 1 -- 172.21.15.27:0/344769288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2c8fd77d0 con 0x55b2c9294630 2024-04-25T01:33:03.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269c8ea640 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c9294a10 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f2694013610 tx=0x7f26940097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abeafbca70c82592 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f269d8ec640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f26840a2d00 0x7f26840a51c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f26900ee400 tx=0x7f26900dd040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f26837fe640 1 -- 172.21.15.27:0/344769288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269400f520 con 0x55b2c9294630 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f26837fe640 1 -- 172.21.15.27:0/344769288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f269400fba0 con 0x55b2c9294630 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.454+0000 7f26837fe640 1 -- 172.21.15.27:0/344769288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26940119d0 con 0x55b2c9294630 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 -- 172.21.15.27:0/344769288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 msgr2=0x55b2c9294a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c9294a10 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f2694013610 tx=0x7f26940097d0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 -- 172.21.15.27:0/344769288 shutdown_connections 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c92a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c929f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 --2- 172.21.15.27:0/344769288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c9294a10 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 -- 172.21.15.27:0/344769288 >> 172.21.15.27:0/344769288 conn(0x55b2c9290950 msgr2=0x55b2c9290d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 -- 172.21.15.27:0/344769288 shutdown_connections 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 -- 172.21.15.27:0/344769288 wait complete. 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 -- start start 2024-04-25T01:33:03.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.455+0000 7f26a2057740 1 Processor -- start 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f26a2057740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294630 0x55b2c929c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f26a2057740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294f50 0x55b2c929ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f26a2057740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c929cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f26a2057740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b2c92a2410 con 0x55b2c9294f50 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f26a2057740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b2c91b4f50 con 0x55b2c929f620 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f26a2057740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b2c92a2590 con 0x55b2c9294630 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f26a2057740 10 client.0 _ll_register_callbacks cb 0x55b2c91f0000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269d0eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294f50 0x55b2c929ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269d0eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294f50 0x55b2c929ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33580/0 (socket says 172.21.15.27:33580) 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269d0eb640 1 -- 172.21.15.27:0/781077799 learned_addr learned my addr 172.21.15.27:0/781077799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269c8ea640 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294630 0x55b2c929c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269c8ea640 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c929cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269d0eb640 1 -- 172.21.15.27:0/781077799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294630 msgr2=0x55b2c929c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269d0eb640 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294630 0x55b2c929c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269d0eb640 1 -- 172.21.15.27:0/781077799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 msgr2=0x55b2c929cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269d0eb640 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c929cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.456+0000 7f269d0eb640 1 -- 172.21.15.27:0/781077799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2c8f0d3f0 con 0x55b2c9294f50 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f269d0eb640 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294f50 0x55b2c929ca80 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f268c00b870 tx=0x7f268c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9d9eb12dc88911 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f2680ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f2680ff9640 1 -- 172.21.15.27:0/781077799 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c020060 con 0x55b2c9294f50 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f2680ff9640 1 -- 172.21.15.27:0/781077799 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f268c004440 con 0x55b2c9294f50 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 1 -- 172.21.15.27:0/781077799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f269400fba0 con 0x55b2c9294f50 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f2680ff9640 1 -- 172.21.15.27:0/781077799 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f268c027050 con 0x55b2c9294f50 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 10 client.17241 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 1 -- 172.21.15.27:0/781077799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b2c8ecd0f0 con 0x55b2c9294f50 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f2680ff9640 1 -- 172.21.15.27:0/781077799 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f269400fba0 con 0x55b2c9294f50 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f2680ff9640 1 -- 172.21.15.27:0/781077799 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f268c01a030 con 0x55b2c9294f50 2024-04-25T01:33:03.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 10 client.17241 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 10 client.17241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 20 client.17241 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b2c92afa40 0x55b2c92b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 4 client.17241 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.457+0000 7f26a2057740 1 -- 172.21.15.27:0/781077799 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2c9206bf0 con 0x55b2c92afa40 2024-04-25T01:33:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.458+0000 7f269c8ea640 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b2c92afa40 0x55b2c92b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.458+0000 7f269c8ea640 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b2c92afa40 0x55b2c92b1e60 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb86d147b2142af4 server_cookie=3dd4238eed85370e in_seq=0 out_seq=0 2024-04-25T01:33:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.458+0000 7f2680ff9640 10 client.17241 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f2680ff9640 1 -- 172.21.15.27:0/781077799 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2c9206bf0 con 0x55b2c92afa40 2024-04-25T01:33:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f2680ff9640 10 client.17241 handle_command_reply: tid=0 2024-04-25T01:33:03.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 client.17241 shutdown 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 -- 172.21.15.27:0/781077799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294f50 msgr2=0x55b2c929ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294f50 0x55b2c929ca80 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f268c00b870 tx=0x7f268c00bd30 comp rx=0 tx=0).stop 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 -- 172.21.15.27:0/781077799 shutdown_connections 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b2c92afa40 0x55b2c92b1e60 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c929cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f2680ff9640 0 client.17241 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294f50 0x55b2c929ca80 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 --2- 172.21.15.27:0/781077799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294630 0x55b2c929c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 -- 172.21.15.27:0/781077799 >> 172.21.15.27:0/781077799 conn(0x55b2c9290950 msgr2=0x55b2c9296650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 -- 172.21.15.27:0/781077799 shutdown_connections 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 1 -- 172.21.15.27:0/781077799 wait complete. 2024-04-25T01:33:03.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.459+0000 7f26a2057740 20 client.17241 trim_cache size 0 max 16384 2024-04-25T01:33:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.477+0000 7f26a2057740 1 Processor -- start 2024-04-25T01:33:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.477+0000 7f26a2057740 1 -- start start 2024-04-25T01:33:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.477+0000 7f26a2057740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c929ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.477+0000 7f26a2057740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c9294f50 0x55b2c92ab330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.477+0000 7f26a2057740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c929f620 0x55b2c929cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.477+0000 7f26a2057740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26981af2c0 con 0x55b2c9294630 2024-04-25T01:33:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.477+0000 7f26a2057740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f26981acb90 con 0x55b2c9294f50 2024-04-25T01:33:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.477+0000 7f26a2057740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26981aaa00 con 0x55b2c929f620 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c929ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c929ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33588/0 (socket says 172.21.15.27:33588) 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 -- 172.21.15.27:0/1516051378 learned_addr learned my addr 172.21.15.27:0/1516051378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269d0eb640 1 --2- 172.21.15.27:0/1516051378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c9294f50 0x55b2c92ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 -- 172.21.15.27:0/1516051378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c929f620 msgr2=0x55b2c929cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 --2- 172.21.15.27:0/1516051378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c929f620 0x55b2c929cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 -- 172.21.15.27:0/1516051378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c9294f50 msgr2=0x55b2c92ab330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 --2- 172.21.15.27:0/1516051378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c9294f50 0x55b2c92ab330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 -- 172.21.15.27:0/1516051378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2c92ae540 con 0x55b2c9294630 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f269c8ea640 1 --2- 172.21.15.27:0/1516051378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c929ec70 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2694000c00 tx=0x7f26940097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28737990c896c20f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f2682ffd640 1 -- 172.21.15.27:0/1516051378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694005040 con 0x55b2c9294630 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f2682ffd640 1 -- 172.21.15.27:0/1516051378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2694004660 con 0x55b2c9294630 2024-04-25T01:33:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.478+0000 7f2682ffd640 1 -- 172.21.15.27:0/1516051378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269400f780 con 0x55b2c9294630 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 -- 172.21.15.27:0/1516051378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 msgr2=0x55b2c929ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 --2- 172.21.15.27:0/1516051378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c929ec70 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2694000c00 tx=0x7f26940097d0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 -- 172.21.15.27:0/1516051378 shutdown_connections 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 --2- 172.21.15.27:0/1516051378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c929f620 0x55b2c929cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 --2- 172.21.15.27:0/1516051378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c9294f50 0x55b2c92ab330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 --2- 172.21.15.27:0/1516051378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c929ec70 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 -- 172.21.15.27:0/1516051378 >> 172.21.15.27:0/1516051378 conn(0x55b2c9290950 msgr2=0x55b2c9295d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 -- 172.21.15.27:0/1516051378 shutdown_connections 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 -- 172.21.15.27:0/1516051378 wait complete. 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 -- start start 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 Processor -- start 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.479+0000 7f26a2057740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c92a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f26a2057740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c9298db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f26a2057740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c92992f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f26a2057740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26981ab050 con 0x55b2c9294630 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f26a2057740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f26981ac4c0 con 0x55b2c929f620 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f26a2057740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26981ab720 con 0x55b2c9294f50 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f26a2057740 10 client.0 _ll_register_callbacks cb 0x7f268c0107c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c92a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c92a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33590/0 (socket says 172.21.15.27:33590) 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 -- 172.21.15.27:0/1441172296 learned_addr learned my addr 172.21.15.27:0/1441172296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269d0eb640 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c9298db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 -- 172.21.15.27:0/1441172296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 msgr2=0x55b2c9298db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c9298db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 -- 172.21.15.27:0/1441172296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 msgr2=0x55b2c92992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c92992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 -- 172.21.15.27:0/1441172296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2c92998b0 con 0x55b2c9294630 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269d0eb640 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c9298db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f269c8ea640 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c92a7a30 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2694013450 tx=0x7f2694004810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2270711d7e9b9ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f2680ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f2680ff9640 1 -- 172.21.15.27:0/1441172296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694004cf0 con 0x55b2c9294630 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f2680ff9640 1 -- 172.21.15.27:0/1441172296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f2694005000 con 0x55b2c9294630 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.480+0000 7f26a2057740 1 -- 172.21.15.27:0/1441172296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2694004660 con 0x55b2c9294630 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f2680ff9640 1 -- 172.21.15.27:0/1441172296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f269403e050 con 0x55b2c9294630 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f26a2057740 10 client.17247 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f26a2057740 1 -- 172.21.15.27:0/1441172296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b2c9315de0 con 0x55b2c9294630 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f2680ff9640 1 -- 172.21.15.27:0/1441172296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2694004660 con 0x55b2c9294630 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f2680ff9640 1 -- 172.21.15.27:0/1441172296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f269402c020 con 0x55b2c9294630 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f26a2057740 10 client.17247 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f26a2057740 10 client.17247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f26a2057740 20 client.17247 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f26a2057740 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b2c93162a0 0x55b2c9318690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f26a2057740 4 client.17247 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f26a2057740 1 -- 172.21.15.27:0/1441172296 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b2c9200f40 con 0x55b2c93162a0 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f269d0eb640 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b2c93162a0 0x55b2c9318690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f269d0eb640 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b2c93162a0 0x55b2c9318690 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bf032a1fd57e723 server_cookie=b5eb884c1cd711d in_seq=0 out_seq=0 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.481+0000 7f2680ff9640 10 client.17247 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f2680ff9640 1 -- 172.21.15.27:0/1441172296 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b2c9200f40 con 0x55b2c93162a0 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f2680ff9640 10 client.17247 handle_command_reply: tid=0 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 client.17247 shutdown 2024-04-25T01:33:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 -- 172.21.15.27:0/1441172296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 msgr2=0x55b2c92a7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c92a7a30 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2694013450 tx=0x7f2694004810 comp rx=0 tx=0).stop 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 -- 172.21.15.27:0/1441172296 shutdown_connections 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b2c93162a0 0x55b2c9318690 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b2c929f620 0x55b2c92992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f2680ff9640 0 client.17247 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b2c9294f50 0x55b2c9298db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 --2- 172.21.15.27:0/1441172296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2c9294630 0x55b2c92a7a30 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 -- 172.21.15.27:0/1441172296 >> 172.21.15.27:0/1441172296 conn(0x55b2c9290950 msgr2=0x55b2c9296610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 -- 172.21.15.27:0/1441172296 shutdown_connections 2024-04-25T01:33:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 1 -- 172.21.15.27:0/1441172296 wait complete. 2024-04-25T01:33:03.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.482+0000 7f26a2057740 20 client.17247 trim_cache size 0 max 16384 2024-04-25T01:33:03.469 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:33:03.469 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:33:03.469 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:33:03.469 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:33:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 -- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f26840a2d00 msgr2=0x7f26840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f26840a2d00 0x7f26840a51c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f26900ee400 tx=0x7f26900dd040 comp rx=0 tx=0).stop 2024-04-25T01:33:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 -- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698106630 msgr2=0x7f269819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698106630 0x7f269819c900 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2690033040 tx=0x7f2690031e60 comp rx=0 tx=0).stop 2024-04-25T01:33:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 -- 172.21.15.27:0/2905296296 shutdown_connections 2024-04-25T01:33:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f26840a2d00 0x7f26840a51c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2698106630 0x7f269819c900 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2698100fe0 0x7f2698198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 --2- 172.21.15.27:0/2905296296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2698100620 0x7f2698198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 -- 172.21.15.27:0/2905296296 >> 172.21.15.27:0/2905296296 conn(0x7f26980776a0 msgr2=0x7f269806a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 -- 172.21.15.27:0/2905296296 shutdown_connections 2024-04-25T01:33:03.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.483+0000 7f269f34f640 1 -- 172.21.15.27:0/2905296296 wait complete. 2024-04-25T01:33:03.472 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:33:03.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:33:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.596+0000 7f286e17e640 1 -- 172.21.15.27:0/1284326982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286810bd60 msgr2=0x7f286810e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.596+0000 7f286e17e640 1 --2- 172.21.15.27:0/1284326982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286810bd60 0x7f286810e1d0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f2860009ca0 tx=0x7f286002a950 comp rx=0 tx=0).stop 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 -- 172.21.15.27:0/1284326982 shutdown_connections 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 --2- 172.21.15.27:0/1284326982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286810bd60 0x7f286810e1d0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 --2- 172.21.15.27:0/1284326982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2868103340 0x7f286810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 --2- 172.21.15.27:0/1284326982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868102a20 0x7f2868102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 -- 172.21.15.27:0/1284326982 >> 172.21.15.27:0/1284326982 conn(0x7f28680777a0 msgr2=0x7f2868077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 -- 172.21.15.27:0/1284326982 shutdown_connections 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 -- 172.21.15.27:0/1284326982 wait complete. 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 Processor -- start 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.597+0000 7f286e17e640 1 -- start start 2024-04-25T01:33:03.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f286e17e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868102a20 0x7f2868100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f286e17e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2868103340 0x7f2868100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f286e17e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f286810bd60 0x7f28681010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f286e17e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2868111b80 con 0x7f2868103340 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f286e17e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2868111a00 con 0x7f286810bd60 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f286e17e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2868111d00 con 0x7f2868102a20 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2868103340 0x7f2868100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2868103340 0x7f2868100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33608/0 (socket says 172.21.15.27:33608) 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 -- 172.21.15.27:0/3278507455 learned_addr learned my addr 172.21.15.27:0/3278507455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f28677fe640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868102a20 0x7f2868100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2867fff640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f286810bd60 0x7f28681010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 -- 172.21.15.27:0/3278507455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868102a20 msgr2=0x7f2868100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868102a20 0x7f2868100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 -- 172.21.15.27:0/3278507455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f286810bd60 msgr2=0x7f28681010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f286810bd60 0x7f28681010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 -- 172.21.15.27:0/3278507455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28681a63c0 con 0x7f2868103340 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2867fff640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f286810bd60 0x7f28681010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2866ffd640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2868103340 0x7f2868100bb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f285800d770 tx=0x7f285800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2864ff9640 1 -- 172.21.15.27:0/3278507455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858017070 con 0x7f2868103340 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28681a66b0 con 0x7f2868103340 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2864ff9640 1 -- 172.21.15.27:0/3278507455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2858005500 con 0x7f2868103340 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f2864ff9640 1 -- 172.21.15.27:0/3278507455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858004e00 con 0x7f2868103340 2024-04-25T01:33:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.598+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28681a6c00 con 0x7f2868103340 2024-04-25T01:33:03.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.599+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28680689b0 con 0x7f2868103340 2024-04-25T01:33:03.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.600+0000 7f2864ff9640 1 -- 172.21.15.27:0/3278507455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2858004fa0 con 0x7f2868103340 2024-04-25T01:33:03.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.601+0000 7f2864ff9640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f28540a2db0 0x7f28540a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.601+0000 7f2864ff9640 1 -- 172.21.15.27:0/3278507455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f28580cf250 con 0x7f2868103340 2024-04-25T01:33:03.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.601+0000 7f28677fe640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f28540a2db0 0x7f28540a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.602+0000 7f28677fe640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f28540a2db0 0x7f28540a5270 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2850005e10 tx=0x7f2850005da0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.606+0000 7f2864ff9640 1 -- 172.21.15.27:0/3278507455 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f285809ba80 con 0x7f2868103340 2024-04-25T01:33:03.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.827+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28681a6e60 con 0x7f2868103340 2024-04-25T01:33:03.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.828+0000 7f2864ff9640 1 -- 172.21.15.27:0/3278507455 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7f285809b380 con 0x7f2868103340 2024-04-25T01:33:03.806 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:33:03.806 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:33:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f28540a2db0 msgr2=0x7f28540a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f28540a2db0 0x7f28540a5270 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2850005e10 tx=0x7f2850005da0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2868103340 msgr2=0x7f2868100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2868103340 0x7f2868100bb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f285800d770 tx=0x7f285800dc30 comp rx=0 tx=0).stop 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 shutdown_connections 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f286810bd60 0x7f28681010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f28540a2db0 0x7f28540a5270 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2868103340 0x7f2868100bb0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 --2- 172.21.15.27:0/3278507455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868102a20 0x7f2868100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 >> 172.21.15.27:0/3278507455 conn(0x7f28680777a0 msgr2=0x7f28680fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 shutdown_connections 2024-04-25T01:33:03.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.830+0000 7f286e17e640 1 -- 172.21.15.27:0/3278507455 wait complete. 2024-04-25T01:33:03.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:33:03.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 -- 172.21.15.27:0/1282978828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 msgr2=0x7f964c10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 --2- 172.21.15.27:0/1282978828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 0x7f964c10e5e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9644009cd0 tx=0x7f964402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 -- 172.21.15.27:0/1282978828 shutdown_connections 2024-04-25T01:33:03.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 --2- 172.21.15.27:0/1282978828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 0x7f964c10e5e0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 --2- 172.21.15.27:0/1282978828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f964c0ffc40 0x7f964c1000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 --2- 172.21.15.27:0/1282978828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f964c107720 0x7f964c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 -- 172.21.15.27:0/1282978828 >> 172.21.15.27:0/1282978828 conn(0x7f964c076c00 msgr2=0x7f964c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 -- 172.21.15.27:0/1282978828 shutdown_connections 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.944+0000 7f9653a72640 1 -- 172.21.15.27:0/1282978828 wait complete. 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f9653a72640 1 Processor -- start 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f9653a72640 1 -- start start 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f9653a72640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f964c0ffc40 0x7f964c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f9653a72640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 0x7f964c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f9653a72640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f964c107720 0x7f964c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f9653a72640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f964c111b80 con 0x7f964c100600 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f9653a72640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f964c111a00 con 0x7f964c107720 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f9653a72640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f964c111d00 con 0x7f964c0ffc40 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f965100d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 0x7f964c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f965100d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 0x7f964c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33628/0 (socket says 172.21.15.27:33628) 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f965100d640 1 -- 172.21.15.27:0/882849568 learned_addr learned my addr 172.21.15.27:0/882849568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f965200f640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f964c107720 0x7f964c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f965180e640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f964c0ffc40 0x7f964c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f965100d640 1 -- 172.21.15.27:0/882849568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f964c0ffc40 msgr2=0x7f964c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.945+0000 7f965100d640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f964c0ffc40 0x7f964c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f965100d640 1 -- 172.21.15.27:0/882849568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f964c107720 msgr2=0x7f964c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f965100d640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f964c107720 0x7f964c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f965100d640 1 -- 172.21.15.27:0/882849568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964c19cfa0 con 0x7f964c100600 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f965200f640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f964c107720 0x7f964c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f965100d640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 0x7f964c198540 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f964000d950 tx=0x7f964000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f963effd640 1 -- 172.21.15.27:0/882849568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640017070 con 0x7f964c100600 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f963effd640 1 -- 172.21.15.27:0/882849568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9640009c30 con 0x7f964c100600 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f9653a72640 1 -- 172.21.15.27:0/882849568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f964c19d290 con 0x7f964c100600 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f963effd640 1 -- 172.21.15.27:0/882849568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964000f6b0 con 0x7f964c100600 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.946+0000 7f9653a72640 1 -- 172.21.15.27:0/882849568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f964c1a4b20 con 0x7f964c100600 2024-04-25T01:33:03.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.948+0000 7f963effd640 1 -- 172.21.15.27:0/882849568 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f96400109d0 con 0x7f964c100600 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.948+0000 7f965677d740 1 Processor -- start 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965677d740 1 -- start start 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965677d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f40960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965677d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f4b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965677d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 0x556d49f4d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965677d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556d49ea5670 con 0x556d49f40580 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965677d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556d49e4d970 con 0x556d49f40ea0 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965677d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556d49ea1630 con 0x556d49f4b570 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f40960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f963effd640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f96380a2d80 0x7f96380a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965180e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f4b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f40960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33642/0 (socket says 172.21.15.27:33642) 2024-04-25T01:33:03.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 -- 172.21.15.27:0/402199205 learned_addr learned my addr 172.21.15.27:0/402199205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965100d640 1 --2- 172.21.15.27:0/402199205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 0x556d49f4d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f96380a2d80 0x7f96380a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 -- 172.21.15.27:0/402199205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 msgr2=0x556d49f4d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 --2- 172.21.15.27:0/402199205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 0x556d49f4d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 -- 172.21.15.27:0/402199205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 msgr2=0x556d49f4b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 --2- 172.21.15.27:0/402199205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f4b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f963effd640 1 -- 172.21.15.27:0/882849568 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f96400cf480 con 0x7f964c100600 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.949+0000 7f965200f640 1 -- 172.21.15.27:0/402199205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d49e71400 con 0x556d49f40580 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.950+0000 7f965200f640 1 --2- 172.21.15.27:0/402199205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f40960 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9644009cd0 tx=0x7f9644031c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53439851a8b7d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.950+0000 7f9637fff640 1 -- 172.21.15.27:0/402199205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644004240 con 0x556d49f40580 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.950+0000 7f965200f640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f96380a2d80 0x7f96380a5240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f964404d000 tx=0x7f964404b000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.950+0000 7f9637fff640 1 -- 172.21.15.27:0/402199205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9644036040 con 0x556d49f40580 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.950+0000 7f9637fff640 1 -- 172.21.15.27:0/402199205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644030d40 con 0x556d49f40580 2024-04-25T01:33:03.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 -- 172.21.15.27:0/402199205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 msgr2=0x556d49f40960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 --2- 172.21.15.27:0/402199205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f40960 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9644009cd0 tx=0x7f9644031c00 comp rx=0 tx=0).stop 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 -- 172.21.15.27:0/402199205 shutdown_connections 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 --2- 172.21.15.27:0/402199205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 0x556d49f4d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 --2- 172.21.15.27:0/402199205 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f4b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 --2- 172.21.15.27:0/402199205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f40960 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 -- 172.21.15.27:0/402199205 >> 172.21.15.27:0/402199205 conn(0x556d49f3c8a0 msgr2=0x556d49f3cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 -- 172.21.15.27:0/402199205 shutdown_connections 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 -- 172.21.15.27:0/402199205 wait complete. 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 -- start start 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 Processor -- start 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f4af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.951+0000 7f965677d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f481f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965677d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 0x556d49f48770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965677d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556d49f4e360 con 0x556d49f40580 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965677d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556d49e61050 con 0x556d49f40ea0 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965677d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556d49f4e4e0 con 0x556d49f4b570 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f4af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965677d740 10 client.0 _ll_register_callbacks cb 0x556d49bb8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f4af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33652/0 (socket says 172.21.15.27:33652) 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 -- 172.21.15.27:0/124000072 learned_addr learned my addr 172.21.15.27:0/124000072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965100d640 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965200f640 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 0x556d49f48770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 -- 172.21.15.27:0/124000072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 msgr2=0x556d49f48770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 0x556d49f48770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 -- 172.21.15.27:0/124000072 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 msgr2=0x556d49f481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f481f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 -- 172.21.15.27:0/124000072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d49e32dc0 con 0x556d49f40580 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965100d640 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965180e640 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f4af00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f96480106b0 tx=0x7f9648010b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e3fd84aaa4db83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f96357fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f96357fa640 1 -- 172.21.15.27:0/124000072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964801d060 con 0x556d49f40580 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f96357fa640 1 -- 172.21.15.27:0/124000072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f964800ed20 con 0x556d49f40580 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965677d740 1 -- 172.21.15.27:0/124000072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9644036040 con 0x556d49f40580 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f96357fa640 1 -- 172.21.15.27:0/124000072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9648024050 con 0x556d49f40580 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965677d740 10 client.17265 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.952+0000 7f965677d740 1 -- 172.21.15.27:0/124000072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d49bb93f0 con 0x556d49f40580 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f96357fa640 1 -- 172.21.15.27:0/124000072 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9644036040 con 0x556d49f40580 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f96357fa640 1 -- 172.21.15.27:0/124000072 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9648016450 con 0x556d49f40580 2024-04-25T01:33:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f965677d740 10 client.17265 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f965677d740 10 client.17265 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f965677d740 20 client.17265 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f965677d740 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556d49f5b690 0x556d49f5db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f965677d740 4 client.17265 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f965677d740 1 -- 172.21.15.27:0/124000072 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d49eb2bb0 con 0x556d49f5b690 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.953+0000 7f965100d640 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556d49f5b690 0x556d49f5db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.954+0000 7f96357fa640 10 client.17265 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.954+0000 7f965100d640 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556d49f5b690 0x556d49f5db20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=91caf809019aaeb1 server_cookie=e0b02c379990e727 in_seq=0 out_seq=0 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.954+0000 7f96357fa640 1 -- 172.21.15.27:0/124000072 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d49eb2bb0 con 0x556d49f5b690 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.954+0000 7f96357fa640 10 client.17265 handle_command_reply: tid=0 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 client.17265 shutdown 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 -- 172.21.15.27:0/124000072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 msgr2=0x556d49f4af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f4af00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f96480106b0 tx=0x7f9648010b70 comp rx=0 tx=0).stop 2024-04-25T01:33:03.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 -- 172.21.15.27:0/124000072 shutdown_connections 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556d49f5b690 0x556d49f5db20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f4b570 0x556d49f48770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f40ea0 0x556d49f481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f96357fa640 0 client.17265 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 --2- 172.21.15.27:0/124000072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f4af00 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 -- 172.21.15.27:0/124000072 >> 172.21.15.27:0/124000072 conn(0x556d49f3c8a0 msgr2=0x556d49f3bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 -- 172.21.15.27:0/124000072 shutdown_connections 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 1 -- 172.21.15.27:0/124000072 wait complete. 2024-04-25T01:33:03.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.955+0000 7f965677d740 20 client.17265 trim_cache size 0 max 16384 2024-04-25T01:33:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.968+0000 7f965677d740 1 Processor -- start 2024-04-25T01:33:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.968+0000 7f965677d740 1 -- start start 2024-04-25T01:33:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.968+0000 7f965677d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f57220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965677d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 0x556d49f53b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965677d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 0x556d49f4aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965677d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f964c1b15b0 con 0x556d49f40580 2024-04-25T01:33:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965677d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f964c1aee80 con 0x556d49f4b570 2024-04-25T01:33:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965677d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f964c1accf0 con 0x556d49f40ea0 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f57220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f57220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33658/0 (socket says 172.21.15.27:33658) 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 -- 172.21.15.27:0/3913005946 learned_addr learned my addr 172.21.15.27:0/3913005946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 -- 172.21.15.27:0/3913005946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 msgr2=0x556d49f53b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 --2- 172.21.15.27:0/3913005946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 0x556d49f53b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 -- 172.21.15.27:0/3913005946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 msgr2=0x556d49f4aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 --2- 172.21.15.27:0/3913005946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 0x556d49f4aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 -- 172.21.15.27:0/3913005946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d49f48ae0 con 0x556d49f40580 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f965180e640 1 --2- 172.21.15.27:0/3913005946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f57220 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f9648000c00 tx=0x7f964800e800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e058ec2f5a940ad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f96377fe640 1 -- 172.21.15.27:0/3913005946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648016330 con 0x556d49f40580 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f96377fe640 1 -- 172.21.15.27:0/3913005946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96480164d0 con 0x556d49f40580 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.969+0000 7f96377fe640 1 -- 172.21.15.27:0/3913005946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648028a20 con 0x556d49f40580 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 -- 172.21.15.27:0/3913005946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 msgr2=0x556d49f57220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 --2- 172.21.15.27:0/3913005946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f57220 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f9648000c00 tx=0x7f964800e800 comp rx=0 tx=0).stop 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 -- 172.21.15.27:0/3913005946 shutdown_connections 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 --2- 172.21.15.27:0/3913005946 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 0x556d49f4aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 --2- 172.21.15.27:0/3913005946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 0x556d49f53b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 --2- 172.21.15.27:0/3913005946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49f57220 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 -- 172.21.15.27:0/3913005946 >> 172.21.15.27:0/3913005946 conn(0x556d49f3c8a0 msgr2=0x556d49f43860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 -- 172.21.15.27:0/3913005946 shutdown_connections 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 -- 172.21.15.27:0/3913005946 wait complete. 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 -- start start 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 Processor -- start 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49fb9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.970+0000 7f965677d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 0x556d49fba160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965677d740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 0x556d49fc4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965677d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f964c1ad340 con 0x556d49f40580 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965677d740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f964c1ae7b0 con 0x556d49f4b570 2024-04-25T01:33:03.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965677d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f964c1ada10 con 0x556d49f40ea0 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965677d740 10 client.0 _ll_register_callbacks cb 0x7f964800b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49fb9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49fb9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33662/0 (socket says 172.21.15.27:33662) 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 -- 172.21.15.27:0/1193978376 learned_addr learned my addr 172.21.15.27:0/1193978376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965100d640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 0x556d49fba160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 -- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 msgr2=0x556d49fba160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 0x556d49fba160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965200f640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 0x556d49fc4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 -- 172.21.15.27:0/1193978376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 msgr2=0x556d49fc4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 0x556d49fc4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 -- 172.21.15.27:0/1193978376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d49c837d0 con 0x556d49f40580 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965100d640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 0x556d49fba160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965200f640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 0x556d49fc4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:03.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965180e640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49fb9c20 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f96480285a0 tx=0x7f964800baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f7b54b2562d332 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f96357fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f96357fa640 1 -- 172.21.15.27:0/1193978376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964801d040 con 0x556d49f40580 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f96357fa640 1 -- 172.21.15.27:0/1193978376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f964800eb20 con 0x556d49f40580 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965677d740 1 -- 172.21.15.27:0/1193978376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96480164d0 con 0x556d49f40580 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f96357fa640 1 -- 172.21.15.27:0/1193978376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f964800ed20 con 0x556d49f40580 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965677d740 10 client.17271 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.971+0000 7f965677d740 1 -- 172.21.15.27:0/1193978376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d49fc6130 con 0x556d49f40580 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f96357fa640 1 -- 172.21.15.27:0/1193978376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96480164d0 con 0x556d49f40580 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f96357fa640 1 -- 172.21.15.27:0/1193978376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9648018550 con 0x556d49f40580 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f965677d740 10 client.17271 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f965677d740 10 client.17271 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f965677d740 20 client.17271 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f965677d740 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556d49fc6730 0x556d49fc8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f965677d740 4 client.17271 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f965677d740 1 -- 172.21.15.27:0/1193978376 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d49eacf00 con 0x556d49fc6730 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.972+0000 7f965100d640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556d49fc6730 0x556d49fc8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965100d640 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556d49fc6730 0x556d49fc8b50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2cd97aa440d8feb server_cookie=7c7008294c916062 in_seq=0 out_seq=0 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f96357fa640 10 client.17271 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f96357fa640 1 -- 172.21.15.27:0/1193978376 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d49eacf00 con 0x556d49fc6730 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f96357fa640 10 client.17271 handle_command_reply: tid=0 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 client.17271 shutdown 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 -- 172.21.15.27:0/1193978376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 msgr2=0x556d49fb9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49fb9c20 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f96480285a0 tx=0x7f964800baf0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 -- 172.21.15.27:0/1193978376 shutdown_connections 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x556d49fc6730 0x556d49fc8b50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556d49f4b570 0x556d49fc4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556d49f40ea0 0x556d49fba160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f96357fa640 0 client.17271 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:33:03.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 --2- 172.21.15.27:0/1193978376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d49f40580 0x556d49fb9c20 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 -- 172.21.15.27:0/1193978376 >> 172.21.15.27:0/1193978376 conn(0x556d49f3c8a0 msgr2=0x556d49f41cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 -- 172.21.15.27:0/1193978376 shutdown_connections 2024-04-25T01:33:03.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 1 -- 172.21.15.27:0/1193978376 wait complete. 2024-04-25T01:33:03.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.973+0000 7f965677d740 20 client.17271 trim_cache size 0 max 16384 2024-04-25T01:33:03.960 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:33:03.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 -- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f96380a2d80 msgr2=0x7f96380a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f96380a2d80 0x7f96380a5240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f964404d000 tx=0x7f964404b000 comp rx=0 tx=0).stop 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 -- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 msgr2=0x7f964c198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 0x7f964c198540 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f964000d950 tx=0x7f964000de10 comp rx=0 tx=0).stop 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 -- 172.21.15.27:0/882849568 shutdown_connections 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f964c107720 0x7f964c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f96380a2d80 0x7f96380a5240 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f964c100600 0x7f964c198540 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 --2- 172.21.15.27:0/882849568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f964c0ffc40 0x7f964c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 -- 172.21.15.27:0/882849568 >> 172.21.15.27:0/882849568 conn(0x7f964c076c00 msgr2=0x7f964c0fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 -- 172.21.15.27:0/882849568 shutdown_connections 2024-04-25T01:33:03.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:03.974+0000 7f9653a72640 1 -- 172.21.15.27:0/882849568 wait complete. 2024-04-25T01:33:03.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:33:04.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.088+0000 7f9c89d5a640 1 -- 172.21.15.27:0/1895347791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 msgr2=0x7f9c84107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.088+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/1895347791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 0x7f9c84107b10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009cd0 tx=0x7f9c7402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.088+0000 7f9c89d5a640 1 -- 172.21.15.27:0/1895347791 shutdown_connections 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.088+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/1895347791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c8406a780 0x7f9c8410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.088+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/1895347791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c84069dc0 0x7f9c8406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.088+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/1895347791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 0x7f9c84107b10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.089+0000 7f9c89d5a640 1 -- 172.21.15.27:0/1895347791 >> 172.21.15.27:0/1895347791 conn(0x7f9c840776a0 msgr2=0x7f9c84077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.089+0000 7f9c89d5a640 1 -- 172.21.15.27:0/1895347791 shutdown_connections 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.089+0000 7f9c89d5a640 1 -- 172.21.15.27:0/1895347791 wait complete. 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.089+0000 7f9c89d5a640 1 Processor -- start 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.089+0000 7f9c89d5a640 1 -- start start 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c89d5a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c84069dc0 0x7f9c84197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c89d5a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c8406a780 0x7f9c84198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c89d5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 0x7f9c8419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c89d5a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c84111b40 con 0x7f9c84107730 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c89d5a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9c841119c0 con 0x7f9c84069dc0 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c89d5a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9c84111cc0 con 0x7f9c8406a780 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 0x7f9c8419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 0x7f9c8419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33674/0 (socket says 172.21.15.27:33674) 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 -- 172.21.15.27:0/2335573506 learned_addr learned my addr 172.21.15.27:0/2335573506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c8406a780 0x7f9c84198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c837fe640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c84069dc0 0x7f9c84197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 -- 172.21.15.27:0/2335573506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c8406a780 msgr2=0x7f9c84198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c8406a780 0x7f9c84198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 -- 172.21.15.27:0/2335573506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c84069dc0 msgr2=0x7f9c84197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c84069dc0 0x7f9c84197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 -- 172.21.15.27:0/2335573506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8419cfb0 con 0x7f9c84107730 2024-04-25T01:33:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c83fff640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 0x7f9c8419c880 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00d750 tx=0x7f9c7c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c80ff9640 1 -- 172.21.15.27:0/2335573506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c017070 con 0x7f9c84107730 2024-04-25T01:33:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c89d5a640 1 -- 172.21.15.27:0/2335573506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c8419d2a0 con 0x7f9c84107730 2024-04-25T01:33:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c80ff9640 1 -- 172.21.15.27:0/2335573506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c7c005500 con 0x7f9c84107730 2024-04-25T01:33:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c80ff9640 1 -- 172.21.15.27:0/2335573506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c004d90 con 0x7f9c84107730 2024-04-25T01:33:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.090+0000 7f9c89d5a640 1 -- 172.21.15.27:0/2335573506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c841a4b30 con 0x7f9c84107730 2024-04-25T01:33:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c80ff9640 1 -- 172.21.15.27:0/2335573506 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9c7c005020 con 0x7f9c84107730 2024-04-25T01:33:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c8ca67740 1 Processor -- start 2024-04-25T01:33:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c8ca67740 1 -- start start 2024-04-25T01:33:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c80ff9640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9c700a2d00 0x7f9c700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998460b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 0x55998461a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x55998461cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559984570670 con 0x55998460b480 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559984518970 con 0x55998460bda0 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55998456c630 con 0x55998461a980 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9c700a2d00 0x7f9c700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c837fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998460b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c837fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998460b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33686/0 (socket says 172.21.15.27:33686) 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c837fe640 1 -- 172.21.15.27:0/886006226 learned_addr learned my addr 172.21.15.27:0/886006226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.093+0000 7f9c80ff9640 1 -- 172.21.15.27:0/2335573506 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f9c7c0cf170 con 0x7f9c84107730 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c83fff640 1 --2- 172.21.15.27:0/886006226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x55998461cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c837fe640 1 -- 172.21.15.27:0/886006226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 msgr2=0x55998461cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c837fe640 1 --2- 172.21.15.27:0/886006226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x55998461cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c837fe640 1 -- 172.21.15.27:0/886006226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 msgr2=0x55998461a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c837fe640 1 --2- 172.21.15.27:0/886006226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 0x55998461a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c837fe640 1 -- 172.21.15.27:0/886006226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55998453c400 con 0x55998460b480 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9c700a2d00 0x7f9c700a51c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0058a0 tx=0x7f9c6c00b770 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c837fe640 1 --2- 172.21.15.27:0/886006226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998460b860 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9c74000f80 tx=0x7f9c74009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367cb481106aeb7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c79ffb640 1 -- 172.21.15.27:0/886006226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c740318c0 con 0x55998460b480 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c79ffb640 1 -- 172.21.15.27:0/886006226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c74036040 con 0x55998460b480 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.094+0000 7f9c79ffb640 1 -- 172.21.15.27:0/886006226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c74004540 con 0x55998460b480 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 -- 172.21.15.27:0/886006226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 msgr2=0x55998460b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/886006226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998460b860 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9c74000f80 tx=0x7f9c74009a20 comp rx=0 tx=0).stop 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 -- 172.21.15.27:0/886006226 shutdown_connections 2024-04-25T01:33:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/886006226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x55998461cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/886006226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 0x55998461a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/886006226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998460b860 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 -- 172.21.15.27:0/886006226 >> 172.21.15.27:0/886006226 conn(0x5599846078a0 msgr2=0x559984607c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 -- 172.21.15.27:0/886006226 shutdown_connections 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 -- 172.21.15.27:0/886006226 wait complete. 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 -- start start 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.095+0000 7f9c8ca67740 1 Processor -- start 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998461a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 0x5599846175a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x559984617ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55998461d6e0 con 0x55998460b480 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55998452c050 con 0x55998460bda0 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55998461d860 con 0x55998461a980 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c8ca67740 10 client.0 _ll_register_callbacks cb 0x559984283be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998461a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998461a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33688/0 (socket says 172.21.15.27:33688) 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 -- 172.21.15.27:0/2904095363 learned_addr learned my addr 172.21.15.27:0/2904095363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 0x5599846175a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c83fff640 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x559984617ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 -- 172.21.15.27:0/2904095363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 msgr2=0x559984617ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x559984617ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 -- 172.21.15.27:0/2904095363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 msgr2=0x5599846175a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 0x5599846175a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 -- 172.21.15.27:0/2904095363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599844fddc0 con 0x55998460b480 2024-04-25T01:33:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 0x5599846175a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c837fe640 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998461a2b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9c7402ae00 tx=0x7f9c74005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21dec066c6d8f8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c62ffd640 1 -- 172.21.15.27:0/2904095363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c74055060 con 0x55998460b480 2024-04-25T01:33:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c8ca67740 1 -- 172.21.15.27:0/2904095363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c74036040 con 0x55998460b480 2024-04-25T01:33:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.096+0000 7f9c62ffd640 1 -- 172.21.15.27:0/2904095363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f9c7402e040 con 0x55998460b480 2024-04-25T01:33:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c62ffd640 1 -- 172.21.15.27:0/2904095363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9c7405c050 con 0x55998460b480 2024-04-25T01:33:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c8ca67740 10 client.17283 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:04.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c8ca67740 1 -- 172.21.15.27:0/2904095363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599842843f0 con 0x55998460b480 2024-04-25T01:33:04.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c62ffd640 1 -- 172.21.15.27:0/2904095363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c74036040 con 0x55998460b480 2024-04-25T01:33:04.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c62ffd640 1 -- 172.21.15.27:0/2904095363 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9c74030070 con 0x55998460b480 2024-04-25T01:33:04.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c8ca67740 10 client.17283 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:04.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c8ca67740 10 client.17283 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:33:04.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c8ca67740 20 client.17283 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55998462a7f0 0x55998462cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c8ca67740 4 client.17283 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c8ca67740 1 -- 172.21.15.27:0/2904095363 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55998457dbb0 con 0x55998462a7f0 2024-04-25T01:33:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.097+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55998462a7f0 0x55998462cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.098+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55998462a7f0 0x55998462cbe0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=79c179733314fb06 server_cookie=fc0f2b451f8ae747 in_seq=0 out_seq=0 2024-04-25T01:33:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.098+0000 7f9c62ffd640 10 client.17283 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:33:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c62ffd640 1 -- 172.21.15.27:0/2904095363 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55998457dbb0 con 0x55998462a7f0 2024-04-25T01:33:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c62ffd640 10 client.17283 handle_command_reply: tid=0 2024-04-25T01:33:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 client.17283 shutdown 2024-04-25T01:33:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 -- 172.21.15.27:0/2904095363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 msgr2=0x55998461a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998461a2b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9c7402ae00 tx=0x7f9c74005fa0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 -- 172.21.15.27:0/2904095363 shutdown_connections 2024-04-25T01:33:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55998462a7f0 0x55998462cbe0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x559984617ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c62ffd640 0 client.17283 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:33:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460bda0 0x5599846175a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/2904095363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460b480 0x55998461a2b0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 -- 172.21.15.27:0/2904095363 >> 172.21.15.27:0/2904095363 conn(0x5599846078a0 msgr2=0x559984606b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 -- 172.21.15.27:0/2904095363 shutdown_connections 2024-04-25T01:33:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 1 -- 172.21.15.27:0/2904095363 wait complete. 2024-04-25T01:33:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.099+0000 7f9c8ca67740 20 client.17283 trim_cache size 0 max 16384 2024-04-25T01:33:04.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c8ca67740 1 Processor -- start 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c8ca67740 1 -- start start 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 0x559984612c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998460bda0 0x559984619de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998461a980 0x559984632180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c841af380 con 0x55998461a980 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9c841acc50 con 0x55998460b480 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9c841aaac0 con 0x55998460bda0 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998461a980 0x559984632180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998461a980 0x559984632180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33694/0 (socket says 172.21.15.27:33694) 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 -- 172.21.15.27:0/3005034825 learned_addr learned my addr 172.21.15.27:0/3005034825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c837fe640 1 --2- 172.21.15.27:0/3005034825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 0x559984612c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 -- 172.21.15.27:0/3005034825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998460bda0 msgr2=0x559984619de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 --2- 172.21.15.27:0/3005034825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998460bda0 0x559984619de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 -- 172.21.15.27:0/3005034825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 msgr2=0x559984612c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 --2- 172.21.15.27:0/3005034825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 0x559984612c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 -- 172.21.15.27:0/3005034825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55998434e7d0 con 0x55998461a980 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c83fff640 1 --2- 172.21.15.27:0/3005034825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998461a980 0x559984632180 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009d20 tx=0x7f9c7c087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72010cdefde592a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.113+0000 7f9c797fa640 1 -- 172.21.15.27:0/3005034825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c020040 con 0x55998461a980 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c797fa640 1 -- 172.21.15.27:0/3005034825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c7c09b9a0 con 0x55998461a980 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c797fa640 1 -- 172.21.15.27:0/3005034825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0993d0 con 0x55998461a980 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 -- 172.21.15.27:0/3005034825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998461a980 msgr2=0x559984632180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/3005034825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998461a980 0x559984632180 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009d20 tx=0x7f9c7c087040 comp rx=0 tx=0).stop 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 -- 172.21.15.27:0/3005034825 shutdown_connections 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/3005034825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998461a980 0x559984632180 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/3005034825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998460bda0 0x559984619de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/3005034825 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 0x559984612c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 -- 172.21.15.27:0/3005034825 >> 172.21.15.27:0/3005034825 conn(0x5599846078a0 msgr2=0x5599846062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 -- 172.21.15.27:0/3005034825 shutdown_connections 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 -- 172.21.15.27:0/3005034825 wait complete. 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.114+0000 7f9c8ca67740 1 -- start start 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c8ca67740 1 Processor -- start 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 0x559984622fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460bda0 0x55998461e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c8ca67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x55998461e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c841ab110 con 0x55998460bda0 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f9c841ac580 con 0x55998460b480 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c8ca67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9c841ab7e0 con 0x55998461a980 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c8ca67740 10 client.0 _ll_register_callbacks cb 0x7f9c7c004f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460bda0 0x55998461e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460bda0 0x55998461e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33698/0 (socket says 172.21.15.27:33698) 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 -- 172.21.15.27:0/1253851427 learned_addr learned my addr 172.21.15.27:0/1253851427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c837fe640 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 0x559984622fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 -- 172.21.15.27:0/1253851427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 msgr2=0x55998461e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x55998461e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 -- 172.21.15.27:0/1253851427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 msgr2=0x559984622fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 0x559984622fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 -- 172.21.15.27:0/1253851427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55998461f150 con 0x55998460bda0 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c82ffd640 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460bda0 0x55998461e310 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00b3d0 tx=0x7f9c6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807d43613db26c61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.115+0000 7f9c62ffd640 1 -- 172.21.15.27:0/1253851427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c020610 con 0x55998460bda0 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 1 -- 172.21.15.27:0/1253851427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c7c09b9a0 con 0x55998460bda0 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c62ffd640 1 -- 172.21.15.27:0/1253851427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f9c6c0207b0 con 0x55998460bda0 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c62ffd640 1 -- 172.21.15.27:0/1253851427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9c6c037050 con 0x55998460bda0 2024-04-25T01:33:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 10 client.17289 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 1 -- 172.21.15.27:0/1253851427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559984690b80 con 0x55998460bda0 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c62ffd640 1 -- 172.21.15.27:0/1253851427 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c7c09b9a0 con 0x55998460bda0 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c62ffd640 1 -- 172.21.15.27:0/1253851427 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9c6c02fa80 con 0x55998460bda0 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 10 client.17289 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 10 client.17289 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 20 client.17289 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5599846912e0 0x559984693700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 4 client.17289 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c8ca67740 1 -- 172.21.15.27:0/1253851427 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559984617e70 con 0x5599846912e0 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.116+0000 7f9c837fe640 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5599846912e0 0x559984693700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c62ffd640 10 client.17289 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c837fe640 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5599846912e0 0x559984693700 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b93ccd342897f9e0 server_cookie=934b8f3e3e26c633 in_seq=0 out_seq=0 2024-04-25T01:33:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c62ffd640 1 -- 172.21.15.27:0/1253851427 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9c74036020 con 0x5599846912e0 2024-04-25T01:33:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c62ffd640 10 client.17289 handle_command_reply: tid=0 2024-04-25T01:33:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 client.17289 shutdown 2024-04-25T01:33:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 -- 172.21.15.27:0/1253851427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460bda0 msgr2=0x55998461e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460bda0 0x55998461e310 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00b3d0 tx=0x7f9c6c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 -- 172.21.15.27:0/1253851427 shutdown_connections 2024-04-25T01:33:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5599846912e0 0x559984693700 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55998461a980 0x55998461e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55998460bda0 0x55998461e310 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 --2- 172.21.15.27:0/1253851427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55998460b480 0x559984622fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c62ffd640 0 client.17289 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:33:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 -- 172.21.15.27:0/1253851427 >> 172.21.15.27:0/1253851427 conn(0x5599846078a0 msgr2=0x559984612af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 -- 172.21.15.27:0/1253851427 shutdown_connections 2024-04-25T01:33:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 1 -- 172.21.15.27:0/1253851427 wait complete. 2024-04-25T01:33:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.117+0000 7f9c8ca67740 20 client.17289 trim_cache size 0 max 16384 2024-04-25T01:33:04.104 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:33:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 -- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9c700a2d00 msgr2=0x7f9c700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9c700a2d00 0x7f9c700a51c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0058a0 tx=0x7f9c6c00b770 comp rx=0 tx=0).stop 2024-04-25T01:33:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 -- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 msgr2=0x7f9c8419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 0x7f9c8419c880 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00d750 tx=0x7f9c7c00dc10 comp rx=0 tx=0).stop 2024-04-25T01:33:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 -- 172.21.15.27:0/2335573506 shutdown_connections 2024-04-25T01:33:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f9c700a2d00 0x7f9c700a51c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c84107730 0x7f9c8419c880 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c8406a780 0x7f9c84198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 --2- 172.21.15.27:0/2335573506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c84069dc0 0x7f9c84197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 -- 172.21.15.27:0/2335573506 >> 172.21.15.27:0/2335573506 conn(0x7f9c840776a0 msgr2=0x7f9c840fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 -- 172.21.15.27:0/2335573506 shutdown_connections 2024-04-25T01:33:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.118+0000 7f9c89d5a640 1 -- 172.21.15.27:0/2335573506 wait complete. 2024-04-25T01:33:04.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:33:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 -- 172.21.15.27:0/2531273916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 msgr2=0x7f75d810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2531273916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 0x7f75d810e1a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009ca0 tx=0x7f75d402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 -- 172.21.15.27:0/2531273916 shutdown_connections 2024-04-25T01:33:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2531273916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 0x7f75d810e1a0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2531273916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d81032e0 0x7f75d810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2531273916 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75d81029c0 0x7f75d8102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 -- 172.21.15.27:0/2531273916 >> 172.21.15.27:0/2531273916 conn(0x7f75d8077740 msgr2=0x7f75d8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 -- 172.21.15.27:0/2531273916 shutdown_connections 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.233+0000 7f75dff5a640 1 -- 172.21.15.27:0/2531273916 wait complete. 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75dff5a640 1 Processor -- start 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75dff5a640 1 -- start start 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75dff5a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d81029c0 0x7f75d81981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75dff5a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75d81032e0 0x7f75d81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75dff5a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 0x7f75d819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75dff5a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f75d8111bc0 con 0x7f75d810bd00 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75dff5a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f75d8111a40 con 0x7f75d81032e0 2024-04-25T01:33:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75dff5a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75d8111d40 con 0x7f75d81029c0 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75de4f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 0x7f75d819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75de4f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 0x7f75d819ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33716/0 (socket says 172.21.15.27:33716) 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.234+0000 7f75de4f7640 1 -- 172.21.15.27:0/2806783154 learned_addr learned my addr 172.21.15.27:0/2806783154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d81029c0 0x7f75d81981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75de4f7640 1 -- 172.21.15.27:0/2806783154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d81029c0 msgr2=0x7f75d81981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75de4f7640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d81029c0 0x7f75d81981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75d81032e0 0x7f75d81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75de4f7640 1 -- 172.21.15.27:0/2806783154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75d81032e0 msgr2=0x7f75d81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75de4f7640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75d81032e0 0x7f75d81986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75de4f7640 1 -- 172.21.15.27:0/2806783154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d819d190 con 0x7f75d810bd00 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d81029c0 0x7f75d81981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75de4f7640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 0x7f75d819ca60 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009dc0 tx=0x7f75d4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75ceffd640 1 -- 172.21.15.27:0/2806783154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d4046070 con 0x7f75d810bd00 2024-04-25T01:33:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75ceffd640 1 -- 172.21.15.27:0/2806783154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d402e870 con 0x7f75d810bd00 2024-04-25T01:33:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75dff5a640 1 -- 172.21.15.27:0/2806783154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d819d420 con 0x7f75d810bd00 2024-04-25T01:33:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75ceffd640 1 -- 172.21.15.27:0/2806783154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d4033070 con 0x7f75d810bd00 2024-04-25T01:33:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.235+0000 7f75dff5a640 1 -- 172.21.15.27:0/2806783154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d81a4d10 con 0x7f75d810bd00 2024-04-25T01:33:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.238+0000 7f75ceffd640 1 -- 172.21.15.27:0/2806783154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f75d4049070 con 0x7f75d810bd00 2024-04-25T01:33:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.238+0000 7f75e2c62740 1 Processor -- start 2024-04-25T01:33:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.238+0000 7f75e2c62740 1 -- start start 2024-04-25T01:33:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.238+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975d1580 0x564f975d1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.238+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975dc570 0x564f975de960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75e2c62740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564f97536670 con 0x564f975d1ea0 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75e2c62740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564f974de970 con 0x564f975dc570 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75e2c62740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564f97532630 con 0x564f975d1580 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33732/0 (socket says 172.21.15.27:33732) 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 -- 172.21.15.27:0/2380647055 learned_addr learned my addr 172.21.15.27:0/2380647055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975d1580 0x564f975d1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ceffd640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f75c40a2d80 0x7f75c40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975dc570 0x564f975de960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75de4f7640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f75c40a2d80 0x7f75c40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 -- 172.21.15.27:0/2380647055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975d1580 msgr2=0x564f975d1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975d1580 0x564f975d1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 -- 172.21.15.27:0/2380647055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975dc570 msgr2=0x564f975de960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975dc570 0x564f975de960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 -- 172.21.15.27:0/2380647055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f97502400 con 0x564f975d1ea0 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ceffd640 1 -- 172.21.15.27:0/2806783154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f75d40f0510 con 0x7f75d810bd00 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dc030 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f75d000aa10 tx=0x7f75d000aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e964aad17a865699 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75c3fff640 1 -- 172.21.15.27:0/2380647055 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0015070 con 0x564f975d1ea0 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75c3fff640 1 -- 172.21.15.27:0/2380647055 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d000cc60 con 0x564f975d1ea0 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75de4f7640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f75c40a2d80 0x7f75c40a5240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f75d40b8aa0 tx=0x7f75d408b570 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.239+0000 7f75c3fff640 1 -- 172.21.15.27:0/2380647055 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0018690 con 0x564f975d1ea0 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 -- 172.21.15.27:0/2380647055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 msgr2=0x564f975dc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dc030 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f75d000aa10 tx=0x7f75d000aed0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 -- 172.21.15.27:0/2380647055 shutdown_connections 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975dc570 0x564f975de960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dc030 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 --2- 172.21.15.27:0/2380647055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975d1580 0x564f975d1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 -- 172.21.15.27:0/2380647055 >> 172.21.15.27:0/2380647055 conn(0x564f975cd8a0 msgr2=0x564f975cdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 -- 172.21.15.27:0/2380647055 shutdown_connections 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.240+0000 7f75e2c62740 1 -- 172.21.15.27:0/2380647055 wait complete. 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 1 -- start start 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 1 Processor -- start 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1580 0x564f975dbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1ea0 0x564f975d9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975d97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564f975df360 con 0x564f975d1580 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564f974f2050 con 0x564f975d1ea0 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564f975df4e0 con 0x564f975dc570 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75e2c62740 10 client.0 _ll_register_callbacks cb 0x564f97249be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1580 0x564f975dbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1580 0x564f975dbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33748/0 (socket says 172.21.15.27:33748) 2024-04-25T01:33:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 -- 172.21.15.27:0/1009056976 learned_addr learned my addr 172.21.15.27:0/1009056976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975d97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1ea0 0x564f975d9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 -- 172.21.15.27:0/1009056976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 msgr2=0x564f975d97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975d97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 -- 172.21.15.27:0/1009056976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1ea0 msgr2=0x564f975d9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1ea0 0x564f975d9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.241+0000 7f75dd4f5640 1 -- 172.21.15.27:0/1009056976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f974c3dc0 con 0x564f975d1580 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1580 0x564f975dbf10 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f75c80189f0 tx=0x7f75c80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1f4b9c5abd4e36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75c17fa640 1 -- 172.21.15.27:0/1009056976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c800f040 con 0x564f975d1580 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75c17fa640 1 -- 172.21.15.27:0/1009056976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f75c8011cb0 con 0x564f975d1580 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 1 -- 172.21.15.27:0/1009056976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75d000cc60 con 0x564f975d1580 2024-04-25T01:33:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75c17fa640 1 -- 172.21.15.27:0/1009056976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f75c8027050 con 0x564f975d1580 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 10 client.17301 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 1 -- 172.21.15.27:0/1009056976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f9724a3f0 con 0x564f975d1580 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75c17fa640 1 -- 172.21.15.27:0/1009056976 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d000cc60 con 0x564f975d1580 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75c17fa640 1 -- 172.21.15.27:0/1009056976 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f75c8015040 con 0x564f975d1580 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 10 client.17301 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 10 client.17301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 20 client.17301 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564f975ec660 0x564f975eeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 4 client.17301 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.242+0000 7f75e2c62740 1 -- 172.21.15.27:0/1009056976 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f97543bb0 con 0x564f975ec660 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.243+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564f975ec660 0x564f975eeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.243+0000 7f75c17fa640 10 client.17301 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.243+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564f975ec660 0x564f975eeae0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a928ad73b686f897 server_cookie=e78483193b676fd7 in_seq=0 out_seq=0 2024-04-25T01:33:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75c17fa640 1 -- 172.21.15.27:0/1009056976 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f97543bb0 con 0x564f975ec660 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75c17fa640 10 client.17301 handle_command_reply: tid=0 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 client.17301 shutdown 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 -- 172.21.15.27:0/1009056976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1580 msgr2=0x564f975dbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1580 0x564f975dbf10 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f75c80189f0 tx=0x7f75c80047d0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 -- 172.21.15.27:0/1009056976 shutdown_connections 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564f975ec660 0x564f975eeae0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975d97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1ea0 0x564f975d9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75c17fa640 0 client.17301 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1009056976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1580 0x564f975dbf10 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 -- 172.21.15.27:0/1009056976 >> 172.21.15.27:0/1009056976 conn(0x564f975cd8a0 msgr2=0x564f975ccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 -- 172.21.15.27:0/1009056976 shutdown_connections 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 1 -- 172.21.15.27:0/1009056976 wait complete. 2024-04-25T01:33:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.244+0000 7f75e2c62740 20 client.17301 trim_cache size 0 max 16384 2024-04-25T01:33:04.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75e2c62740 1 Processor -- start 2024-04-25T01:33:04.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75e2c62740 1 -- start start 2024-04-25T01:33:04.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75e2c62740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81b16d0 con 0x564f975d1ea0 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75e2c62740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81aefa0 con 0x564f975d1580 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75e2c62740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81ace10 con 0x564f975dc570 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33752/0 (socket says 172.21.15.27:33752) 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 -- 172.21.15.27:0/359328061 learned_addr learned my addr 172.21.15.27:0/359328061 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 -- 172.21.15.27:0/359328061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 msgr2=0x564f975f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 -- 172.21.15.27:0/359328061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 msgr2=0x564f975d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 -- 172.21.15.27:0/359328061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f973147d0 con 0x564f975d1ea0 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:04.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.258+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dba40 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f75d000a680 tx=0x7f75d000a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13563d573851d748 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75c37fe640 1 -- 172.21.15.27:0/359328061 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0027020 con 0x564f975d1ea0 2024-04-25T01:33:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75c37fe640 1 -- 172.21.15.27:0/359328061 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d0023dc0 con 0x564f975d1ea0 2024-04-25T01:33:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75c37fe640 1 -- 172.21.15.27:0/359328061 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0015040 con 0x564f975d1ea0 2024-04-25T01:33:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 -- 172.21.15.27:0/359328061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 msgr2=0x564f975dba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dba40 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f75d000a680 tx=0x7f75d000a180 comp rx=0 tx=0).stop 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 -- 172.21.15.27:0/359328061 shutdown_connections 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dba40 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 --2- 172.21.15.27:0/359328061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 -- 172.21.15.27:0/359328061 >> 172.21.15.27:0/359328061 conn(0x564f975cd8a0 msgr2=0x564f975d2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 -- 172.21.15.27:0/359328061 shutdown_connections 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.259+0000 7f75e2c62740 1 -- 172.21.15.27:0/359328061 wait complete. 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 1 -- start start 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 1 Processor -- start 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975e0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81ad460 con 0x564f975d1ea0 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81ae8d0 con 0x564f975d1580 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81adb30 con 0x564f975dc570 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75e2c62740 10 client.0 _ll_register_callbacks cb 0x7f75c8006b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75ddcf6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75ddcf6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33756/0 (socket says 172.21.15.27:33756) 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75ddcf6640 1 -- 172.21.15.27:0/1133001517 learned_addr learned my addr 172.21.15.27:0/1133001517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975e0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75ddcf6640 1 -- 172.21.15.27:0/1133001517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 msgr2=0x564f975e0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975e0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75ddcf6640 1 -- 172.21.15.27:0/1133001517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 msgr2=0x564f975e4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975e4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.260+0000 7f75ddcf6640 1 -- 172.21.15.27:0/1133001517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f975e0c70 con 0x564f975d1ea0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975e0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75ddcf6640 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dff00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006a70 tx=0x7f75d0006be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe8c33e5533db23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75c17fa640 1 -- 172.21.15.27:0/1133001517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0027030 con 0x564f975d1ea0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75c17fa640 1 -- 172.21.15.27:0/1133001517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f75d00232c0 con 0x564f975d1ea0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 1 -- 172.21.15.27:0/1133001517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75d0023dc0 con 0x564f975d1ea0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75c17fa640 1 -- 172.21.15.27:0/1133001517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f75d003c050 con 0x564f975d1ea0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 10 client.17307 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 1 -- 172.21.15.27:0/1133001517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f97652bf0 con 0x564f975d1ea0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75c17fa640 1 -- 172.21.15.27:0/1133001517 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d0023dc0 con 0x564f975d1ea0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75c17fa640 1 -- 172.21.15.27:0/1133001517 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f75d00384e0 con 0x564f975d1ea0 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 10 client.17307 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:04.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 20 client.17307 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564f97653020 0x564f97655410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 4 client.17307 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.261+0000 7f75e2c62740 1 -- 172.21.15.27:0/1133001517 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564f9753df00 con 0x564f97653020 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.262+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564f97653020 0x564f97655410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.262+0000 7f75c17fa640 10 client.17307 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.262+0000 7f75dd4f5640 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564f97653020 0x564f97655410 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c60df822e606bd1d server_cookie=a128bd43eea2f282 in_seq=0 out_seq=0 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.262+0000 7f75c17fa640 1 -- 172.21.15.27:0/1133001517 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564f9753df00 con 0x564f97653020 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.262+0000 7f75c17fa640 10 client.17307 handle_command_reply: tid=0 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.262+0000 7f75e2c62740 1 client.17307 shutdown 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.262+0000 7f75e2c62740 1 -- 172.21.15.27:0/1133001517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 msgr2=0x564f975dff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.262+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dff00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006a70 tx=0x7f75d0006be0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 1 -- 172.21.15.27:0/1133001517 shutdown_connections 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564f97653020 0x564f97655410 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f975dc570 0x564f975e0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564f975d1ea0 0x564f975dff00 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75c17fa640 0 client.17307 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 1 --2- 172.21.15.27:0/1133001517 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564f975d1580 0x564f975e4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 1 -- 172.21.15.27:0/1133001517 >> 172.21.15.27:0/1133001517 conn(0x564f975cd8a0 msgr2=0x564f975e7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 1 -- 172.21.15.27:0/1133001517 shutdown_connections 2024-04-25T01:33:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 1 -- 172.21.15.27:0/1133001517 wait complete. 2024-04-25T01:33:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75e2c62740 20 client.17307 trim_cache size 0 max 16384 2024-04-25T01:33:04.247 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:33:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 -- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f75c40a2d80 msgr2=0x7f75c40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f75c40a2d80 0x7f75c40a5240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f75d40b8aa0 tx=0x7f75d408b570 comp rx=0 tx=0).stop 2024-04-25T01:33:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 -- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 msgr2=0x7f75d819ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 0x7f75d819ca60 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009dc0 tx=0x7f75d4003f10 comp rx=0 tx=0).stop 2024-04-25T01:33:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 -- 172.21.15.27:0/2806783154 shutdown_connections 2024-04-25T01:33:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f75c40a2d80 0x7f75c40a5240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75d810bd00 0x7f75d819ca60 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75d81032e0 0x7f75d81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 --2- 172.21.15.27:0/2806783154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d81029c0 0x7f75d81981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 -- 172.21.15.27:0/2806783154 >> 172.21.15.27:0/2806783154 conn(0x7f75d8077740 msgr2=0x7f75d80fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 -- 172.21.15.27:0/2806783154 shutdown_connections 2024-04-25T01:33:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:04.263+0000 7f75dff5a640 1 -- 172.21.15.27:0/2806783154 wait complete. 2024-04-25T01:33:04.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:04 smithi027 ceph-mon[29197]: pgmap v234: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:33:04.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3278507455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:04.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:04 smithi153 ceph-mon[32621]: pgmap v234: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:33:04.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:04 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3278507455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:04.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:04 smithi139 ceph-mon[32760]: pgmap v234: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:33:04.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:04 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3278507455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:05.252 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:33:05.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:33:05.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:33:05.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.392+0000 7f3c30ab3640 1 -- 172.21.15.27:0/2446309672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c100620 msgr2=0x7f3c2c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.392+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/2446309672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c100620 0x7f3c2c100aa0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009930 tx=0x7f3c1402a420 comp rx=0 tx=0).stop 2024-04-25T01:33:05.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.392+0000 7f3c30ab3640 1 -- 172.21.15.27:0/2446309672 shutdown_connections 2024-04-25T01:33:05.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.392+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/2446309672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c2c100fe0 0x7f3c2c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.392+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/2446309672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c100620 0x7f3c2c100aa0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.392+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/2446309672 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c2c106630 0x7f3c2c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.392+0000 7f3c30ab3640 1 -- 172.21.15.27:0/2446309672 >> 172.21.15.27:0/2446309672 conn(0x7f3c2c0776a0 msgr2=0x7f3c2c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.392+0000 7f3c30ab3640 1 -- 172.21.15.27:0/2446309672 shutdown_connections 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.393+0000 7f3c30ab3640 1 -- 172.21.15.27:0/2446309672 wait complete. 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.393+0000 7f3c30ab3640 1 Processor -- start 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.393+0000 7f3c30ab3640 1 -- start start 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.393+0000 7f3c30ab3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c2c100620 0x7f3c2c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.393+0000 7f3c30ab3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c2c100fe0 0x7f3c2c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.393+0000 7f3c30ab3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c106630 0x7f3c2c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.393+0000 7f3c30ab3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c110a40 con 0x7f3c2c106630 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c30ab3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c1108c0 con 0x7f3c2c100620 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c30ab3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c110bc0 con 0x7f3c2c100fe0 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c106630 0x7f3c2c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c106630 0x7f3c2c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33770/0 (socket says 172.21.15.27:33770) 2024-04-25T01:33:05.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 -- 172.21.15.27:0/1216133351 learned_addr learned my addr 172.21.15.27:0/1216133351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c23fff640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c2c100fe0 0x7f3c2c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c2c100620 0x7f3c2c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 -- 172.21.15.27:0/1216133351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c2c100fe0 msgr2=0x7f3c2c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c2c100fe0 0x7f3c2c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 -- 172.21.15.27:0/1216133351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c2c100620 msgr2=0x7f3c2c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c2c100620 0x7f3c2c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 -- 172.21.15.27:0/1216133351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c2c19d010 con 0x7f3c2c106630 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c106630 0x7f3c2c19c8e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400eaa0 tx=0x7f3c2400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c237fe640 1 -- 172.21.15.27:0/1216133351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400cdb0 con 0x7f3c2c106630 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c237fe640 1 -- 172.21.15.27:0/1216133351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c24005020 con 0x7f3c2c106630 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c30ab3640 1 -- 172.21.15.27:0/1216133351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c2c19d300 con 0x7f3c2c106630 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c237fe640 1 -- 172.21.15.27:0/1216133351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c240116b0 con 0x7f3c2c106630 2024-04-25T01:33:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.394+0000 7f3c30ab3640 1 -- 172.21.15.27:0/1216133351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c2c1a4b90 con 0x7f3c2c106630 2024-04-25T01:33:05.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.396+0000 7f3c237fe640 1 -- 172.21.15.27:0/1216133351 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3c24005560 con 0x7f3c2c106630 2024-04-25T01:33:05.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c337c1740 1 Processor -- start 2024-04-25T01:33:05.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c337c1740 1 -- start start 2024-04-25T01:33:05.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c237fe640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c180a2d00 0x7f3c180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 0x5616101becc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 0x5616101bf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101ca010 0x5616101cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c337c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5616101232c0 con 0x5616101ca010 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c337c1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5616100cb910 con 0x5616101bf290 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c337c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56161011f280 con 0x5616101c0980 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c2ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101ca010 0x5616101cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c23fff640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c180a2d00 0x7f3c180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c2ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101ca010 0x5616101cc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33776/0 (socket says 172.21.15.27:33776) 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.397+0000 7f3c2ad9d640 1 -- 172.21.15.27:0/2698707035 learned_addr learned my addr 172.21.15.27:0/2698707035 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/2698707035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 0x5616101becc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c237fe640 1 -- 172.21.15.27:0/1216133351 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f3c240cfb40 con 0x7f3c2c106630 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c2ad9d640 1 -- 172.21.15.27:0/2698707035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 msgr2=0x5616101becc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/2698707035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 0x5616101becc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c2ad9d640 1 -- 172.21.15.27:0/2698707035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 msgr2=0x5616101bf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/2698707035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 0x5616101bf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c2ad9d640 1 -- 172.21.15.27:0/2698707035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616100dcfb0 con 0x5616101ca010 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c23fff640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c180a2d00 0x7f3c180a51c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3c14000c00 tx=0x7f3c140314d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/2698707035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101ca010 0x5616101cc400 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f3c240ccf00 tx=0x7f3c240058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df73daeee0a2b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c20ff9640 1 -- 172.21.15.27:0/2698707035 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24092070 con 0x5616101ca010 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c20ff9640 1 -- 172.21.15.27:0/2698707035 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c2409bc70 con 0x5616101ca010 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.398+0000 7f3c20ff9640 1 -- 172.21.15.27:0/2698707035 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2401f070 con 0x5616101ca010 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 -- 172.21.15.27:0/2698707035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101ca010 msgr2=0x5616101cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 --2- 172.21.15.27:0/2698707035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101ca010 0x5616101cc400 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f3c240ccf00 tx=0x7f3c240058d0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 -- 172.21.15.27:0/2698707035 shutdown_connections 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 --2- 172.21.15.27:0/2698707035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101ca010 0x5616101cc400 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 --2- 172.21.15.27:0/2698707035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 0x5616101bf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 --2- 172.21.15.27:0/2698707035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 0x5616101becc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 -- 172.21.15.27:0/2698707035 >> 172.21.15.27:0/2698707035 conn(0x561610187950 msgr2=0x5616101bb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 -- 172.21.15.27:0/2698707035 shutdown_connections 2024-04-25T01:33:05.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 -- 172.21.15.27:0/2698707035 wait complete. 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 -- start start 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.399+0000 7f3c337c1740 1 Processor -- start 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101bf290 0x5616101c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 0x5616101c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 0x5616101c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c337c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5616101ccde0 con 0x5616101bf290 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c337c1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5616100dee90 con 0x5616101ca010 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c337c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5616101ccf60 con 0x5616101c0980 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c337c1740 10 client.0 _ll_register_callbacks cb 0x56161011a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101bf290 0x5616101c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101bf290 0x5616101c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33782/0 (socket says 172.21.15.27:33782) 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 -- 172.21.15.27:0/3500887350 learned_addr learned my addr 172.21.15.27:0/3500887350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 0x5616101c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c23fff640 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 0x5616101c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 -- 172.21.15.27:0/3500887350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 msgr2=0x5616101c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 0x5616101c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 -- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 msgr2=0x5616101c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 0x5616101c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 -- 172.21.15.27:0/3500887350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56160feff7e0 con 0x5616101bf290 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 0x5616101c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101bf290 0x5616101c6ef0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009df0 tx=0x7f3c1c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0ebda2314dba67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.400+0000 7f3c11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3500887350 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c012860 con 0x5616101bf290 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3500887350 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f3c1c012a00 con 0x5616101bf290 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 1 -- 172.21.15.27:0/3500887350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c2409bc70 con 0x5616101bf290 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3500887350 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c1c025050 con 0x5616101bf290 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 10 client.17319 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 1 -- 172.21.15.27:0/3500887350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56160ff5ff70 con 0x5616101bf290 2024-04-25T01:33:05.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3500887350 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c2409bc70 con 0x5616101bf290 2024-04-25T01:33:05.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3500887350 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3c1c00f040 con 0x5616101bf290 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 10 client.17319 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 10 client.17319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 20 client.17319 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5616101da0f0 0x5616101dc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 4 client.17319 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.401+0000 7f3c337c1740 1 -- 172.21.15.27:0/3500887350 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616100f75a0 con 0x5616101da0f0 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.402+0000 7f3c23fff640 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5616101da0f0 0x5616101dc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.402+0000 7f3c11ffb640 10 client.17319 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.402+0000 7f3c23fff640 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5616101da0f0 0x5616101dc590 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5c5d3e786a2232f server_cookie=c489f7fbe9cabb49 in_seq=0 out_seq=0 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3500887350 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616100f75a0 con 0x5616101da0f0 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c11ffb640 10 client.17319 handle_command_reply: tid=0 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 client.17319 shutdown 2024-04-25T01:33:05.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 -- 172.21.15.27:0/3500887350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101bf290 msgr2=0x5616101c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101bf290 0x5616101c6ef0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009df0 tx=0x7f3c1c0117b0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 -- 172.21.15.27:0/3500887350 shutdown_connections 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5616101da0f0 0x5616101dc590 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 0x5616101c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101c0980 0x5616101c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3500887350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101bf290 0x5616101c6ef0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c11ffb640 0 client.17319 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 -- 172.21.15.27:0/3500887350 >> 172.21.15.27:0/3500887350 conn(0x561610187950 msgr2=0x5616101c1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 -- 172.21.15.27:0/3500887350 shutdown_connections 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.403+0000 7f3c337c1740 1 -- 172.21.15.27:0/3500887350 wait complete. 2024-04-25T01:33:05.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.404+0000 7f3c337c1740 20 client.17319 trim_cache size 0 max 16384 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.421+0000 7f3c337c1740 1 Processor -- start 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.421+0000 7f3c337c1740 1 -- start start 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101bf290 0x5616101d5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 0x5616101e1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c337c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c1b16b0 con 0x5616101c0980 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c337c1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c1aef80 con 0x5616101ca010 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c337c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c1acdf0 con 0x5616101bf290 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33784/0 (socket says 172.21.15.27:33784) 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 -- 172.21.15.27:0/2239492100 learned_addr learned my addr 172.21.15.27:0/2239492100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101bf290 0x5616101d5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 -- 172.21.15.27:0/2239492100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101bf290 msgr2=0x5616101d5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101bf290 0x5616101d5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 -- 172.21.15.27:0/2239492100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 msgr2=0x5616101e1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 0x5616101e1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 -- 172.21.15.27:0/2239492100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56160fdf7170 con 0x5616101c0980 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101bf290 0x5616101d5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c23fff640 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101c66a0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f3c14033000 tx=0x7f3c1403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5eaec530cf5e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c13fff640 1 -- 172.21.15.27:0/2239492100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14035050 con 0x5616101c0980 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c13fff640 1 -- 172.21.15.27:0/2239492100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c14056070 con 0x5616101c0980 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.422+0000 7f3c13fff640 1 -- 172.21.15.27:0/2239492100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c140505a0 con 0x5616101c0980 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 -- 172.21.15.27:0/2239492100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 msgr2=0x5616101c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101c66a0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f3c14033000 tx=0x7f3c1403e000 comp rx=0 tx=0).stop 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 -- 172.21.15.27:0/2239492100 shutdown_connections 2024-04-25T01:33:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101ca010 0x5616101e1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101c66a0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 --2- 172.21.15.27:0/2239492100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101bf290 0x5616101d5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 -- 172.21.15.27:0/2239492100 >> 172.21.15.27:0/2239492100 conn(0x561610187950 msgr2=0x5616101c1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 -- 172.21.15.27:0/2239492100 shutdown_connections 2024-04-25T01:33:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 -- 172.21.15.27:0/2239492100 wait complete. 2024-04-25T01:33:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 -- start start 2024-04-25T01:33:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.423+0000 7f3c337c1740 1 Processor -- start 2024-04-25T01:33:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 0x5616101ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101cea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c337c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101ca010 0x561610239710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c337c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c1ad440 con 0x5616101c0980 2024-04-25T01:33:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c337c1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c1ae8b0 con 0x5616101bf290 2024-04-25T01:33:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c337c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c1adb10 con 0x5616101ca010 2024-04-25T01:33:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c337c1740 10 client.0 _ll_register_callbacks cb 0x7f3c1c009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101cea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33794/0 (socket says 172.21.15.27:33794) 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 -- 172.21.15.27:0/3664611493 learned_addr learned my addr 172.21.15.27:0/3664611493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 0x5616101ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101ca010 0x561610239710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 -- 172.21.15.27:0/3664611493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101ca010 msgr2=0x561610239710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101ca010 0x561610239710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 -- 172.21.15.27:0/3664611493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 msgr2=0x5616101ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 0x5616101ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 -- 172.21.15.27:0/3664611493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561610239d80 con 0x5616101c0980 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c2ad9d640 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101ca010 0x561610239710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c23fff640 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101cea30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f3c14032fd0 tx=0x7f3c14031810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f2492e3d7f7053 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.424+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3664611493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14005460 con 0x5616101c0980 2024-04-25T01:33:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3664611493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f3c14050220 con 0x5616101c0980 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 1 -- 172.21.15.27:0/3664611493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c14056070 con 0x5616101c0980 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3664611493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c14005600 con 0x5616101c0980 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 10 client.17325 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 1 -- 172.21.15.27:0/3664611493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561610241160 con 0x5616101c0980 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3664611493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c14056070 con 0x5616101c0980 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3664611493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3c1403d450 con 0x5616101c0980 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 10 client.17325 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 10 client.17325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 20 client.17325 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5616102418e0 0x561610243d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 4 client.17325 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c337c1740 1 -- 172.21.15.27:0/3664611493 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561610130800 con 0x5616102418e0 2024-04-25T01:33:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.425+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5616102418e0 0x561610243d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.426+0000 7f3c2a59c640 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5616102418e0 0x561610243d00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a234e140728f9012 server_cookie=31266e165f720bb8 in_seq=0 out_seq=0 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.426+0000 7f3c11ffb640 10 client.17325 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c11ffb640 1 -- 172.21.15.27:0/3664611493 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561610130800 con 0x5616102418e0 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c11ffb640 10 client.17325 handle_command_reply: tid=0 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 client.17325 shutdown 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 -- 172.21.15.27:0/3664611493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 msgr2=0x5616101cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101cea30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f3c14032fd0 tx=0x7f3c14031810 comp rx=0 tx=0).stop 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 -- 172.21.15.27:0/3664611493 shutdown_connections 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5616102418e0 0x561610243d00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5616101ca010 0x561610239710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616101c0980 0x5616101cea30 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 --2- 172.21.15.27:0/3664611493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5616101bf290 0x5616101ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c11ffb640 0 client.17325 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 -- 172.21.15.27:0/3664611493 >> 172.21.15.27:0/3664611493 conn(0x561610187950 msgr2=0x5616101c76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 -- 172.21.15.27:0/3664611493 shutdown_connections 2024-04-25T01:33:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 1 -- 172.21.15.27:0/3664611493 wait complete. 2024-04-25T01:33:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.427+0000 7f3c337c1740 20 client.17325 trim_cache size 0 max 16384 2024-04-25T01:33:05.411 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:33:05.411 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:33:05.411 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "ae762e8f-6f1d-4061-9b7d-d8578b9f6256", 2024-04-25T01:33:05.411 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:33:05.411 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 -- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c180a2d00 msgr2=0x7f3c180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c180a2d00 0x7f3c180a51c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3c14000c00 tx=0x7f3c140314d0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 -- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c106630 msgr2=0x7f3c2c19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c106630 0x7f3c2c19c8e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400eaa0 tx=0x7f3c2400ef60 comp rx=0 tx=0).stop 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 -- 172.21.15.27:0/1216133351 shutdown_connections 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f3c180a2d00 0x7f3c180a51c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c2c106630 0x7f3c2c19c8e0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c2c100fe0 0x7f3c2c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 --2- 172.21.15.27:0/1216133351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c2c100620 0x7f3c2c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 -- 172.21.15.27:0/1216133351 >> 172.21.15.27:0/1216133351 conn(0x7f3c2c0776a0 msgr2=0x7f3c2c06a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 -- 172.21.15.27:0/1216133351 shutdown_connections 2024-04-25T01:33:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.428+0000 7f3c30ab3640 1 -- 172.21.15.27:0/1216133351 wait complete. 2024-04-25T01:33:05.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:33:05.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:33:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 -- 172.21.15.27:0/3016677754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858107730 msgr2=0x7f7858107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 --2- 172.21.15.27:0/3016677754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858107730 0x7f7858107b10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7848009930 tx=0x7f784802a420 comp rx=0 tx=0).stop 2024-04-25T01:33:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 -- 172.21.15.27:0/3016677754 shutdown_connections 2024-04-25T01:33:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 --2- 172.21.15.27:0/3016677754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f785806a780 0x7f785810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 --2- 172.21.15.27:0/3016677754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7858069dc0 0x7f785806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 --2- 172.21.15.27:0/3016677754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858107730 0x7f7858107b10 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 -- 172.21.15.27:0/3016677754 >> 172.21.15.27:0/3016677754 conn(0x7f78580776a0 msgr2=0x7f7858077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 -- 172.21.15.27:0/3016677754 shutdown_connections 2024-04-25T01:33:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.543+0000 7f785e646640 1 -- 172.21.15.27:0/3016677754 wait complete. 2024-04-25T01:33:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785e646640 1 Processor -- start 2024-04-25T01:33:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785e646640 1 -- start start 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785e646640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858069dc0 0x7f785819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785e646640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f785806a780 0x7f785819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785e646640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7858107730 0x7f785819ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785e646640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7858102350 con 0x7f7858069dc0 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785e646640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78581021d0 con 0x7f7858107730 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785e646640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78581024d0 con 0x7f785806a780 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f7857fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858069dc0 0x7f785819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f7857fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858069dc0 0x7f785819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33820/0 (socket says 172.21.15.27:33820) 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f7857fff640 1 -- 172.21.15.27:0/3632006185 learned_addr learned my addr 172.21.15.27:0/3632006185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f785cbe3640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7858107730 0x7f785819ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f78577fe640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f785806a780 0x7f785819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f7857fff640 1 -- 172.21.15.27:0/3632006185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f785806a780 msgr2=0x7f785819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f7857fff640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f785806a780 0x7f785819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f7857fff640 1 -- 172.21.15.27:0/3632006185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7858107730 msgr2=0x7f785819ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f7857fff640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7858107730 0x7f785819ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f7857fff640 1 -- 172.21.15.27:0/3632006185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785819f1b0 con 0x7f7858069dc0 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.544+0000 7f78577fe640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f785806a780 0x7f785819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.545+0000 7f7857fff640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858069dc0 0x7f785819a1c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7848009230 tx=0x7f7848004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.545+0000 7f78557fa640 1 -- 172.21.15.27:0/3632006185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784803f030 con 0x7f7858069dc0 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.545+0000 7f78557fa640 1 -- 172.21.15.27:0/3632006185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848003ad0 con 0x7f7858069dc0 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.545+0000 7f785e646640 1 -- 172.21.15.27:0/3632006185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785819f440 con 0x7f7858069dc0 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.545+0000 7f78557fa640 1 -- 172.21.15.27:0/3632006185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848035070 con 0x7f7858069dc0 2024-04-25T01:33:05.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.545+0000 7f785e646640 1 -- 172.21.15.27:0/3632006185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78581a6d30 con 0x7f7858069dc0 2024-04-25T01:33:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7861432740 1 Processor -- start 2024-04-25T01:33:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7861432740 1 -- start start 2024-04-25T01:33:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f78557fa640 1 -- 172.21.15.27:0/3632006185 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7848042030 con 0x7f7858069dc0 2024-04-25T01:33:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7861432740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 0x55a57f0f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7861432740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0fd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7861432740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0fd620 0x55a57f0ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7861432740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a57f0576b0 con 0x55a57f0fd620 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7861432740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a57efff880 con 0x55a57f0f2f50 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7861432740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a57f053670 con 0x55a57f0f2630 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f78577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0fd620 0x55a57f0ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f78577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0fd620 0x55a57f0ffa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33828/0 (socket says 172.21.15.27:33828) 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f78577fe640 1 -- 172.21.15.27:0/3974064909 learned_addr learned my addr 172.21.15.27:0/3974064909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f785cbe3640 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 0x55a57f0f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.548+0000 7f7857fff640 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0fd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f78577fe640 1 -- 172.21.15.27:0/3974064909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 msgr2=0x55a57f0f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f78577fe640 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 0x55a57f0f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f78577fe640 1 -- 172.21.15.27:0/3974064909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 msgr2=0x55a57f0fd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f78577fe640 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0fd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f78577fe640 1 -- 172.21.15.27:0/3974064909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a57ee357d0 con 0x55a57f0fd620 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f7857fff640 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f78557fa640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78440a2d00 0x7f78440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f78577fe640 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0fd620 0x55a57f0ffa10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7840005fd0 tx=0x7f784000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cf88b956f4d988 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f785cbe3640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78440a2d00 0x7f78440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f784e7fc640 1 -- 172.21.15.27:0/3974064909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7840016070 con 0x55a57f0fd620 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f78557fa640 1 -- 172.21.15.27:0/3632006185 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f78480efdf0 con 0x7f7858069dc0 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f784e7fc640 1 -- 172.21.15.27:0/3974064909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7840005500 con 0x55a57f0fd620 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.549+0000 7f784e7fc640 1 -- 172.21.15.27:0/3974064909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7840011470 con 0x55a57f0fd620 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 -- 172.21.15.27:0/3974064909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0fd620 msgr2=0x55a57f0ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f785cbe3640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78440a2d00 0x7f78440a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f78500092f0 tx=0x7f7850009280 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0fd620 0x55a57f0ffa10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7840005fd0 tx=0x7f784000e6c0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 -- 172.21.15.27:0/3974064909 shutdown_connections 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0fd620 0x55a57f0ffa10 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 --2- 172.21.15.27:0/3974064909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 0x55a57f0f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 -- 172.21.15.27:0/3974064909 >> 172.21.15.27:0/3974064909 conn(0x55a57f0ee950 msgr2=0x55a57f0eed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 -- 172.21.15.27:0/3974064909 shutdown_connections 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 -- 172.21.15.27:0/3974064909 wait complete. 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 -- start start 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.550+0000 7f7861432740 1 Processor -- start 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7861432740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 0x55a57f0fa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7861432740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0faa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7861432740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0fd620 0x55a57f0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7861432740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a57f100410 con 0x55a57f0f2f50 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7861432740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a57f012f50 con 0x55a57f0fd620 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7861432740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a57f100590 con 0x55a57f0f2630 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7861432740 10 client.0 _ll_register_callbacks cb 0x55a57f04e000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0faa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33834/0 (socket says 172.21.15.27:33834) 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 -- 172.21.15.27:0/1550707470 learned_addr learned my addr 172.21.15.27:0/1550707470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7857fff640 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 0x55a57f0fa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f785cbe3640 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0fd620 0x55a57f0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 -- 172.21.15.27:0/1550707470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 msgr2=0x55a57f0fa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 0x55a57f0fa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 -- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0fd620 msgr2=0x55a57f0fafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0fd620 0x55a57f0fafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 -- 172.21.15.27:0/1550707470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a57ed6b3f0 con 0x55a57f0f2f50 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f785cbe3640 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0fd620 0x55a57f0fafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78577fe640 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0faa60 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7840000c00 tx=0x7f78400099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801f242ef8868fe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78377fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78377fe640 1 -- 172.21.15.27:0/1550707470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7840028060 con 0x55a57f0f2f50 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f78377fe640 1 -- 172.21.15.27:0/1550707470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f7840004f40 con 0x55a57f0f2f50 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.551+0000 7f7861432740 1 -- 172.21.15.27:0/1550707470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7840005500 con 0x55a57f0f2f50 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f78377fe640 1 -- 172.21.15.27:0/1550707470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f784002f050 con 0x55a57f0f2f50 2024-04-25T01:33:05.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7861432740 10 client.17337 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7861432740 1 -- 172.21.15.27:0/1550707470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a57ed2b0f0 con 0x55a57f0f2f50 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f78377fe640 1 -- 172.21.15.27:0/1550707470 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7840005500 con 0x55a57f0f2f50 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f78377fe640 1 -- 172.21.15.27:0/1550707470 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f784000f040 con 0x55a57f0f2f50 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7861432740 10 client.17337 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7861432740 10 client.17337 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7861432740 20 client.17337 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7861432740 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a57f10d6d0 0x55a57f10fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7861432740 4 client.17337 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7861432740 1 -- 172.21.15.27:0/1550707470 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a57f064bf0 con 0x55a57f10d6d0 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.552+0000 7f7857fff640 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a57f10d6d0 0x55a57f10fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.553+0000 7f7857fff640 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a57f10d6d0 0x55a57f10fac0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c15a6690a6a0ee17 server_cookie=237af5ccad535a4e in_seq=0 out_seq=0 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.553+0000 7f78377fe640 10 client.17337 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.553+0000 7f78377fe640 1 -- 172.21.15.27:0/1550707470 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a57f064bf0 con 0x55a57f10d6d0 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.553+0000 7f78377fe640 10 client.17337 handle_command_reply: tid=0 2024-04-25T01:33:05.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 client.17337 shutdown 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 -- 172.21.15.27:0/1550707470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 msgr2=0x55a57f0faa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0faa60 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7840000c00 tx=0x7f78400099b0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 -- 172.21.15.27:0/1550707470 shutdown_connections 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a57f10d6d0 0x55a57f10fac0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0fd620 0x55a57f0fafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0faa60 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f78377fe640 0 client.17337 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 --2- 172.21.15.27:0/1550707470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0f2630 0x55a57f0fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 -- 172.21.15.27:0/1550707470 >> 172.21.15.27:0/1550707470 conn(0x55a57f0ee950 msgr2=0x55a57f0f4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 -- 172.21.15.27:0/1550707470 shutdown_connections 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 1 -- 172.21.15.27:0/1550707470 wait complete. 2024-04-25T01:33:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.554+0000 7f7861432740 20 client.17337 trim_cache size 0 max 16384 2024-04-25T01:33:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.571+0000 7f7861432740 1 Processor -- start 2024-04-25T01:33:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.571+0000 7f7861432740 1 -- start start 2024-04-25T01:33:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.571+0000 7f7861432740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2630 0x55a57f0fcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.571+0000 7f7861432740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0f96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.571+0000 7f7861432740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 0x55a57f0f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.571+0000 7f7861432740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b37f0 con 0x55a57f0f2630 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.571+0000 7f7861432740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b10c0 con 0x55a57f0f2f50 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.571+0000 7f7861432740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78581aef30 con 0x55a57f0fd620 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2630 0x55a57f0fcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2630 0x55a57f0fcbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33842/0 (socket says 172.21.15.27:33842) 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 -- 172.21.15.27:0/2647372443 learned_addr learned my addr 172.21.15.27:0/2647372443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f785cbe3640 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 0x55a57f0f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 -- 172.21.15.27:0/2647372443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 msgr2=0x55a57f0f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 0x55a57f0f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 -- 172.21.15.27:0/2647372443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 msgr2=0x55a57f0f96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f78577fe640 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0f96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0f96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 -- 172.21.15.27:0/2647372443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a57f0e0090 con 0x55a57f0f2630 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f7857fff640 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2630 0x55a57f0fcbe0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f78480ed750 tx=0x7f78480dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e761393da59de3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f784dffb640 1 -- 172.21.15.27:0/2647372443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848049040 con 0x55a57f0f2630 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f784dffb640 1 -- 172.21.15.27:0/2647372443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78480c4b90 con 0x55a57f0f2630 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.572+0000 7f784dffb640 1 -- 172.21.15.27:0/2647372443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78480ed560 con 0x55a57f0f2630 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 -- 172.21.15.27:0/2647372443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2630 msgr2=0x55a57f0fcbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2630 0x55a57f0fcbe0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f78480ed750 tx=0x7f78480dc040 comp rx=0 tx=0).stop 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 -- 172.21.15.27:0/2647372443 shutdown_connections 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 0x55a57f0f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2f50 0x55a57f0f96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 --2- 172.21.15.27:0/2647372443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2630 0x55a57f0fcbe0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 -- 172.21.15.27:0/2647372443 >> 172.21.15.27:0/2647372443 conn(0x55a57f0ee950 msgr2=0x55a57f0ed260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 -- 172.21.15.27:0/2647372443 shutdown_connections 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 -- 172.21.15.27:0/2647372443 wait complete. 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 -- start start 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.573+0000 7f7861432740 1 Processor -- start 2024-04-25T01:33:05.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7861432740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2630 0x55a57f105ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7861432740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7861432740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 0x55a57f0f7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7861432740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78581af580 con 0x55a57f0f2f50 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7861432740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b09f0 con 0x55a57f0f2630 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7861432740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78581afc50 con 0x55a57f0fd620 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7861432740 10 client.0 _ll_register_callbacks cb 0x7f7840009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0f6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33852/0 (socket says 172.21.15.27:33852) 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 -- 172.21.15.27:0/2592472359 learned_addr learned my addr 172.21.15.27:0/2592472359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7857fff640 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2630 0x55a57f105ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f785cbe3640 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 0x55a57f0f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 -- 172.21.15.27:0/2592472359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 msgr2=0x55a57f0f7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 0x55a57f0f7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 -- 172.21.15.27:0/2592472359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2630 msgr2=0x55a57f105ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2630 0x55a57f105ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 -- 172.21.15.27:0/2592472359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a57f0f7a50 con 0x55a57f0f2f50 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78577fe640 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0f6e50 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7840011dd0 tx=0x7f7840005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60876d65347a73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78377fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78377fe640 1 -- 172.21.15.27:0/2592472359 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7840028050 con 0x55a57f0f2f50 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f7861432740 1 -- 172.21.15.27:0/2592472359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78480c4b90 con 0x55a57f0f2f50 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.574+0000 7f78377fe640 1 -- 172.21.15.27:0/2592472359 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f7840004890 con 0x55a57f0f2f50 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f78377fe640 1 -- 172.21.15.27:0/2592472359 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7840016040 con 0x55a57f0f2f50 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7861432740 10 client.17343 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7861432740 1 -- 172.21.15.27:0/2592472359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a57f173880 con 0x55a57f0f2f50 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f78377fe640 1 -- 172.21.15.27:0/2592472359 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78480c4b90 con 0x55a57f0f2f50 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f78377fe640 1 -- 172.21.15.27:0/2592472359 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7840005290 con 0x55a57f0f2f50 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7861432740 10 client.17343 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:05.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7861432740 10 client.17343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7861432740 20 client.17343 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7861432740 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a57f174000 0x55a57f176420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7861432740 4 client.17343 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7861432740 1 -- 172.21.15.27:0/2592472359 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a57f05ef40 con 0x55a57f174000 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.575+0000 7f7857fff640 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a57f174000 0x55a57f176420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7857fff640 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a57f174000 0x55a57f176420 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87b3d1acbe48381a server_cookie=51160f3e3a2e7bb1 in_seq=0 out_seq=0 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f78377fe640 10 client.17343 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f78377fe640 1 -- 172.21.15.27:0/2592472359 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a57f05ef40 con 0x55a57f174000 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f78377fe640 10 client.17343 handle_command_reply: tid=0 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 client.17343 shutdown 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 -- 172.21.15.27:0/2592472359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 msgr2=0x55a57f0f6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0f6e50 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7840011dd0 tx=0x7f7840005900 comp rx=0 tx=0).stop 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 -- 172.21.15.27:0/2592472359 shutdown_connections 2024-04-25T01:33:05.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a57f174000 0x55a57f176420 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a57f0fd620 0x55a57f0f7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a57f0f2f50 0x55a57f0f6e50 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 --2- 172.21.15.27:0/2592472359 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a57f0f2630 0x55a57f105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f78377fe640 0 client.17343 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 -- 172.21.15.27:0/2592472359 >> 172.21.15.27:0/2592472359 conn(0x55a57f0ee950 msgr2=0x55a57f0edaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 -- 172.21.15.27:0/2592472359 shutdown_connections 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 1 -- 172.21.15.27:0/2592472359 wait complete. 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.576+0000 7f7861432740 20 client.17343 trim_cache size 0 max 16384 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:33:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:33:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:33:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "ae762e8f-6f1d-4061-9b7d-d8578b9f6256": { 2024-04-25T01:33:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:33:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "ae762e8f-6f1d-4061-9b7d-d8578b9f6256", 2024-04-25T01:33:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:33:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:33:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:33:05.561 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 -- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78440a2d00 msgr2=0x7f78440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78440a2d00 0x7f78440a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f78500092f0 tx=0x7f7850009280 comp rx=0 tx=0).stop 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 -- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858069dc0 msgr2=0x7f785819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858069dc0 0x7f785819a1c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7848009230 tx=0x7f7848004270 comp rx=0 tx=0).stop 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 -- 172.21.15.27:0/3632006185 shutdown_connections 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7858107730 0x7f785819ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f785806a780 0x7f785819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78440a2d00 0x7f78440a51c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 --2- 172.21.15.27:0/3632006185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7858069dc0 0x7f785819a1c0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 -- 172.21.15.27:0/3632006185 >> 172.21.15.27:0/3632006185 conn(0x7f78580776a0 msgr2=0x7f78580fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 -- 172.21.15.27:0/3632006185 shutdown_connections 2024-04-25T01:33:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:05.577+0000 7f785e646640 1 -- 172.21.15.27:0/3632006185 wait complete. 2024-04-25T01:33:05.565 INFO:tasks.cephfs.filesystem:scrub status for tag:ae762e8f-6f1d-4061-9b7d-d8578b9f6256 - {'path': '/', 'tag': 'ae762e8f-6f1d-4061-9b7d-d8578b9f6256', 'options': 'recursive,force'} 2024-04-25T01:33:06.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:06 smithi153 ceph-mon[32621]: pgmap v235: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:33:06.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:06 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:33:06.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:06 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:06.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:06 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:33:06.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:06 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:33:06.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:06 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:06.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:06 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:06.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:06 smithi139 ceph-mon[32760]: pgmap v235: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:33:06.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:06 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:33:06.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:06 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:06.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:06 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:33:06.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:06 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:33:06.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:06 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:06.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:06 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:06.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:06 smithi027 ceph-mon[29197]: pgmap v235: 129 pgs: 129 active+clean; 3.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:33:06.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:06 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:33:06.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:06 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:06.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:06 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:33:06.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:06 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:33:06.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:06 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:06.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:06 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:08.523 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:08 smithi153 ceph-mon[32621]: pgmap v236: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 61 KiB/s wr, 32 op/s 2024-04-25T01:33:08.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:08 smithi139 ceph-mon[32760]: pgmap v236: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 61 KiB/s wr, 32 op/s 2024-04-25T01:33:08.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:08 smithi027 ceph-mon[29197]: pgmap v236: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 61 KiB/s wr, 32 op/s 2024-04-25T01:33:09.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:09 smithi153 ceph-mon[32621]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T01:33:09.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:09 smithi139 ceph-mon[32760]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T01:33:09.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:09 smithi027 ceph-mon[29197]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T01:33:10.522 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:10 smithi153 ceph-mon[32621]: pgmap v238: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:33:10.539 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:10 smithi139 ceph-mon[32760]: pgmap v238: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:33:10.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:10 smithi027 ceph-mon[29197]: pgmap v238: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-04-25T01:33:12.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:12 smithi027 ceph-mon[29197]: pgmap v239: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:12.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:12 smithi153 ceph-mon[32621]: pgmap v239: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:12.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:12 smithi139 ceph-mon[32760]: pgmap v239: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:13.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:13 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:33:13.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:13 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:33:13.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:13 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:33:13.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:13 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:33:13.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:13 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:33:13.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:13 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:33:14.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:14 smithi027 ceph-mon[29197]: pgmap v240: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:14.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:14 smithi153 ceph-mon[32621]: pgmap v240: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:14.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:14 smithi139 ceph-mon[32760]: pgmap v240: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:16 smithi027 ceph-mon[29197]: pgmap v241: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:16 smithi027 ceph-mon[29197]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T01:33:16.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:16 smithi153 ceph-mon[32621]: pgmap v241: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:16.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:16 smithi153 ceph-mon[32621]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T01:33:16.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:16 smithi139 ceph-mon[32760]: pgmap v241: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:33:16.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:16 smithi139 ceph-mon[32760]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T01:33:18.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:18 smithi027 ceph-mon[29197]: pgmap v243: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-04-25T01:33:18.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:18 smithi153 ceph-mon[32621]: pgmap v243: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-04-25T01:33:18.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:18 smithi139 ceph-mon[32760]: pgmap v243: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-04-25T01:33:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:20 smithi027 ceph-mon[29197]: pgmap v244: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:20 smithi153 ceph-mon[32621]: pgmap v244: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:20.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:20 smithi139 ceph-mon[32760]: pgmap v244: 129 pgs: 129 active+clean; 3.8 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:22.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:22 smithi027 ceph-mon[29197]: pgmap v245: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:22.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:22 smithi153 ceph-mon[32621]: pgmap v245: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:22.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:22 smithi139 ceph-mon[32760]: pgmap v245: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:24 smithi027 ceph-mon[29197]: pgmap v246: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:24.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:24 smithi153 ceph-mon[32621]: pgmap v246: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:24.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:24 smithi139 ceph-mon[32760]: pgmap v246: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:26.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:26 smithi027 ceph-mon[29197]: pgmap v247: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:26.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:26 smithi153 ceph-mon[32621]: pgmap v247: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:26.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:26 smithi139 ceph-mon[32760]: pgmap v247: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T01:33:28.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:28 smithi027 ceph-mon[29197]: pgmap v248: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-04-25T01:33:28.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:28 smithi153 ceph-mon[32621]: pgmap v248: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-04-25T01:33:28.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:28 smithi139 ceph-mon[32760]: pgmap v248: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-04-25T01:33:30.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:30 smithi027 ceph-mon[29197]: pgmap v249: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:33:30.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:30 smithi153 ceph-mon[32621]: pgmap v249: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:33:30.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:30 smithi139 ceph-mon[32760]: pgmap v249: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:33:32.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:32 smithi027 ceph-mon[29197]: pgmap v250: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:33:32.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:32 smithi153 ceph-mon[32621]: pgmap v250: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:33:32.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:32 smithi139 ceph-mon[32760]: pgmap v250: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:33:34.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:34 smithi027 ceph-mon[29197]: pgmap v251: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:33:34.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:34 smithi153 ceph-mon[32621]: pgmap v251: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:33:34.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:34 smithi139 ceph-mon[32760]: pgmap v251: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:33:35.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:33:35.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:33:35.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 -- 172.21.15.27:0/2878249800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e04107720 msgr2=0x7f1e04107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/2878249800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e04107720 0x7f1e04107b00 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009930 tx=0x7f1e0002a420 comp rx=0 tx=0).stop 2024-04-25T01:33:35.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 -- 172.21.15.27:0/2878249800 shutdown_connections 2024-04-25T01:33:35.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/2878249800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e04100600 0x7f1e0410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/2878249800 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e040ffc40 0x7f1e041000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/2878249800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e04107720 0x7f1e04107b00 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 -- 172.21.15.27:0/2878249800 >> 172.21.15.27:0/2878249800 conn(0x7f1e04076c00 msgr2=0x7f1e040fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 -- 172.21.15.27:0/2878249800 shutdown_connections 2024-04-25T01:33:35.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.706+0000 7f1e0c47d640 1 -- 172.21.15.27:0/2878249800 wait complete. 2024-04-25T01:33:35.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0c47d640 1 Processor -- start 2024-04-25T01:33:35.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0c47d640 1 -- start start 2024-04-25T01:33:35.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0c47d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e040ffc40 0x7f1e041980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0c47d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e04100600 0x7f1e04198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0c47d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e04107720 0x7f1e0419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0c47d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04111b80 con 0x7f1e040ffc40 2024-04-25T01:33:35.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0c47d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04111a00 con 0x7f1e04100600 2024-04-25T01:33:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0c47d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04111d00 con 0x7f1e04107720 2024-04-25T01:33:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0a219640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e040ffc40 0x7f1e041980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0a219640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e040ffc40 0x7f1e041980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39558/0 (socket says 172.21.15.27:39558) 2024-04-25T01:33:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.707+0000 7f1e0a219640 1 -- 172.21.15.27:0/472390540 learned_addr learned my addr 172.21.15.27:0/472390540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e09a18640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e04100600 0x7f1e04198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e04107720 0x7f1e0419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0a219640 1 -- 172.21.15.27:0/472390540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e04107720 msgr2=0x7f1e0419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0a219640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e04107720 0x7f1e0419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0a219640 1 -- 172.21.15.27:0/472390540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e04100600 msgr2=0x7f1e04198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0a219640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e04100600 0x7f1e04198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0a219640 1 -- 172.21.15.27:0/472390540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0419d0b0 con 0x7f1e040ffc40 2024-04-25T01:33:35.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0a219640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e040ffc40 0x7f1e041980c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1e0002a940 tx=0x7f1e00003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1dfb7fe640 1 -- 172.21.15.27:0/472390540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0003f070 con 0x7f1e040ffc40 2024-04-25T01:33:35.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1dfb7fe640 1 -- 172.21.15.27:0/472390540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e00031c90 con 0x7f1e040ffc40 2024-04-25T01:33:35.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0c47d640 1 -- 172.21.15.27:0/472390540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e0419d340 con 0x7f1e040ffc40 2024-04-25T01:33:35.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1dfb7fe640 1 -- 172.21.15.27:0/472390540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00035030 con 0x7f1e040ffc40 2024-04-25T01:33:35.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.708+0000 7f1e0c47d640 1 -- 172.21.15.27:0/472390540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e041a4c30 con 0x7f1e040ffc40 2024-04-25T01:33:35.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.710+0000 7f1dfb7fe640 1 -- 172.21.15.27:0/472390540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1e00042070 con 0x7f1e040ffc40 2024-04-25T01:33:35.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.711+0000 7f1e0f18c740 1 Processor -- start 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.711+0000 7f1e0f18c740 1 -- start start 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.711+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672c630 0x561a6672ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.711+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672cf50 0x561a667370e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.711+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a66739a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.711+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561a666916b0 con 0x561a6672c630 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.711+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561a66639880 con 0x561a6672cf50 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.711+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a6668d670 con 0x561a66737620 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1dfb7fe640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1df00a2d00 0x7f1df00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672c630 0x561a6672ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672c630 0x561a6672ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39564/0 (socket says 172.21.15.27:39564) 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 -- 172.21.15.27:0/1700464635 learned_addr learned my addr 172.21.15.27:0/1700464635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e09a18640 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a66739a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1df00a2d00 0x7f1df00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0a219640 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672cf50 0x561a667370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1dfb7fe640 1 -- 172.21.15.27:0/472390540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1e000f0090 con 0x7f1e040ffc40 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 -- 172.21.15.27:0/1700464635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 msgr2=0x561a66739a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a66739a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 -- 172.21.15.27:0/1700464635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672cf50 msgr2=0x561a667370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672cf50 0x561a667370e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 -- 172.21.15.27:0/1700464635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a6646f7d0 con 0x561a6672c630 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672c630 0x561a6672ca10 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc014c10 tx=0x7f1dfc016720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9a1e26ebfb0c4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1df00a2d00 0x7f1df00a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc006d20 tx=0x7f1dfc01a000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.712+0000 7f1df8bfe640 1 -- 172.21.15.27:0/1700464635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc01c070 con 0x561a6672c630 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1df8bfe640 1 -- 172.21.15.27:0/1700464635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc014e80 con 0x561a6672c630 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1df8bfe640 1 -- 172.21.15.27:0/1700464635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0173c0 con 0x561a6672c630 2024-04-25T01:33:35.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 -- 172.21.15.27:0/1700464635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672c630 msgr2=0x561a6672ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672c630 0x561a6672ca10 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc014c10 tx=0x7f1dfc016720 comp rx=0 tx=0).stop 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 -- 172.21.15.27:0/1700464635 shutdown_connections 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a66739a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672cf50 0x561a667370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/1700464635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672c630 0x561a6672ca10 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 -- 172.21.15.27:0/1700464635 >> 172.21.15.27:0/1700464635 conn(0x561a66728950 msgr2=0x561a66728d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 -- 172.21.15.27:0/1700464635 shutdown_connections 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.713+0000 7f1e0f18c740 1 -- 172.21.15.27:0/1700464635 wait complete. 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 1 -- start start 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 1 Processor -- start 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a6672c630 0x561a66734660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66734ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a66737620 0x561a667350e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561a6673a350 con 0x561a6672cf50 2024-04-25T01:33:35.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561a6664cf50 con 0x561a66737620 2024-04-25T01:33:35.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a6673a4d0 con 0x561a6672c630 2024-04-25T01:33:35.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0f18c740 10 client.0 _ll_register_callbacks cb 0x561a66688000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:35.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e09a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66734ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e09a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66734ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39572/0 (socket says 172.21.15.27:39572) 2024-04-25T01:33:35.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e09a18640 1 -- 172.21.15.27:0/308418619 learned_addr learned my addr 172.21.15.27:0/308418619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0a219640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a6672c630 0x561a66734660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a66737620 0x561a667350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e09a18640 1 -- 172.21.15.27:0/308418619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a6672c630 msgr2=0x561a66734660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e09a18640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a6672c630 0x561a66734660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e09a18640 1 -- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a66737620 msgr2=0x561a667350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e09a18640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a66737620 0x561a667350e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e09a18640 1 -- 172.21.15.27:0/308418619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a663a53f0 con 0x561a6672cf50 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.714+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a66737620 0x561a667350e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0a219640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a6672c630 0x561a66734660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e09a18640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66734ba0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1df400bee0 tx=0x7f1df4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3221a457b8bce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1dedffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1dedffb640 1 -- 172.21.15.27:0/308418619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4004230 con 0x561a6672cf50 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1dedffb640 1 -- 172.21.15.27:0/308418619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1df40043d0 con 0x561a6672cf50 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 1 -- 172.21.15.27:0/308418619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561a6669ebf0 con 0x561a6672cf50 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1dedffb640 1 -- 172.21.15.27:0/308418619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1df40106a0 con 0x561a6672cf50 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 10 client.17355 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 1 -- 172.21.15.27:0/308418619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561a663650f0 con 0x561a6672cf50 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1dedffb640 1 -- 172.21.15.27:0/308418619 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561a6669ebf0 con 0x561a6672cf50 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1dedffb640 1 -- 172.21.15.27:0/308418619 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1df4003c80 con 0x561a6672cf50 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 10 client.17355 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 10 client.17355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 20 client.17355 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x561a66747960 0x561a66749e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 4 client.17355 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.715+0000 7f1e0f18c740 1 -- 172.21.15.27:0/308418619 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a66698f40 con 0x561a66747960 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.716+0000 7f1e0a219640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x561a66747960 0x561a66749e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.716+0000 7f1e0a219640 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x561a66747960 0x561a66749e00 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20682602a98ee8c3 server_cookie=b1f6d3b3bfb111c5 in_seq=0 out_seq=0 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.716+0000 7f1dedffb640 10 client.17355 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1dedffb640 1 -- 172.21.15.27:0/308418619 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a66698f40 con 0x561a66747960 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1dedffb640 10 client.17355 handle_command_reply: tid=0 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 client.17355 shutdown 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 -- 172.21.15.27:0/308418619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 msgr2=0x561a66734ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66734ba0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1df400bee0 tx=0x7f1df4012820 comp rx=0 tx=0).stop 2024-04-25T01:33:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 -- 172.21.15.27:0/308418619 shutdown_connections 2024-04-25T01:33:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x561a66747960 0x561a66749e00 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a66737620 0x561a667350e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66734ba0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1dedffb640 0 client.17355 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/308418619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a6672c630 0x561a66734660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 -- 172.21.15.27:0/308418619 >> 172.21.15.27:0/308418619 conn(0x561a66728950 msgr2=0x561a6672e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 -- 172.21.15.27:0/308418619 shutdown_connections 2024-04-25T01:33:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 1 -- 172.21.15.27:0/308418619 wait complete. 2024-04-25T01:33:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.717+0000 7f1e0f18c740 20 client.17355 trim_cache size 0 max 16384 2024-04-25T01:33:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e0f18c740 1 Processor -- start 2024-04-25T01:33:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e0f18c740 1 -- start start 2024-04-25T01:33:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 0x561a66736d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66743210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a66727970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041b1640 con 0x561a6672cf50 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041aef10 con 0x561a6672c630 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041acd80 con 0x561a66737620 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e09a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66743210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.735+0000 7f1e09a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66743210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39580/0 (socket says 172.21.15.27:39580) 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1e09a18640 1 -- 172.21.15.27:0/434684162 learned_addr learned my addr 172.21.15.27:0/434684162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1e09a18640 1 -- 172.21.15.27:0/434684162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 msgr2=0x561a66727970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1e09a18640 1 --2- 172.21.15.27:0/434684162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a66727970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1e09a18640 1 -- 172.21.15.27:0/434684162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 msgr2=0x561a66736d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1e09a18640 1 --2- 172.21.15.27:0/434684162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 0x561a66736d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1e09a18640 1 -- 172.21.15.27:0/434684162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a6671a090 con 0x561a6672cf50 2024-04-25T01:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1e09a18640 1 --2- 172.21.15.27:0/434684162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66743210 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1df401b5a0 tx=0x7f1df4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1941974b5d61b9e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1deffff640 1 -- 172.21.15.27:0/434684162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df401bca0 con 0x561a6672cf50 2024-04-25T01:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1deffff640 1 -- 172.21.15.27:0/434684162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df401be40 con 0x561a6672cf50 2024-04-25T01:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.736+0000 7f1deffff640 1 -- 172.21.15.27:0/434684162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4005610 con 0x561a6672cf50 2024-04-25T01:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- 172.21.15.27:0/434684162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 msgr2=0x561a66743210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/434684162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66743210 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1df401b5a0 tx=0x7f1df4000bc0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- 172.21.15.27:0/434684162 shutdown_connections 2024-04-25T01:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/434684162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a66727970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/434684162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66743210 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/434684162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 0x561a66736d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- 172.21.15.27:0/434684162 >> 172.21.15.27:0/434684162 conn(0x561a66728950 msgr2=0x561a66727260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- 172.21.15.27:0/434684162 shutdown_connections 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- 172.21.15.27:0/434684162 wait complete. 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- start start 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 Processor -- start 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 0x561a667316a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66731be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a667acf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041ad3d0 con 0x561a6672cf50 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041ae840 con 0x561a6672c630 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041adaa0 con 0x561a66737620 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e0f18c740 10 client.0 _ll_register_callbacks cb 0x7f1df400b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.737+0000 7f1e09a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66731be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e09a18640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66731be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39594/0 (socket says 172.21.15.27:39594) 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e09a18640 1 -- 172.21.15.27:0/2297870537 learned_addr learned my addr 172.21.15.27:0/2297870537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e0aa1a640 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a667acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e0a219640 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 0x561a667316a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e09a18640 1 -- 172.21.15.27:0/2297870537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 msgr2=0x561a667acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e09a18640 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a667acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e09a18640 1 -- 172.21.15.27:0/2297870537 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 msgr2=0x561a667316a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e09a18640 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 0x561a667316a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e09a18640 1 -- 172.21.15.27:0/2297870537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a667324b0 con 0x561a6672cf50 2024-04-25T01:33:35.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e09a18640 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66731be0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1df400bee0 tx=0x7f1df400bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7649eaf804d7d21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1dedffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:35.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1dedffb640 1 -- 172.21.15.27:0/2297870537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4004240 con 0x561a6672cf50 2024-04-25T01:33:35.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1dedffb640 1 -- 172.21.15.27:0/2297870537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1df40043e0 con 0x561a6672cf50 2024-04-25T01:33:35.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e0f18c740 1 -- 172.21.15.27:0/2297870537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561a666166f0 con 0x561a6672cf50 2024-04-25T01:33:35.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1dedffb640 1 -- 172.21.15.27:0/2297870537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1df4044050 con 0x561a6672cf50 2024-04-25T01:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e0f18c740 10 client.17361 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.738+0000 7f1e0f18c740 1 -- 172.21.15.27:0/2297870537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561a667ae5d0 con 0x561a6672cf50 2024-04-25T01:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1dedffb640 1 -- 172.21.15.27:0/2297870537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561a666166f0 con 0x561a6672cf50 2024-04-25T01:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1dedffb640 1 -- 172.21.15.27:0/2297870537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1df4004aa0 con 0x561a6672cf50 2024-04-25T01:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1e0f18c740 10 client.17361 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1e0f18c740 10 client.17361 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1e0f18c740 20 client.17361 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x561a667aebe0 0x561a667b1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1e0f18c740 4 client.17361 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1e0f18c740 1 -- 172.21.15.27:0/2297870537 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561a66662d50 con 0x561a667aebe0 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1e0a219640 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x561a667aebe0 0x561a667b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1e0a219640 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x561a667aebe0 0x561a667b1000 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86cfb01f3f2b1bc9 server_cookie=f7994f6b05f8447 in_seq=0 out_seq=0 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.739+0000 7f1dedffb640 10 client.17361 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1dedffb640 1 -- 172.21.15.27:0/2297870537 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561a66662d50 con 0x561a667aebe0 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1dedffb640 10 client.17361 handle_command_reply: tid=0 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 client.17361 shutdown 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 -- 172.21.15.27:0/2297870537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 msgr2=0x561a66731be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66731be0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1df400bee0 tx=0x7f1df400bb50 comp rx=0 tx=0).stop 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 -- 172.21.15.27:0/2297870537 shutdown_connections 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x561a667aebe0 0x561a667b1000 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a66737620 0x561a667acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a6672cf50 0x561a66731be0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 --2- 172.21.15.27:0/2297870537 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561a6672c630 0x561a667316a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1dedffb640 0 client.17361 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 -- 172.21.15.27:0/2297870537 >> 172.21.15.27:0/2297870537 conn(0x561a66728950 msgr2=0x561a66727af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 -- 172.21.15.27:0/2297870537 shutdown_connections 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 1 -- 172.21.15.27:0/2297870537 wait complete. 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0f18c740 20 client.17361 trim_cache size 0 max 16384 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:33:35.725 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:33:35.726 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:33:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 -- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1df00a2d00 msgr2=0x7f1df00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1df00a2d00 0x7f1df00a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc006d20 tx=0x7f1dfc01a000 comp rx=0 tx=0).stop 2024-04-25T01:33:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 -- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e040ffc40 msgr2=0x7f1e041980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e040ffc40 0x7f1e041980c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1e0002a940 tx=0x7f1e00003b90 comp rx=0 tx=0).stop 2024-04-25T01:33:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 -- 172.21.15.27:0/472390540 shutdown_connections 2024-04-25T01:33:35.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e04107720 0x7f1e0419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e04100600 0x7f1e04198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1df00a2d00 0x7f1df00a51c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 --2- 172.21.15.27:0/472390540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e040ffc40 0x7f1e041980c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 -- 172.21.15.27:0/472390540 >> 172.21.15.27:0/472390540 conn(0x7f1e04076c00 msgr2=0x7f1e040fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.740+0000 7f1e0c47d640 1 -- 172.21.15.27:0/472390540 shutdown_connections 2024-04-25T01:33:35.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.741+0000 7f1e0c47d640 1 -- 172.21.15.27:0/472390540 wait complete. 2024-04-25T01:33:35.729 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:33:35.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:33:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 -- 172.21.15.27:0/402453606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 msgr2=0x7f581010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 --2- 172.21.15.27:0/402453606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 0x7f581010d4f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f5808009930 tx=0x7f580802a420 comp rx=0 tx=0).stop 2024-04-25T01:33:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 -- 172.21.15.27:0/402453606 shutdown_connections 2024-04-25T01:33:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 --2- 172.21.15.27:0/402453606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 0x7f581010d4f0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 --2- 172.21.15.27:0/402453606 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5810100620 0x7f5810100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 --2- 172.21.15.27:0/402453606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5810106630 0x7f5810106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 -- 172.21.15.27:0/402453606 >> 172.21.15.27:0/402453606 conn(0x7f5810077740 msgr2=0x7f5810077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 -- 172.21.15.27:0/402453606 shutdown_connections 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 -- 172.21.15.27:0/402453606 wait complete. 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 Processor -- start 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.855+0000 7f58184f7640 1 -- start start 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f58184f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5810100620 0x7f581010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f58184f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 0x7f581010e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f58184f7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5810106630 0x7f581010ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f58184f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5810111b40 con 0x7f5810100fe0 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f58184f7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f58101119c0 con 0x7f5810106630 2024-04-25T01:33:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f58184f7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5810111cc0 con 0x7f5810100620 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 0x7f581010e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 0x7f581010e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39608/0 (socket says 172.21.15.27:39608) 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 -- 172.21.15.27:0/3977475478 learned_addr learned my addr 172.21.15.27:0/3977475478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5816a94640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5810106630 0x7f581010ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5816293640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5810100620 0x7f581010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 -- 172.21.15.27:0/3977475478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5810100620 msgr2=0x7f581010df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5810100620 0x7f581010df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 -- 172.21.15.27:0/3977475478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5810106630 msgr2=0x7f581010ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5810106630 0x7f581010ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 -- 172.21.15.27:0/3977475478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58101a6460 con 0x7f5810100fe0 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5816a94640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5810106630 0x7f581010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f5815a92640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 0x7f581010e4c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f580000cce0 tx=0x7f580000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f580f7fe640 1 -- 172.21.15.27:0/3977475478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580000f650 con 0x7f5810100fe0 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f580f7fe640 1 -- 172.21.15.27:0/3977475478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f580000fcd0 con 0x7f5810100fe0 2024-04-25T01:33:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58101a6750 con 0x7f5810100fe0 2024-04-25T01:33:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.856+0000 7f580f7fe640 1 -- 172.21.15.27:0/3977475478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5800018c70 con 0x7f5810100fe0 2024-04-25T01:33:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.857+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58101a6ca0 con 0x7f5810100fe0 2024-04-25T01:33:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.857+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5810068560 con 0x7f5810100fe0 2024-04-25T01:33:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.858+0000 7f580f7fe640 1 -- 172.21.15.27:0/3977475478 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f580000f7f0 con 0x7f5810100fe0 2024-04-25T01:33:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.859+0000 7f580f7fe640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57f80a2d30 0x7f57f80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.860+0000 7f5816293640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57f80a2d30 0x7f57f80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.860+0000 7f580f7fe640 1 -- 172.21.15.27:0/3977475478 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f58000cfd80 con 0x7f5810100fe0 2024-04-25T01:33:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.860+0000 7f5816293640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57f80a2d30 0x7f57f80a51f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5804005e00 tx=0x7f5804005c50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:35.864+0000 7f580f7fe640 1 -- 172.21.15.27:0/3977475478 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f58000d3030 con 0x7f5810100fe0 2024-04-25T01:33:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.088+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f581010f460 con 0x7f5810100fe0 2024-04-25T01:33:36.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.089+0000 7f580f7fe640 1 -- 172.21.15.27:0/3977475478 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7f5800014030 con 0x7f5810100fe0 2024-04-25T01:33:36.065 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:33:36.066 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:33:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:33:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57f80a2d30 msgr2=0x7f57f80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57f80a2d30 0x7f57f80a51f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5804005e00 tx=0x7f5804005c50 comp rx=0 tx=0).stop 2024-04-25T01:33:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 msgr2=0x7f581010e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 0x7f581010e4c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f580000cce0 tx=0x7f580000a620 comp rx=0 tx=0).stop 2024-04-25T01:33:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 shutdown_connections 2024-04-25T01:33:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5810106630 0x7f581010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f57f80a2d30 0x7f57f80a51f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810100fe0 0x7f581010e4c0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 --2- 172.21.15.27:0/3977475478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5810100620 0x7f581010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 >> 172.21.15.27:0/3977475478 conn(0x7f5810077740 msgr2=0x7f5810102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 shutdown_connections 2024-04-25T01:33:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.091+0000 7f58184f7640 1 -- 172.21.15.27:0/3977475478 wait complete. 2024-04-25T01:33:36.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:33:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.207+0000 7fc331d96640 1 -- 172.21.15.27:0/97274553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1032e0 msgr2=0x7fc32c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.207+0000 7fc331d96640 1 --2- 172.21.15.27:0/97274553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1032e0 0x7fc32c10b7c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc320009900 tx=0x7fc32002a510 comp rx=0 tx=0).stop 2024-04-25T01:33:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.207+0000 7fc331d96640 1 -- 172.21.15.27:0/97274553 shutdown_connections 2024-04-25T01:33:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.207+0000 7fc331d96640 1 --2- 172.21.15.27:0/97274553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c10bd00 0x7fc32c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.207+0000 7fc331d96640 1 --2- 172.21.15.27:0/97274553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1032e0 0x7fc32c10b7c0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.207+0000 7fc331d96640 1 --2- 172.21.15.27:0/97274553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc32c1029c0 0x7fc32c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.207+0000 7fc331d96640 1 -- 172.21.15.27:0/97274553 >> 172.21.15.27:0/97274553 conn(0x7fc32c077740 msgr2=0x7fc32c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.208+0000 7fc331d96640 1 -- 172.21.15.27:0/97274553 shutdown_connections 2024-04-25T01:33:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.208+0000 7fc331d96640 1 -- 172.21.15.27:0/97274553 wait complete. 2024-04-25T01:33:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.208+0000 7fc331d96640 1 Processor -- start 2024-04-25T01:33:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.208+0000 7fc331d96640 1 -- start start 2024-04-25T01:33:36.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc331d96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1029c0 0x7fc32c100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc331d96640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c1032e0 0x7fc32c100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc331d96640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc32c10bd00 0x7fc32c101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc331d96640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c111b00 con 0x7fc32c1029c0 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc331d96640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c111980 con 0x7fc32c10bd00 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc331d96640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c111c80 con 0x7fc32c1032e0 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1029c0 0x7fc32c100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1029c0 0x7fc32c100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39620/0 (socket says 172.21.15.27:39620) 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 -- 172.21.15.27:0/3317091866 learned_addr learned my addr 172.21.15.27:0/3317091866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32affd640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c1032e0 0x7fc32c100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32bfff640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc32c10bd00 0x7fc32c101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 -- 172.21.15.27:0/3317091866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c1032e0 msgr2=0x7fc32c100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c1032e0 0x7fc32c100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 -- 172.21.15.27:0/3317091866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc32c10bd00 msgr2=0x7fc32c101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc32c10bd00 0x7fc32c101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 -- 172.21.15.27:0/3317091866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c1a64e0 con 0x7fc32c1029c0 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.209+0000 7fc32b7fe640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1029c0 0x7fc32c100610 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e9f0 tx=0x7fc31c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.210+0000 7fc328ff9640 1 -- 172.21.15.27:0/3317091866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c00cd80 con 0x7fc32c1029c0 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.210+0000 7fc328ff9640 1 -- 172.21.15.27:0/3317091866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31c010910 con 0x7fc32c1029c0 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.210+0000 7fc331d96640 1 -- 172.21.15.27:0/3317091866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c197d90 con 0x7fc32c1029c0 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.210+0000 7fc328ff9640 1 -- 172.21.15.27:0/3317091866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c018990 con 0x7fc32c1029c0 2024-04-25T01:33:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.210+0000 7fc331d96640 1 -- 172.21.15.27:0/3317091866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c198300 con 0x7fc32c1029c0 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc328ff9640 1 -- 172.21.15.27:0/3317091866 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc31c018b30 con 0x7fc32c1029c0 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc334b76740 1 Processor -- start 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc334b76740 1 -- start start 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc334b76740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1580 0x5568b06e1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc334b76740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1ea0 0x5568b06ec030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc334b76740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ee960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc334b76740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5568b0646670 con 0x5568b06e1580 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc334b76740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5568b05ee970 con 0x5568b06ec570 2024-04-25T01:33:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc334b76740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5568b0642630 con 0x5568b06e1ea0 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1580 0x5568b06e1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1ea0 0x5568b06ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1580 0x5568b06e1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39632/0 (socket says 172.21.15.27:39632) 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc32bfff640 1 -- 172.21.15.27:0/757976542 learned_addr learned my addr 172.21.15.27:0/757976542 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc32affd640 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc32bfff640 1 -- 172.21.15.27:0/757976542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1ea0 msgr2=0x5568b06ec030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc32bfff640 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1ea0 0x5568b06ec030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.213+0000 7fc32bfff640 1 -- 172.21.15.27:0/757976542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 msgr2=0x5568b06ee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc32bfff640 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ee960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc32bfff640 1 -- 172.21.15.27:0/757976542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568b0612400 con 0x5568b06e1580 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc32affd640 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc328ff9640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3140a2d80 0x7fc3140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc32b7fe640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3140a2d80 0x7fc3140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc32bfff640 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1580 0x5568b06e1960 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fc324009f50 tx=0x7fc3240118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb632fbdec46e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc328ff9640 1 -- 172.21.15.27:0/3317091866 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fc31c0cfaf0 con 0x7fc32c1029c0 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc311ffb640 1 -- 172.21.15.27:0/757976542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324004d20 con 0x5568b06e1580 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc311ffb640 1 -- 172.21.15.27:0/757976542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc324004050 con 0x5568b06e1580 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.214+0000 7fc311ffb640 1 -- 172.21.15.27:0/757976542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324005280 con 0x5568b06e1580 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 -- 172.21.15.27:0/757976542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1580 msgr2=0x5568b06e1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1580 0x5568b06e1960 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fc324009f50 tx=0x7fc3240118d0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc32b7fe640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3140a2d80 0x7fc3140a5240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc31c0ccf00 tx=0x7fc31c0d3640 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 -- 172.21.15.27:0/757976542 shutdown_connections 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1ea0 0x5568b06ec030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 --2- 172.21.15.27:0/757976542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1580 0x5568b06e1960 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 -- 172.21.15.27:0/757976542 >> 172.21.15.27:0/757976542 conn(0x5568b06dd8a0 msgr2=0x5568b06ddc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 -- 172.21.15.27:0/757976542 shutdown_connections 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 -- 172.21.15.27:0/757976542 wait complete. 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.215+0000 7fc334b76740 1 -- start start 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc334b76740 1 Processor -- start 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc334b76740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc334b76740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06e91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc334b76740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b06e9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc334b76740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5568b06ef360 con 0x5568b06ec570 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc334b76740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5568b0602050 con 0x5568b06e1ea0 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc334b76740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5568b06ef4e0 con 0x5568b06e1580 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc334b76740 10 client.0 _ll_register_callbacks cb 0x5568b0359be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b06e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b06e9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39636/0 (socket says 172.21.15.27:39636) 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 -- 172.21.15.27:0/1504430851 learned_addr learned my addr 172.21.15.27:0/1504430851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32affd640 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06e91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 -- 172.21.15.27:0/1504430851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 msgr2=0x5568b06ebf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06ebf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 -- 172.21.15.27:0/1504430851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 msgr2=0x5568b06e91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06e91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 -- 172.21.15.27:0/1504430851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568b05d3dc0 con 0x5568b06ec570 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc32bfff640 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b06e9770 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fc324009250 tx=0x7fc324005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655ee52e5cb67985 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.216+0000 7fc30affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc30affd640 1 -- 172.21.15.27:0/1504430851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32402b020 con 0x5568b06ec570 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc30affd640 1 -- 172.21.15.27:0/1504430851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fc324025450 con 0x5568b06ec570 2024-04-25T01:33:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 1 -- 172.21.15.27:0/1504430851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc324004050 con 0x5568b06ec570 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc30affd640 1 -- 172.21.15.27:0/1504430851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc324003e40 con 0x5568b06ec570 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 10 client.17379 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 1 -- 172.21.15.27:0/1504430851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5568b035a3f0 con 0x5568b06ec570 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc30affd640 1 -- 172.21.15.27:0/1504430851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc324004050 con 0x5568b06ec570 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc30affd640 1 -- 172.21.15.27:0/1504430851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc3240277d0 con 0x5568b06ec570 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 10 client.17379 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 10 client.17379 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 20 client.17379 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5568b06fc720 0x5568b06feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 4 client.17379 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.217+0000 7fc334b76740 1 -- 172.21.15.27:0/1504430851 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568b0653bb0 con 0x5568b06fc720 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.218+0000 7fc32affd640 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5568b06fc720 0x5568b06feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.218+0000 7fc32affd640 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5568b06fc720 0x5568b06feb10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2b3a910cf7284466 server_cookie=f5ef139b6e280bda in_seq=0 out_seq=0 2024-04-25T01:33:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.218+0000 7fc30affd640 10 client.17379 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc30affd640 1 -- 172.21.15.27:0/1504430851 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5568b0653bb0 con 0x5568b06fc720 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc30affd640 10 client.17379 handle_command_reply: tid=0 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 client.17379 shutdown 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 -- 172.21.15.27:0/1504430851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 msgr2=0x5568b06e9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b06e9770 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fc324009250 tx=0x7fc324005bd0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 -- 172.21.15.27:0/1504430851 shutdown_connections 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5568b06fc720 0x5568b06feb10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b06e9770 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06e91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 --2- 172.21.15.27:0/1504430851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 -- 172.21.15.27:0/1504430851 >> 172.21.15.27:0/1504430851 conn(0x5568b06dd8a0 msgr2=0x5568b06e36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc30affd640 0 client.17379 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 -- 172.21.15.27:0/1504430851 shutdown_connections 2024-04-25T01:33:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 1 -- 172.21.15.27:0/1504430851 wait complete. 2024-04-25T01:33:36.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.219+0000 7fc334b76740 20 client.17379 trim_cache size 0 max 16384 2024-04-25T01:33:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.239+0000 7fc334b76740 1 Processor -- start 2024-04-25T01:33:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.239+0000 7fc334b76740 1 -- start start 2024-04-25T01:33:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc334b76740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06e46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc334b76740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06eba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc334b76740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b0701660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc334b76740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1b7c00 con 0x5568b06ec570 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc334b76740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1b54d0 con 0x5568b06e1ea0 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc334b76740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1b3340 con 0x5568b06e1580 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b0701660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06eba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06eba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:33470/0 (socket says 172.21.15.27:33470) 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b0701660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39650/0 (socket says 172.21.15.27:39650) 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 -- 172.21.15.27:0/1241985324 learned_addr learned my addr 172.21.15.27:0/1241985324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32bfff640 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06e46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 -- 172.21.15.27:0/1241985324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 msgr2=0x5568b06e46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06e46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 -- 172.21.15.27:0/1241985324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 msgr2=0x5568b0701660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b0701660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 -- 172.21.15.27:0/1241985324 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568b04247d0 con 0x5568b06e1ea0 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32bfff640 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b0701660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc32affd640 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06eba30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc320009340 tx=0x7fc320004310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d125f484e14111e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.240+0000 7fc3117fa640 1 -- 172.21.15.27:0/1241985324 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320035040 con 0x5568b06e1ea0 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc3117fa640 1 -- 172.21.15.27:0/1241985324 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3200044c0 con 0x5568b06e1ea0 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc3117fa640 1 -- 172.21.15.27:0/1241985324 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320005280 con 0x5568b06e1ea0 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 -- 172.21.15.27:0/1241985324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 msgr2=0x5568b06eba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06eba30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc320009340 tx=0x7fc320004310 comp rx=0 tx=0).stop 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 -- 172.21.15.27:0/1241985324 shutdown_connections 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06ec570 0x5568b0701660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06e1ea0 0x5568b06eba30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 --2- 172.21.15.27:0/1241985324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06e46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 -- 172.21.15.27:0/1241985324 >> 172.21.15.27:0/1241985324 conn(0x5568b06dd8a0 msgr2=0x5568b06e3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 -- 172.21.15.27:0/1241985324 shutdown_connections 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.241+0000 7fc334b76740 1 -- 172.21.15.27:0/1241985324 wait complete. 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 1 -- start start 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 1 Processor -- start 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06ff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1ea0 0x5568b06ff5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1b3990 con 0x5568b06e1ea0 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1b4e00 con 0x5568b06ec570 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1b4060 con 0x5568b06e1580 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc334b76740 10 client.0 _ll_register_callbacks cb 0x7fc3240093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc32affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1ea0 0x5568b06ff5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc32affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1ea0 0x5568b06ff5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39660/0 (socket says 172.21.15.27:39660) 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc32affd640 1 -- 172.21.15.27:0/2672687821 learned_addr learned my addr 172.21.15.27:0/2672687821 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc32bfff640 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06ff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.242+0000 7fc32bfff640 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc32affd640 1 -- 172.21.15.27:0/2672687821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 msgr2=0x5568b06ff0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc32affd640 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06ff0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc32affd640 1 -- 172.21.15.27:0/2672687821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 msgr2=0x5568b06ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc32affd640 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc32affd640 1 -- 172.21.15.27:0/2672687821 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568b0761900 con 0x5568b06e1ea0 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc32affd640 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1ea0 0x5568b06ff5e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fc32004b000 tx=0x7fc32002a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ad9987a1c21707 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc30affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc30affd640 1 -- 172.21.15.27:0/2672687821 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320035070 con 0x5568b06e1ea0 2024-04-25T01:33:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc334b76740 1 -- 172.21.15.27:0/2672687821 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5568b064df00 con 0x5568b06e1ea0 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc30affd640 1 -- 172.21.15.27:0/2672687821 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fc320031840 con 0x5568b06e1ea0 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc30affd640 1 -- 172.21.15.27:0/2672687821 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc320060050 con 0x5568b06e1ea0 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc334b76740 10 client.17385 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc334b76740 1 -- 172.21.15.27:0/2672687821 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5568b0762be0 con 0x5568b06e1ea0 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.243+0000 7fc30affd640 1 -- 172.21.15.27:0/2672687821 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5568b064df00 con 0x5568b06e1ea0 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc30affd640 1 -- 172.21.15.27:0/2672687821 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc3200047e0 con 0x5568b06e1ea0 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc334b76740 10 client.17385 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc334b76740 10 client.17385 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc334b76740 20 client.17385 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc334b76740 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5568b0763330 0x5568b0765750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc334b76740 4 client.17385 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc334b76740 1 -- 172.21.15.27:0/2672687821 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568b05cb820 con 0x5568b0763330 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc32bfff640 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5568b0763330 0x5568b0765750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc30affd640 10 client.17385 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc32bfff640 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5568b0763330 0x5568b0765750 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eae380fed0c94823 server_cookie=3ac30619c87fd117 in_seq=0 out_seq=0 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc30affd640 1 -- 172.21.15.27:0/2672687821 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568b05cb820 con 0x5568b0763330 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.244+0000 7fc30affd640 10 client.17385 handle_command_reply: tid=0 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 client.17385 shutdown 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 -- 172.21.15.27:0/2672687821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1ea0 msgr2=0x5568b06ff5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1ea0 0x5568b06ff5e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fc32004b000 tx=0x7fc32002a510 comp rx=0 tx=0).stop 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 -- 172.21.15.27:0/2672687821 shutdown_connections 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5568b0763330 0x5568b0765750 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5568b06ec570 0x5568b06ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568b06e1ea0 0x5568b06ff5e0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc30affd640 0 client.17385 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 --2- 172.21.15.27:0/2672687821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5568b06e1580 0x5568b06ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 -- 172.21.15.27:0/2672687821 >> 172.21.15.27:0/2672687821 conn(0x5568b06dd8a0 msgr2=0x5568b06dcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 -- 172.21.15.27:0/2672687821 shutdown_connections 2024-04-25T01:33:36.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 1 -- 172.21.15.27:0/2672687821 wait complete. 2024-04-25T01:33:36.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc334b76740 20 client.17385 trim_cache size 0 max 16384 2024-04-25T01:33:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 -- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3140a2d80 msgr2=0x7fc3140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3140a2d80 0x7fc3140a5240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc31c0ccf00 tx=0x7fc31c0d3640 comp rx=0 tx=0).stop 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 -- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1029c0 msgr2=0x7fc32c100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1029c0 0x7fc32c100610 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e9f0 tx=0x7fc31c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 -- 172.21.15.27:0/3317091866 shutdown_connections 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc32c10bd00 0x7fc32c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c1032e0 0x7fc32c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3140a2d80 0x7fc3140a5240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 --2- 172.21.15.27:0/3317091866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc32c1029c0 0x7fc32c100610 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.245+0000 7fc331d96640 1 -- 172.21.15.27:0/3317091866 >> 172.21.15.27:0/3317091866 conn(0x7fc32c077740 msgr2=0x7fc32c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.246+0000 7fc331d96640 1 -- 172.21.15.27:0/3317091866 shutdown_connections 2024-04-25T01:33:36.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.246+0000 7fc331d96640 1 -- 172.21.15.27:0/3317091866 wait complete. 2024-04-25T01:33:36.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:33:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 -- 172.21.15.27:0/2716106864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201032e0 msgr2=0x7f1e2010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/2716106864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201032e0 0x7f1e2010b7c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009900 tx=0x7f1e1002a510 comp rx=0 tx=0).stop 2024-04-25T01:33:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 -- 172.21.15.27:0/2716106864 shutdown_connections 2024-04-25T01:33:36.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/2716106864 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e2010bd00 0x7f1e2010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/2716106864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201032e0 0x7f1e2010b7c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/2716106864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e201029c0 0x7f1e20102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 -- 172.21.15.27:0/2716106864 >> 172.21.15.27:0/2716106864 conn(0x7f1e20077740 msgr2=0x7f1e20077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 -- 172.21.15.27:0/2716106864 shutdown_connections 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.361+0000 7f1e27d6a640 1 -- 172.21.15.27:0/2716106864 wait complete. 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e27d6a640 1 Processor -- start 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e27d6a640 1 -- start start 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e27d6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201029c0 0x7f1e20100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e27d6a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e201032e0 0x7f1e20100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e27d6a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e2010bd00 0x7f1e20101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e27d6a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e20111b00 con 0x7f1e201029c0 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e27d6a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e20111980 con 0x7f1e2010bd00 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e27d6a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e20111c80 con 0x7f1e201032e0 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e25b06640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201029c0 0x7f1e20100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e25b06640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201029c0 0x7f1e20100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39692/0 (socket says 172.21.15.27:39692) 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e25b06640 1 -- 172.21.15.27:0/3816513859 learned_addr learned my addr 172.21.15.27:0/3816513859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e26307640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e2010bd00 0x7f1e20101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.362+0000 7f1e25305640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e201032e0 0x7f1e20100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e25b06640 1 -- 172.21.15.27:0/3816513859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e201032e0 msgr2=0x7f1e20100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e25b06640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e201032e0 0x7f1e20100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e25b06640 1 -- 172.21.15.27:0/3816513859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e2010bd00 msgr2=0x7f1e20101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e25b06640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e2010bd00 0x7f1e20101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e25b06640 1 -- 172.21.15.27:0/3816513859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e201a6520 con 0x7f1e201029c0 2024-04-25T01:33:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e25b06640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201029c0 0x7f1e20100610 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00d950 tx=0x7f1e1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e16ffd640 1 -- 172.21.15.27:0/3816513859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c017070 con 0x7f1e201029c0 2024-04-25T01:33:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e16ffd640 1 -- 172.21.15.27:0/3816513859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e1c009c30 con 0x7f1e201029c0 2024-04-25T01:33:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e27d6a640 1 -- 172.21.15.27:0/3816513859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e201a6810 con 0x7f1e201029c0 2024-04-25T01:33:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e16ffd640 1 -- 172.21.15.27:0/3816513859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c00f6b0 con 0x7f1e201029c0 2024-04-25T01:33:36.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.363+0000 7f1e27d6a640 1 -- 172.21.15.27:0/3816513859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e201a6d30 con 0x7f1e201029c0 2024-04-25T01:33:36.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.366+0000 7f1e16ffd640 1 -- 172.21.15.27:0/3816513859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1e1c0109d0 con 0x7f1e201029c0 2024-04-25T01:33:36.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.366+0000 7f1e2aa70740 1 Processor -- start 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.366+0000 7f1e2aa70740 1 -- start start 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf086960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086ea0 0x559faf091030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf093960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559faefeb670 con 0x559faf091570 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559faef93970 con 0x559faf086580 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559faefe7630 con 0x559faf086ea0 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25305640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf093960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e26307640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf086960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25305640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf093960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39698/0 (socket says 172.21.15.27:39698) 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e26307640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf086960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:33514/0 (socket says 172.21.15.27:33514) 2024-04-25T01:33:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25305640 1 -- 172.21.15.27:0/2647759469 learned_addr learned my addr 172.21.15.27:0/2647759469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25b06640 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086ea0 0x559faf091030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e16ffd640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1e0c0a2d00 0x7f1e0c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25305640 1 -- 172.21.15.27:0/2647759469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086ea0 msgr2=0x559faf091030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25305640 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086ea0 0x559faf091030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25305640 1 -- 172.21.15.27:0/2647759469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 msgr2=0x559faf086960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25305640 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf086960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25305640 1 -- 172.21.15.27:0/2647759469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559faefb7400 con 0x559faf091570 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e25b06640 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086ea0 0x559faf091030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e26307640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1e0c0a2d00 0x7f1e0c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.367+0000 7f1e16ffd640 1 -- 172.21.15.27:0/3816513859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1e1c0cf420 con 0x7f1e201029c0 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.368+0000 7f1e25305640 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf093960 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009900 tx=0x7f1e10005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5885aaf7fd570c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.368+0000 7f1e0bfff640 1 -- 172.21.15.27:0/2647759469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10040070 con 0x559faf091570 2024-04-25T01:33:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.368+0000 7f1e0bfff640 1 -- 172.21.15.27:0/2647759469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e100319f0 con 0x559faf091570 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.368+0000 7f1e0bfff640 1 -- 172.21.15.27:0/2647759469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10035040 con 0x559faf091570 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.368+0000 7f1e26307640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1e0c0a2d00 0x7f1e0c0a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e18005fd0 tx=0x7f1e18004a40 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.368+0000 7f1e2aa70740 1 -- 172.21.15.27:0/2647759469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 msgr2=0x559faf093960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.368+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf093960 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009900 tx=0x7f1e10005e80 comp rx=0 tx=0).stop 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 -- 172.21.15.27:0/2647759469 shutdown_connections 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf093960 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086ea0 0x559faf091030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/2647759469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf086960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 -- 172.21.15.27:0/2647759469 >> 172.21.15.27:0/2647759469 conn(0x559faf0828a0 msgr2=0x559faf082c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 -- 172.21.15.27:0/2647759469 shutdown_connections 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 -- 172.21.15.27:0/2647759469 wait complete. 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 -- start start 2024-04-25T01:33:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 Processor -- start 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086580 0x559faf090e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf08e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559faf094300 con 0x559faf091570 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x559faefa7050 con 0x559faf086ea0 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559faf094480 con 0x559faf086580 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.369+0000 7f1e2aa70740 10 client.0 _ll_register_callbacks cb 0x559faecfebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e26307640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf08e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25b06640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086580 0x559faf090e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25305640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e26307640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf08e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39712/0 (socket says 172.21.15.27:39712) 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25b06640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086580 0x559faf090e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36114/0 (socket says 172.21.15.27:36114) 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e26307640 1 -- 172.21.15.27:0/651481547 learned_addr learned my addr 172.21.15.27:0/651481547 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25b06640 1 -- 172.21.15.27:0/651481547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 msgr2=0x559faf08e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25b06640 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25b06640 1 -- 172.21.15.27:0/651481547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 msgr2=0x559faf08e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25b06640 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf08e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25b06640 1 -- 172.21.15.27:0/651481547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559faef78dc0 con 0x559faf086580 2024-04-25T01:33:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e25b06640 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086580 0x559faf090e60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0cc780 tx=0x7f1e1c010d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8c4353d2ff854ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e097fa640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e097fa640 1 -- 172.21.15.27:0/651481547 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c020070 con 0x559faf086580 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e097fa640 1 -- 172.21.15.27:0/651481547 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1e1c097c70 con 0x559faf086580 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.370+0000 7f1e2aa70740 1 -- 172.21.15.27:0/651481547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e100319f0 con 0x559faf086580 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e097fa640 1 -- 172.21.15.27:0/651481547 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1e1c097040 con 0x559faf086580 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e2aa70740 10 client.26536 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e2aa70740 1 -- 172.21.15.27:0/651481547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559faecff3f0 con 0x559faf086580 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e097fa640 1 -- 172.21.15.27:0/651481547 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e100319f0 con 0x559faf086580 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e097fa640 1 -- 172.21.15.27:0/651481547 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1e1c084b40 con 0x559faf086580 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e2aa70740 10 client.26536 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e2aa70740 10 client.26536 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:33:36.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e2aa70740 20 client.26536 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x559faf0a1390 0x559faf0a3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e2aa70740 4 client.26536 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e2aa70740 1 -- 172.21.15.27:0/651481547 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559faeff8bb0 con 0x559faf0a1390 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.371+0000 7f1e25305640 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x559faf0a1390 0x559faf0a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.372+0000 7f1e25305640 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x559faf0a1390 0x559faf0a3820 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fcc0d5323cda4d7f server_cookie=906f6bb64455e5f4 in_seq=0 out_seq=0 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.372+0000 7f1e097fa640 10 client.26536 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e097fa640 1 -- 172.21.15.27:0/651481547 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559faeff8bb0 con 0x559faf0a1390 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e097fa640 10 client.26536 handle_command_reply: tid=0 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 client.26536 shutdown 2024-04-25T01:33:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 -- 172.21.15.27:0/651481547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086580 msgr2=0x559faf090e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086580 0x559faf090e60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0cc780 tx=0x7f1e1c010d70 comp rx=0 tx=0).stop 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 -- 172.21.15.27:0/651481547 shutdown_connections 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x559faf0a1390 0x559faf0a3820 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf091570 0x559faf08e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e097fa640 0 client.26536 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/651481547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf086580 0x559faf090e60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 -- 172.21.15.27:0/651481547 >> 172.21.15.27:0/651481547 conn(0x559faf0828a0 msgr2=0x559faf081b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 -- 172.21.15.27:0/651481547 shutdown_connections 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 1 -- 172.21.15.27:0/651481547 wait complete. 2024-04-25T01:33:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.373+0000 7f1e2aa70740 20 client.26536 trim_cache size 0 max 16384 2024-04-25T01:33:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e2aa70740 1 Processor -- start 2024-04-25T01:33:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e2aa70740 1 -- start start 2024-04-25T01:33:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf09d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086ea0 0x559faf090990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 0x559faf0a9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e201b36c0 con 0x559faf086ea0 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e201b0f90 con 0x559faf086580 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e201aee00 con 0x559faf091570 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.387+0000 7f1e25305640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086ea0 0x559faf090990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25305640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086ea0 0x559faf090990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39728/0 (socket says 172.21.15.27:39728) 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25305640 1 -- 172.21.15.27:0/1008132680 learned_addr learned my addr 172.21.15.27:0/1008132680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25b06640 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf09d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e26307640 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 0x559faf0a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25305640 1 -- 172.21.15.27:0/1008132680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 msgr2=0x559faf0a9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25305640 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 0x559faf0a9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25305640 1 -- 172.21.15.27:0/1008132680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 msgr2=0x559faf09d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25305640 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf09d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25305640 1 -- 172.21.15.27:0/1008132680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559faedc97d0 con 0x559faf086ea0 2024-04-25T01:33:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25b06640 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf09d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e25305640 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086ea0 0x559faf090990 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1e1002a550 tx=0x7f1e100314c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d574279ca3dbf42d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e0b7fe640 1 -- 172.21.15.27:0/1008132680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10040040 con 0x559faf086ea0 2024-04-25T01:33:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e0b7fe640 1 -- 172.21.15.27:0/1008132680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e10005520 con 0x559faf086ea0 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.388+0000 7f1e0b7fe640 1 -- 172.21.15.27:0/1008132680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10035040 con 0x559faf086ea0 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 -- 172.21.15.27:0/1008132680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086ea0 msgr2=0x559faf090990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086ea0 0x559faf090990 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1e1002a550 tx=0x7f1e100314c0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 -- 172.21.15.27:0/1008132680 shutdown_connections 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 0x559faf0a9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086ea0 0x559faf090990 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/1008132680 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086580 0x559faf09d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 -- 172.21.15.27:0/1008132680 >> 172.21.15.27:0/1008132680 conn(0x559faf0828a0 msgr2=0x559faf089860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 -- 172.21.15.27:0/1008132680 shutdown_connections 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 -- 172.21.15.27:0/1008132680 wait complete. 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 -- start start 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.389+0000 7f1e2aa70740 1 Processor -- start 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086580 0x559faf0998c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e2aa70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 0x559faf08af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e201af450 con 0x559faf086580 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1e201b08c0 con 0x559faf086ea0 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e2aa70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e201afb20 con 0x559faf091570 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e2aa70740 10 client.0 _ll_register_callbacks cb 0x7f1e1c00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25b06640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086580 0x559faf0998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25b06640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086580 0x559faf0998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39736/0 (socket says 172.21.15.27:39736) 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25b06640 1 -- 172.21.15.27:0/472288276 learned_addr learned my addr 172.21.15.27:0/472288276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25305640 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e26307640 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 0x559faf08af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25305640 1 -- 172.21.15.27:0/472288276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 msgr2=0x559faf08af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25305640 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 0x559faf08af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25305640 1 -- 172.21.15.27:0/472288276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086580 msgr2=0x559faf0998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25305640 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086580 0x559faf0998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25305640 1 -- 172.21.15.27:0/472288276 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559faf08b520 con 0x559faf086ea0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.390+0000 7f1e25b06640 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086580 0x559faf0998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e25305640 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08aa20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x559faf082fc0 tx=0x7f1e10005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b304541b764d2744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e097fa640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e097fa640 1 -- 172.21.15.27:0/472288276 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1005f060 con 0x559faf086ea0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 1 -- 172.21.15.27:0/472288276 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e10005520 con 0x559faf086ea0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e097fa640 1 -- 172.21.15.27:0/472288276 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1e100047c0 con 0x559faf086ea0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e097fa640 1 -- 172.21.15.27:0/472288276 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1e10035050 con 0x559faf086ea0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 10 client.26474 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 1 -- 172.21.15.27:0/472288276 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559faf107730 con 0x559faf086ea0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e097fa640 1 -- 172.21.15.27:0/472288276 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e10005520 con 0x559faf086ea0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e097fa640 1 -- 172.21.15.27:0/472288276 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1e100503c0 con 0x559faf086ea0 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 10 client.26474 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:36.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 10 client.26474 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 20 client.26474 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x559faf107a10 0x559faf109e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 4 client.26474 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.391+0000 7f1e2aa70740 1 -- 172.21.15.27:0/472288276 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559faeff2f00 con 0x559faf107a10 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e25b06640 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x559faf107a10 0x559faf109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e25b06640 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x559faf107a10 0x559faf109e00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=396d1de46fd74fc1 server_cookie=fd7e8cfc88bdcca5 in_seq=0 out_seq=0 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e097fa640 10 client.26474 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e097fa640 1 -- 172.21.15.27:0/472288276 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559faeff2f00 con 0x559faf107a10 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e097fa640 10 client.26474 handle_command_reply: tid=0 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 client.26474 shutdown 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 -- 172.21.15.27:0/472288276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 msgr2=0x559faf08aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08aa20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x559faf082fc0 tx=0x7f1e10005e80 comp rx=0 tx=0).stop 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 -- 172.21.15.27:0/472288276 shutdown_connections 2024-04-25T01:33:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x559faf107a10 0x559faf109e00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559faf091570 0x559faf08af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x559faf086ea0 0x559faf08aa20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e097fa640 0 client.26474 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 --2- 172.21.15.27:0/472288276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559faf086580 0x559faf0998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 -- 172.21.15.27:0/472288276 >> 172.21.15.27:0/472288276 conn(0x559faf0828a0 msgr2=0x559faf087cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.392+0000 7f1e2aa70740 1 -- 172.21.15.27:0/472288276 shutdown_connections 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e2aa70740 1 -- 172.21.15.27:0/472288276 wait complete. 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e2aa70740 20 client.26474 trim_cache size 0 max 16384 2024-04-25T01:33:36.377 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:33:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 -- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1e0c0a2d00 msgr2=0x7f1e0c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1e0c0a2d00 0x7f1e0c0a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e18005fd0 tx=0x7f1e18004a40 comp rx=0 tx=0).stop 2024-04-25T01:33:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 -- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201029c0 msgr2=0x7f1e20100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201029c0 0x7f1e20100610 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00d950 tx=0x7f1e1c00de10 comp rx=0 tx=0).stop 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 -- 172.21.15.27:0/3816513859 shutdown_connections 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e2010bd00 0x7f1e20101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e201032e0 0x7f1e20100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1e0c0a2d00 0x7f1e0c0a51c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 --2- 172.21.15.27:0/3816513859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e201029c0 0x7f1e20100610 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 -- 172.21.15.27:0/3816513859 >> 172.21.15.27:0/3816513859 conn(0x7f1e20077740 msgr2=0x7f1e200fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 -- 172.21.15.27:0/3816513859 shutdown_connections 2024-04-25T01:33:36.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.393+0000 7f1e27d6a640 1 -- 172.21.15.27:0/3816513859 wait complete. 2024-04-25T01:33:36.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 -- 172.21.15.27:0/3380548882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 msgr2=0x7fed88102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 --2- 172.21.15.27:0/3380548882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 0x7fed88102da0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fed84009900 tx=0x7fed8402a510 comp rx=0 tx=0).stop 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 -- 172.21.15.27:0/3380548882 shutdown_connections 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 --2- 172.21.15.27:0/3380548882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed8810bd00 0x7fed8810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 --2- 172.21.15.27:0/3380548882 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed881032e0 0x7fed8810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 --2- 172.21.15.27:0/3380548882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 0x7fed88102da0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 -- 172.21.15.27:0/3380548882 >> 172.21.15.27:0/3380548882 conn(0x7fed88077740 msgr2=0x7fed88077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 -- 172.21.15.27:0/3380548882 shutdown_connections 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.509+0000 7fed905b7640 1 -- 172.21.15.27:0/3380548882 wait complete. 2024-04-25T01:33:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed905b7640 1 Processor -- start 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed905b7640 1 -- start start 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed905b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 0x7fed88198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed905b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed881032e0 0x7fed88198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed905b7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed8810bd00 0x7fed8819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed905b7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fed88111b00 con 0x7fed881029c0 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed905b7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fed88111980 con 0x7fed8810bd00 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed905b7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed88111c80 con 0x7fed881032e0 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed8e353640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 0x7fed88198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed8e353640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 0x7fed88198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39746/0 (socket says 172.21.15.27:39746) 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed8e353640 1 -- 172.21.15.27:0/2485171328 learned_addr learned my addr 172.21.15.27:0/2485171328 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed8db52640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed881032e0 0x7fed88198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.510+0000 7fed8eb54640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed8810bd00 0x7fed8819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed8db52640 1 -- 172.21.15.27:0/2485171328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed8810bd00 msgr2=0x7fed8819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed8db52640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed8810bd00 0x7fed8819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed8db52640 1 -- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 msgr2=0x7fed88198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed8db52640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 0x7fed88198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed8db52640 1 -- 172.21.15.27:0/2485171328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8819cff0 con 0x7fed881032e0 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed8e353640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 0x7fed88198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed8db52640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed881032e0 0x7fed88198540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fed7800e9f0 tx=0x7fed7800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed7f7fe640 1 -- 172.21.15.27:0/2485171328 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7800ccc0 con 0x7fed881032e0 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed905b7640 1 -- 172.21.15.27:0/2485171328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8819d2e0 con 0x7fed881032e0 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed7f7fe640 1 -- 172.21.15.27:0/2485171328 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed78010910 con 0x7fed881032e0 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed7f7fe640 1 -- 172.21.15.27:0/2485171328 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78018960 con 0x7fed881032e0 2024-04-25T01:33:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.511+0000 7fed905b7640 1 -- 172.21.15.27:0/2485171328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed881a4b70 con 0x7fed881032e0 2024-04-25T01:33:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.513+0000 7fed7f7fe640 1 -- 172.21.15.27:0/2485171328 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fed78010ac0 con 0x7fed881032e0 2024-04-25T01:33:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed933a4740 1 Processor -- start 2024-04-25T01:33:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed933a4740 1 -- start start 2024-04-25T01:33:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed933a4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011232960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed933a4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x56401123d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed933a4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x56401123f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed933a4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564011197670 con 0x56401123d570 2024-04-25T01:33:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed933a4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56401113f970 con 0x564011232ea0 2024-04-25T01:33:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed933a4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564011193630 con 0x564011232580 2024-04-25T01:33:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed7f7fe640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fed740a2d00 0x7fed740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed8e353640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x56401123d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed8eb54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011232960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed8db52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x56401123f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed8eb54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011232960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36156/0 (socket says 172.21.15.27:36156) 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed8e353640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x56401123d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:33580/0 (socket says 172.21.15.27:33580) 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed8eb54640 1 -- 172.21.15.27:0/2975789333 learned_addr learned my addr 172.21.15.27:0/2975789333 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed8eb54640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fed740a2d00 0x7fed740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.514+0000 7fed7f7fe640 1 -- 172.21.15.27:0/2485171328 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fed780cfbb0 con 0x7fed881032e0 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed8eb54640 1 -- 172.21.15.27:0/2975789333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 msgr2=0x56401123d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed8eb54640 1 --2- 172.21.15.27:0/2975789333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x56401123d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed8eb54640 1 -- 172.21.15.27:0/2975789333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 msgr2=0x56401123f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed8eb54640 1 --2- 172.21.15.27:0/2975789333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x56401123f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed8eb54640 1 -- 172.21.15.27:0/2975789333 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564011163400 con 0x564011232580 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed8eb54640 1 --2- 172.21.15.27:0/2975789333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011232960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fed80011da0 tx=0x7fed8000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aab86303b3fa25d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed8eb54640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fed740a2d00 0x7fed740a51c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fed8000fc90 tx=0x7fed8001e000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed7cbfe640 1 -- 172.21.15.27:0/2975789333 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80022070 con 0x564011232580 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed7cbfe640 1 -- 172.21.15.27:0/2975789333 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed80009dc0 con 0x564011232580 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.515+0000 7fed7cbfe640 1 -- 172.21.15.27:0/2975789333 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed800042b0 con 0x564011232580 2024-04-25T01:33:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 -- 172.21.15.27:0/2975789333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 msgr2=0x564011232960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 --2- 172.21.15.27:0/2975789333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011232960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fed80011da0 tx=0x7fed8000f790 comp rx=0 tx=0).stop 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 -- 172.21.15.27:0/2975789333 shutdown_connections 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 --2- 172.21.15.27:0/2975789333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x56401123f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 --2- 172.21.15.27:0/2975789333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x56401123d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 --2- 172.21.15.27:0/2975789333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011232960 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 -- 172.21.15.27:0/2975789333 >> 172.21.15.27:0/2975789333 conn(0x56401122e8a0 msgr2=0x56401122ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 -- 172.21.15.27:0/2975789333 shutdown_connections 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 -- 172.21.15.27:0/2975789333 wait complete. 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 -- start start 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.516+0000 7fed933a4740 1 Processor -- start 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed933a4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x56401123cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed933a4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401123a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed933a4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56401123d570 0x56401123a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed933a4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564011240300 con 0x564011232ea0 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed933a4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564011153050 con 0x56401123d570 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed933a4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564011240480 con 0x564011232580 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed933a4740 10 client.0 _ll_register_callbacks cb 0x564010eaabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401123a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401123a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39766/0 (socket says 172.21.15.27:39766) 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 -- 172.21.15.27:0/884609053 learned_addr learned my addr 172.21.15.27:0/884609053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8e353640 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x56401123cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8eb54640 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56401123d570 0x56401123a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 -- 172.21.15.27:0/884609053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 msgr2=0x56401123cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x56401123cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 -- 172.21.15.27:0/884609053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56401123d570 msgr2=0x56401123a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56401123d570 0x56401123a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 -- 172.21.15.27:0/884609053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564011124dc0 con 0x564011232ea0 2024-04-25T01:33:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed8db52640 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401123a290 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fed780ccfc0 tx=0x7fed78066e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7067a1c305bd46e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed71ffb640 1 -- 172.21.15.27:0/884609053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78026040 con 0x564011232ea0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed71ffb640 1 -- 172.21.15.27:0/884609053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fed78066a30 con 0x564011232ea0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.517+0000 7fed933a4740 1 -- 172.21.15.27:0/884609053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed80009dc0 con 0x564011232ea0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed71ffb640 1 -- 172.21.15.27:0/884609053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed780a2a80 con 0x564011232ea0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed933a4740 10 client.17415 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed933a4740 1 -- 172.21.15.27:0/884609053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564010eab3f0 con 0x564011232ea0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed71ffb640 1 -- 172.21.15.27:0/884609053 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed80009dc0 con 0x564011232ea0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed71ffb640 1 -- 172.21.15.27:0/884609053 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fed780a44c0 con 0x564011232ea0 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed933a4740 10 client.17415 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed933a4740 10 client.17415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed933a4740 20 client.17415 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed933a4740 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56401124d5b0 0x56401124f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed933a4740 4 client.17415 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:36.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed933a4740 1 -- 172.21.15.27:0/884609053 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640111a4bb0 con 0x56401124d5b0 2024-04-25T01:33:36.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.518+0000 7fed8e353640 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56401124d5b0 0x56401124f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.519+0000 7fed71ffb640 10 client.17415 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:36.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.519+0000 7fed8e353640 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56401124d5b0 0x56401124f9a0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8421f29890f67ca9 server_cookie=76679dcd3fb5879a in_seq=0 out_seq=0 2024-04-25T01:33:36.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.519+0000 7fed71ffb640 1 -- 172.21.15.27:0/884609053 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640111a4bb0 con 0x56401124d5b0 2024-04-25T01:33:36.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.519+0000 7fed71ffb640 10 client.17415 handle_command_reply: tid=0 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.519+0000 7fed933a4740 1 client.17415 shutdown 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.519+0000 7fed933a4740 1 -- 172.21.15.27:0/884609053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 msgr2=0x56401123a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.519+0000 7fed933a4740 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401123a290 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fed780ccfc0 tx=0x7fed78066e60 comp rx=0 tx=0).stop 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 1 -- 172.21.15.27:0/884609053 shutdown_connections 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56401124d5b0 0x56401124f9a0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56401123d570 0x56401123a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401123a290 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 1 --2- 172.21.15.27:0/884609053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x56401123cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 1 -- 172.21.15.27:0/884609053 >> 172.21.15.27:0/884609053 conn(0x56401122e8a0 msgr2=0x5640112346a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed71ffb640 0 client.17415 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 1 -- 172.21.15.27:0/884609053 shutdown_connections 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 1 -- 172.21.15.27:0/884609053 wait complete. 2024-04-25T01:33:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.520+0000 7fed933a4740 20 client.17415 trim_cache size 0 max 16384 2024-04-25T01:33:36.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.548+0000 7fed933a4740 1 Processor -- start 2024-04-25T01:33:36.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.548+0000 7fed933a4740 1 -- start start 2024-04-25T01:33:36.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.548+0000 7fed933a4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232580 0x5640112491c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.548+0000 7fed933a4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401122d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.548+0000 7fed933a4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56401123d570 0x56401123cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.548+0000 7fed933a4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fed881b14c0 con 0x564011232ea0 2024-04-25T01:33:36.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.548+0000 7fed933a4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fed881aed90 con 0x564011232580 2024-04-25T01:33:36.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.548+0000 7fed933a4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed881acc00 con 0x56401123d570 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8db52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401122d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8db52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401122d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39780/0 (socket says 172.21.15.27:39780) 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8e353640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232580 0x5640112491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8db52640 1 -- 172.21.15.27:0/3916662174 learned_addr learned my addr 172.21.15.27:0/3916662174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8eb54640 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56401123d570 0x56401123cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8eb54640 1 -- 172.21.15.27:0/3916662174 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232580 msgr2=0x5640112491c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8eb54640 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232580 0x5640112491c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8eb54640 1 -- 172.21.15.27:0/3916662174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 msgr2=0x56401122d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8eb54640 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401122d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8eb54640 1 -- 172.21.15.27:0/3916662174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56401123ad40 con 0x56401123d570 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8e353640 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232580 0x5640112491c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8db52640 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401122d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed8eb54640 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56401123d570 0x56401123cad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fed80006010 tx=0x7fed800097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c187070f09a80e56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed73fff640 1 -- 172.21.15.27:0/3916662174 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80022040 con 0x56401123d570 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed73fff640 1 -- 172.21.15.27:0/3916662174 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed80011910 con 0x56401123d570 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.549+0000 7fed73fff640 1 -- 172.21.15.27:0/3916662174 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80011050 con 0x56401123d570 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 -- 172.21.15.27:0/3916662174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56401123d570 msgr2=0x56401123cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56401123d570 0x56401123cad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fed80006010 tx=0x7fed800097d0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 -- 172.21.15.27:0/3916662174 shutdown_connections 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56401123d570 0x56401123cad0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564011232ea0 0x56401122d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 --2- 172.21.15.27:0/3916662174 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232580 0x5640112491c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 -- 172.21.15.27:0/3916662174 >> 172.21.15.27:0/3916662174 conn(0x56401122e8a0 msgr2=0x56401122d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 -- 172.21.15.27:0/3916662174 shutdown_connections 2024-04-25T01:33:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.550+0000 7fed933a4740 1 -- 172.21.15.27:0/3916662174 wait complete. 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 1 -- start start 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 1 Processor -- start 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011245a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x564011240df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x564011241330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fed881ad250 con 0x56401123d570 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fed881ae6c0 con 0x564011232ea0 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed881ad920 con 0x564011232580 2024-04-25T01:33:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed933a4740 10 client.0 _ll_register_callbacks cb 0x7fed78010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed8e353640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011245a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed8eb54640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x564011241330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed8e353640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011245a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.27:36178/0 (socket says 172.21.15.27:36178) 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.551+0000 7fed8e353640 1 -- 172.21.15.27:0/1342460839 learned_addr learned my addr 172.21.15.27:0/1342460839 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed8db52640 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x564011240df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed8e353640 1 -- 172.21.15.27:0/1342460839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 msgr2=0x564011240df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed8e353640 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x564011240df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed8e353640 1 -- 172.21.15.27:0/1342460839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 msgr2=0x564011241330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed8e353640 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x564011241330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed8e353640 1 -- 172.21.15.27:0/1342460839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564010f757d0 con 0x564011232580 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed8eb54640 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x564011241330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed8e353640 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011245a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fed84006010 tx=0x7fed84004350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e17d607456ac9b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed71ffb640 1 -- 172.21.15.27:0/1342460839 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84035040 con 0x564011232580 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed71ffb640 1 -- 172.21.15.27:0/1342460839 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fed84046dd0 con 0x564011232580 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed933a4740 1 -- 172.21.15.27:0/1342460839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed80011910 con 0x564011232580 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed71ffb640 1 -- 172.21.15.27:0/1342460839 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed84055050 con 0x564011232580 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed933a4740 10 client.26560 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.552+0000 7fed933a4740 1 -- 172.21.15.27:0/1342460839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5640112b3970 con 0x564011232580 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed71ffb640 1 -- 172.21.15.27:0/1342460839 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed80011910 con 0x564011232580 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed71ffb640 1 -- 172.21.15.27:0/1342460839 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fed84004760 con 0x564011232580 2024-04-25T01:33:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed933a4740 10 client.26560 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed933a4740 10 client.26560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed933a4740 20 client.26560 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed933a4740 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5640112b40c0 0x5640112b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed933a4740 4 client.26560 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed933a4740 1 -- 172.21.15.27:0/1342460839 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56401119ef00 con 0x5640112b40c0 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed8db52640 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5640112b40c0 0x5640112b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed8db52640 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5640112b40c0 0x5640112b64e0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25a6481e78f91e3c server_cookie=e50423ad5177940e in_seq=0 out_seq=0 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.553+0000 7fed71ffb640 10 client.26560 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed71ffb640 1 -- 172.21.15.27:0/1342460839 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56401119ef00 con 0x5640112b40c0 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed71ffb640 10 client.26560 handle_command_reply: tid=0 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 client.26560 shutdown 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 -- 172.21.15.27:0/1342460839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 msgr2=0x564011245a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011245a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fed84006010 tx=0x7fed84004350 comp rx=0 tx=0).stop 2024-04-25T01:33:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 -- 172.21.15.27:0/1342460839 shutdown_connections 2024-04-25T01:33:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5640112b40c0 0x5640112b64e0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56401123d570 0x564011241330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564011232ea0 0x564011240df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed71ffb640 0 client.26560 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 --2- 172.21.15.27:0/1342460839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564011232580 0x564011245a80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 -- 172.21.15.27:0/1342460839 >> 172.21.15.27:0/1342460839 conn(0x56401122e8a0 msgr2=0x56401122d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 -- 172.21.15.27:0/1342460839 shutdown_connections 2024-04-25T01:33:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 1 -- 172.21.15.27:0/1342460839 wait complete. 2024-04-25T01:33:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed933a4740 20 client.26560 trim_cache size 0 max 16384 2024-04-25T01:33:36.540 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:33:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 -- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fed740a2d00 msgr2=0x7fed740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fed740a2d00 0x7fed740a51c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fed8000fc90 tx=0x7fed8001e000 comp rx=0 tx=0).stop 2024-04-25T01:33:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 -- 172.21.15.27:0/2485171328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed881032e0 msgr2=0x7fed88198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed881032e0 0x7fed88198540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fed7800e9f0 tx=0x7fed7800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 -- 172.21.15.27:0/2485171328 shutdown_connections 2024-04-25T01:33:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fed8810bd00 0x7fed8819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed881032e0 0x7fed88198540 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fed740a2d00 0x7fed740a51c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 --2- 172.21.15.27:0/2485171328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed881029c0 0x7fed88198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 -- 172.21.15.27:0/2485171328 >> 172.21.15.27:0/2485171328 conn(0x7fed88077740 msgr2=0x7fed880fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.554+0000 7fed905b7640 1 -- 172.21.15.27:0/2485171328 shutdown_connections 2024-04-25T01:33:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:36.555+0000 7fed905b7640 1 -- 172.21.15.27:0/2485171328 wait complete. 2024-04-25T01:33:36.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:36 smithi027 ceph-mon[29197]: pgmap v252: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:33:36.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:36 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3977475478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:36.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:36 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:36.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:36 smithi153 ceph-mon[32621]: pgmap v252: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:33:36.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:36 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3977475478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:36.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:36 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:36.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:36 smithi139 ceph-mon[32760]: pgmap v252: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2024-04-25T01:33:36.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:36 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3977475478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:36.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:36 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:37.544 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:33:37.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:33:37.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:33:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 -- 172.21.15.27:0/1816372844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec06a6e0 msgr2=0x7fa6ec10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1816372844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec06a6e0 0x7fa6ec10dc10 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009900 tx=0x7fa6d402a410 comp rx=0 tx=0).stop 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 -- 172.21.15.27:0/1816372844 shutdown_connections 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1816372844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6ec10e150 0x7fa6ec110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1816372844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec06a6e0 0x7fa6ec10dc10 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1816372844 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6ec069dc0 0x7fa6ec06a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 -- 172.21.15.27:0/1816372844 >> 172.21.15.27:0/1816372844 conn(0x7fa6ec0776a0 msgr2=0x7fa6ec077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 -- 172.21.15.27:0/1816372844 shutdown_connections 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 -- 172.21.15.27:0/1816372844 wait complete. 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 Processor -- start 2024-04-25T01:33:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.685+0000 7fa6f267d640 1 -- start start 2024-04-25T01:33:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f267d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6ec069dc0 0x7fa6ec19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f267d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6ec06a6e0 0x7fa6ec19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f267d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec10e150 0x7fa6ec19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f267d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec113d10 con 0x7fa6ec10e150 2024-04-25T01:33:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f267d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec113b90 con 0x7fa6ec06a6e0 2024-04-25T01:33:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f267d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec113e90 con 0x7fa6ec069dc0 2024-04-25T01:33:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec10e150 0x7fa6ec19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec10e150 0x7fa6ec19eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39808/0 (socket says 172.21.15.27:39808) 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 -- 172.21.15.27:0/1514556402 learned_addr learned my addr 172.21.15.27:0/1514556402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6ec06a6e0 0x7fa6ec19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6ec069dc0 0x7fa6ec19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 -- 172.21.15.27:0/1514556402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6ec069dc0 msgr2=0x7fa6ec19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6ec069dc0 0x7fa6ec19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 -- 172.21.15.27:0/1514556402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6ec06a6e0 msgr2=0x7fa6ec19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6ec06a6e0 0x7fa6ec19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 -- 172.21.15.27:0/1514556402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ec19f220 con 0x7fa6ec10e150 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6f0c1a640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec10e150 0x7fa6ec19eaf0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400dac0 tx=0x7fa6e400df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.686+0000 7fa6e97fa640 1 -- 172.21.15.27:0/1514556402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4017070 con 0x7fa6ec10e150 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.687+0000 7fa6f267d640 1 -- 172.21.15.27:0/1514556402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ec19f510 con 0x7fa6ec10e150 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.687+0000 7fa6e97fa640 1 -- 172.21.15.27:0/1514556402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6e4003e50 con 0x7fa6ec10e150 2024-04-25T01:33:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.687+0000 7fa6e97fa640 1 -- 172.21.15.27:0/1514556402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e40106c0 con 0x7fa6ec10e150 2024-04-25T01:33:37.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.687+0000 7fa6f267d640 1 -- 172.21.15.27:0/1514556402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ec1a6da0 con 0x7fa6ec10e150 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.689+0000 7fa6e97fa640 1 -- 172.21.15.27:0/1514556402 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa6e4003970 con 0x7fa6ec10e150 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.689+0000 7fa6f538e740 1 Processor -- start 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.689+0000 7fa6f538e740 1 -- start start 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.689+0000 7fa6e97fa640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6d80a2d00 0x7fa6d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5dacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5db710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6f538e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556eca53f2c0 con 0x556eca5dc980 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6f538e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556eca4e7910 con 0x556eca5db290 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6f538e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556eca53b280 con 0x556eca5e6010 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5dacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6e97fa640 1 -- 172.21.15.27:0/1514556402 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fa6e40cf4c0 con 0x7fa6ec10e150 2024-04-25T01:33:37.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5dacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39812/0 (socket says 172.21.15.27:39812) 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 -- 172.21.15.27:0/2085284493 learned_addr learned my addr 172.21.15.27:0/2085284493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6d80a2d00 0x7fa6d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5db710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6f0c1a640 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 -- 172.21.15.27:0/2085284493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 msgr2=0x556eca5e8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 -- 172.21.15.27:0/2085284493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 msgr2=0x556eca5db710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5db710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 -- 172.21.15.27:0/2085284493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eca4f8fb0 con 0x556eca5dc980 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6f0c1a640 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6d80a2d00 0x7fa6d80a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009230 tx=0x7fa6d4005b50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5dacc0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009c80 tx=0x7fa6dc0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0033ed48a3a3a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.690+0000 7fa6e27fc640 1 -- 172.21.15.27:0/2085284493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc012450 con 0x556eca5dc980 2024-04-25T01:33:37.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6e27fc640 1 -- 172.21.15.27:0/2085284493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6dc012ad0 con 0x556eca5dc980 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6e27fc640 1 -- 172.21.15.27:0/2085284493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00fbf0 con 0x556eca5dc980 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 -- 172.21.15.27:0/2085284493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 msgr2=0x556eca5dacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5dacc0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009c80 tx=0x7fa6dc0117b0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 -- 172.21.15.27:0/2085284493 shutdown_connections 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5db710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2085284493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5dacc0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 -- 172.21.15.27:0/2085284493 >> 172.21.15.27:0/2085284493 conn(0x556eca5a3950 msgr2=0x556eca5d77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 -- 172.21.15.27:0/2085284493 shutdown_connections 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.691+0000 7fa6f538e740 1 -- 172.21.15.27:0/2085284493 wait complete. 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 1 -- start start 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 1 Processor -- start 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5e3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556eca5e8ea0 con 0x556eca5dc980 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556eca4fae90 con 0x556eca5db290 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556eca5e9020 con 0x556eca5e6010 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6f538e740 10 client.0 _ll_register_callbacks cb 0x556eca536970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5e3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5e3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39822/0 (socket says 172.21.15.27:39822) 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6eb7fe640 1 -- 172.21.15.27:0/1386239971 learned_addr learned my addr 172.21.15.27:0/1386239971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.692+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6eb7fe640 1 -- 172.21.15.27:0/1386239971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 msgr2=0x556eca5e3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6f0c1a640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6eb7fe640 1 -- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 msgr2=0x556eca5e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5e2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6eb7fe640 1 -- 172.21.15.27:0/1386239971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eca31b7e0 con 0x556eca5dc980 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5e2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6f0c1a640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5e3420 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4033000 tx=0x7fa6d40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a8a1141b2e2bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/1386239971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4035040 con 0x556eca5dc980 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/1386239971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fa6d4003800 con 0x556eca5dc980 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6f538e740 1 -- 172.21.15.27:0/1386239971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6dc012ad0 con 0x556eca5dc980 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/1386239971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6d4058050 con 0x556eca5dc980 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6f538e740 10 client.17433 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6f538e740 1 -- 172.21.15.27:0/1386239971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556eca37bf70 con 0x556eca5dc980 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.693+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/1386239971 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6dc012ad0 con 0x556eca5dc980 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/1386239971 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa6d404aa00 con 0x556eca5dc980 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6f538e740 10 client.17433 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6f538e740 10 client.17433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6f538e740 20 client.17433 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:37.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556eca5f6140 0x556eca5f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6f538e740 4 client.17433 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:37.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6f538e740 1 -- 172.21.15.27:0/1386239971 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556eca5135a0 con 0x556eca5f6140 2024-04-25T01:33:37.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556eca5f6140 0x556eca5f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556eca5f6140 0x556eca5f85a0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f004b9c84c7315cb server_cookie=8a236c3e0ea35d29 in_seq=0 out_seq=0 2024-04-25T01:33:37.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.694+0000 7fa6cb7fe640 10 client.17433 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/1386239971 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556eca5135a0 con 0x556eca5f6140 2024-04-25T01:33:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6cb7fe640 10 client.17433 handle_command_reply: tid=0 2024-04-25T01:33:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 client.17433 shutdown 2024-04-25T01:33:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 -- 172.21.15.27:0/1386239971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 msgr2=0x556eca5e3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5e3420 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4033000 tx=0x7fa6d40058d0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 -- 172.21.15.27:0/1386239971 shutdown_connections 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556eca5f6140 0x556eca5f85a0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5dc980 0x556eca5e3420 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1386239971 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5db290 0x556eca5e2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 -- 172.21.15.27:0/1386239971 >> 172.21.15.27:0/1386239971 conn(0x556eca5a3950 msgr2=0x556eca5dd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6cb7fe640 0 client.17433 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 -- 172.21.15.27:0/1386239971 shutdown_connections 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 1 -- 172.21.15.27:0/1386239971 wait complete. 2024-04-25T01:33:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.695+0000 7fa6f538e740 20 client.17433 trim_cache size 0 max 16384 2024-04-25T01:33:37.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6f538e740 1 Processor -- start 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6f538e740 1 -- start start 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5dc980 0x556eca5fde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5e6010 0x556eca5fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6f538e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b3710 con 0x556eca5db290 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6f538e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b0fe0 con 0x556eca5e6010 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6f538e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1aee50 con 0x556eca5dc980 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55356/0 (socket says 172.21.15.27:55356) 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6ebfff640 1 -- 172.21.15.27:0/1417924515 learned_addr learned my addr 172.21.15.27:0/1417924515 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6ebfff640 1 -- 172.21.15.27:0/1417924515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5dc980 msgr2=0x556eca5fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/1417924515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5dc980 0x556eca5fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6ebfff640 1 -- 172.21.15.27:0/1417924515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5e6010 msgr2=0x556eca5fe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/1417924515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5e6010 0x556eca5fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.713+0000 7fa6ebfff640 1 -- 172.21.15.27:0/1417924515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eca213170 con 0x556eca5db290 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/1417924515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5f1da0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000c00 tx=0x7fa6dc0058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214c780c41897cec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6e1ffb640 1 -- 172.21.15.27:0/1417924515 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc005040 con 0x556eca5db290 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6e1ffb640 1 -- 172.21.15.27:0/1417924515 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6dc00f7e0 con 0x556eca5db290 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6e1ffb640 1 -- 172.21.15.27:0/1417924515 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc003850 con 0x556eca5db290 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 -- 172.21.15.27:0/1417924515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 msgr2=0x556eca5f1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1417924515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5f1da0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000c00 tx=0x7fa6dc0058c0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 -- 172.21.15.27:0/1417924515 shutdown_connections 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1417924515 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5e6010 0x556eca5fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1417924515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5dc980 0x556eca5fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 --2- 172.21.15.27:0/1417924515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5f1da0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 -- 172.21.15.27:0/1417924515 >> 172.21.15.27:0/1417924515 conn(0x556eca5a3950 msgr2=0x556eca5e17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 -- 172.21.15.27:0/1417924515 shutdown_connections 2024-04-25T01:33:37.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.714+0000 7fa6f538e740 1 -- 172.21.15.27:0/1417924515 wait complete. 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 1 -- start start 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 1 Processor -- start 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5ee640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5dc980 0x556eca5e99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1af4a0 con 0x556eca5db290 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b0910 con 0x556eca5dc980 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1afb70 con 0x556eca5e6010 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6f538e740 10 client.0 _ll_register_callbacks cb 0x7fa6d4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5ee640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55372/0 (socket says 172.21.15.27:55372) 2024-04-25T01:33:37.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.715+0000 7fa6ebfff640 1 -- 172.21.15.27:0/2980422813 learned_addr learned my addr 172.21.15.27:0/2980422813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:37 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:37.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:37 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:37.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:37 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5dc980 0x556eca5e99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6f0c1a640 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6ebfff640 1 -- 172.21.15.27:0/2980422813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 msgr2=0x556eca5e9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6ebfff640 1 -- 172.21.15.27:0/2980422813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5dc980 msgr2=0x556eca5e99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5dc980 0x556eca5e99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6ebfff640 1 -- 172.21.15.27:0/2980422813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eca5ea700 con 0x556eca5db290 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6ebfff640 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5ee640 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009c80 tx=0x7fa6dc009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65915ee7bf9795d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/2980422813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00fba0 con 0x556eca5db290 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6f538e740 1 -- 172.21.15.27:0/2980422813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6dc00f7e0 con 0x556eca5db290 2024-04-25T01:33:37.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/2980422813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fa6dc00fd40 con 0x556eca5db290 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/2980422813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6dc0050b0 con 0x556eca5db290 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6f538e740 10 client.17439 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6f538e740 1 -- 172.21.15.27:0/2980422813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556eca65c6f0 con 0x556eca5db290 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.716+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/2980422813 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6dc00f7e0 con 0x556eca5db290 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/2980422813 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa6dc03d030 con 0x556eca5db290 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6f538e740 10 client.17439 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6f538e740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6f538e740 20 client.17439 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556eca65cc00 0x556eca65eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6f538e740 4 client.17439 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6f538e740 1 -- 172.21.15.27:0/2980422813 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556eca54c800 con 0x556eca65cc00 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556eca65cc00 0x556eca65eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6cb7fe640 10 client.17439 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.717+0000 7fa6eb7fe640 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556eca65cc00 0x556eca65eff0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95257fbd4d3749cf server_cookie=e0b6d4da83c90f4c in_seq=0 out_seq=0 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6cb7fe640 1 -- 172.21.15.27:0/2980422813 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556eca54c800 con 0x556eca65cc00 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6cb7fe640 10 client.17439 handle_command_reply: tid=0 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 client.17439 shutdown 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 -- 172.21.15.27:0/2980422813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 msgr2=0x556eca5ee640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5ee640 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009c80 tx=0x7fa6dc009790 comp rx=0 tx=0).stop 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 -- 172.21.15.27:0/2980422813 shutdown_connections 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556eca65cc00 0x556eca65eff0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556eca5e6010 0x556eca5e9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556eca5dc980 0x556eca5e99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6cb7fe640 0 client.17439 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 --2- 172.21.15.27:0/2980422813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556eca5db290 0x556eca5ee640 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.718+0000 7fa6f538e740 1 -- 172.21.15.27:0/2980422813 >> 172.21.15.27:0/2980422813 conn(0x556eca5a3950 msgr2=0x556eca5d60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f538e740 1 -- 172.21.15.27:0/2980422813 shutdown_connections 2024-04-25T01:33:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f538e740 1 -- 172.21.15.27:0/2980422813 wait complete. 2024-04-25T01:33:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f538e740 20 client.17439 trim_cache size 0 max 16384 2024-04-25T01:33:37.701 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:33:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:33:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "ff5b681a-1e55-4712-a6c1-e293bfef6c80", 2024-04-25T01:33:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:33:37.701 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:33:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 -- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6d80a2d00 msgr2=0x7fa6d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6d80a2d00 0x7fa6d80a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009230 tx=0x7fa6d4005b50 comp rx=0 tx=0).stop 2024-04-25T01:33:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 -- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec10e150 msgr2=0x7fa6ec19eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec10e150 0x7fa6ec19eaf0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400dac0 tx=0x7fa6e400df80 comp rx=0 tx=0).stop 2024-04-25T01:33:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 -- 172.21.15.27:0/1514556402 shutdown_connections 2024-04-25T01:33:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6d80a2d00 0x7fa6d80a51c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec10e150 0x7fa6ec19eaf0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6ec06a6e0 0x7fa6ec19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 --2- 172.21.15.27:0/1514556402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6ec069dc0 0x7fa6ec19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 -- 172.21.15.27:0/1514556402 >> 172.21.15.27:0/1514556402 conn(0x7fa6ec0776a0 msgr2=0x7fa6ec10f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 -- 172.21.15.27:0/1514556402 shutdown_connections 2024-04-25T01:33:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.719+0000 7fa6f267d640 1 -- 172.21.15.27:0/1514556402 wait complete. 2024-04-25T01:33:37.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:33:37.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:33:37.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:37 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:37.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:37 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:37.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:37 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:33:37.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:37 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:37.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:37 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:37.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:37 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:33:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 -- 172.21.15.27:0/1181472435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540107730 msgr2=0x7f3540107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 --2- 172.21.15.27:0/1181472435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540107730 0x7f3540107b10 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f3530009930 tx=0x7f353002a420 comp rx=0 tx=0).stop 2024-04-25T01:33:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 -- 172.21.15.27:0/1181472435 shutdown_connections 2024-04-25T01:33:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 --2- 172.21.15.27:0/1181472435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f354006a780 0x7f354010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 --2- 172.21.15.27:0/1181472435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3540069dc0 0x7f354006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 --2- 172.21.15.27:0/1181472435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540107730 0x7f3540107b10 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 -- 172.21.15.27:0/1181472435 >> 172.21.15.27:0/1181472435 conn(0x7f35400776a0 msgr2=0x7f3540077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 -- 172.21.15.27:0/1181472435 shutdown_connections 2024-04-25T01:33:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.834+0000 7f3544b61640 1 -- 172.21.15.27:0/1181472435 wait complete. 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f3544b61640 1 Processor -- start 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f3544b61640 1 -- start start 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f3544b61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540069dc0 0x7f3540198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f3544b61640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f354006a780 0x7f35401985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f3544b61640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3540107730 0x7f354019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f3544b61640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3540111b40 con 0x7f3540069dc0 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f3544b61640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f35401119c0 con 0x7f354006a780 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f3544b61640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3540111cc0 con 0x7f3540107730 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540069dc0 0x7f3540198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540069dc0 0x7f3540198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55394/0 (socket says 172.21.15.27:55394) 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353e59c640 1 -- 172.21.15.27:0/3738364091 learned_addr learned my addr 172.21.15.27:0/3738364091 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353dd9b640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f354006a780 0x7f35401985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353e59c640 1 -- 172.21.15.27:0/3738364091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3540107730 msgr2=0x7f354019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353e59c640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3540107730 0x7f354019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353e59c640 1 -- 172.21.15.27:0/3738364091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f354006a780 msgr2=0x7f35401985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353e59c640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f354006a780 0x7f35401985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353e59c640 1 -- 172.21.15.27:0/3738364091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354019d020 con 0x7f3540069dc0 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.835+0000 7f353dd9b640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f354006a780 0x7f35401985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.836+0000 7f353e59c640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540069dc0 0x7f3540198080 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3530009230 tx=0x7f3530004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.836+0000 7f35377fe640 1 -- 172.21.15.27:0/3738364091 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353003f030 con 0x7f3540069dc0 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.836+0000 7f3544b61640 1 -- 172.21.15.27:0/3738364091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354019d2b0 con 0x7f3540069dc0 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.836+0000 7f35377fe640 1 -- 172.21.15.27:0/3738364091 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3530003ad0 con 0x7f3540069dc0 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.836+0000 7f35377fe640 1 -- 172.21.15.27:0/3738364091 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530035070 con 0x7f3540069dc0 2024-04-25T01:33:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.836+0000 7f3544b61640 1 -- 172.21.15.27:0/3738364091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35401a4ba0 con 0x7f3540069dc0 2024-04-25T01:33:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.838+0000 7f35377fe640 1 -- 172.21.15.27:0/3738364091 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3530042030 con 0x7f3540069dc0 2024-04-25T01:33:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f354786c740 1 Processor -- start 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f354786c740 1 -- start start 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f354786c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80b630 0x5589ca80ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f354786c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca8160e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f354786c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca818a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f354786c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5589ca7706b0 con 0x5589ca816620 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f354786c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5589ca718880 con 0x5589ca80bf50 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f354786c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5589ca76c670 con 0x5589ca80b630 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f35377fe640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f352c0a2d00 0x7f352c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f353dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca818a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f353ed9d640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f352c0a2d00 0x7f352c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f353e59c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca8160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f353ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80b630 0x5589ca80ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f353e59c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca8160e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.27:39690/0 (socket says 172.21.15.27:39690) 2024-04-25T01:33:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f353dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca818a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55402/0 (socket says 172.21.15.27:55402) 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.839+0000 7f353e59c640 1 -- 172.21.15.27:0/3063012064 learned_addr learned my addr 172.21.15.27:0/3063012064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f35377fe640 1 -- 172.21.15.27:0/3738364091 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f35300efe40 con 0x7f3540069dc0 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f353e59c640 1 -- 172.21.15.27:0/3063012064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80b630 msgr2=0x5589ca80ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f353e59c640 1 --2- 172.21.15.27:0/3063012064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80b630 0x5589ca80ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f353e59c640 1 -- 172.21.15.27:0/3063012064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 msgr2=0x5589ca818a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f353e59c640 1 --2- 172.21.15.27:0/3063012064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca818a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f353e59c640 1 -- 172.21.15.27:0/3063012064 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ca54e7d0 con 0x5589ca80bf50 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f353dd9b640 1 --2- 172.21.15.27:0/3063012064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca818a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f353ed9d640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f352c0a2d00 0x7f352c0a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3538009370 tx=0x7f35380097b0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.840+0000 7f353e59c640 1 --2- 172.21.15.27:0/3063012064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca8160e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35300c4650 tx=0x7f353004b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8774a7f1d9ab76cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f3534ff9640 1 -- 172.21.15.27:0/3063012064 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530049070 con 0x5589ca80bf50 2024-04-25T01:33:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f3534ff9640 1 -- 172.21.15.27:0/3063012064 <== mon.2 v2:172.21.15.139:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35300bbf10 con 0x5589ca80bf50 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f3534ff9640 1 -- 172.21.15.27:0/3063012064 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530003c80 con 0x5589ca80bf50 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 -- 172.21.15.27:0/3063012064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 msgr2=0x5589ca8160e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 --2- 172.21.15.27:0/3063012064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca8160e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35300c4650 tx=0x7f353004b000 comp rx=0 tx=0).stop 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 -- 172.21.15.27:0/3063012064 shutdown_connections 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 --2- 172.21.15.27:0/3063012064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca818a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 --2- 172.21.15.27:0/3063012064 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca8160e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 --2- 172.21.15.27:0/3063012064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80b630 0x5589ca80ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 -- 172.21.15.27:0/3063012064 >> 172.21.15.27:0/3063012064 conn(0x5589ca807950 msgr2=0x5589ca807d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 -- 172.21.15.27:0/3063012064 shutdown_connections 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.842+0000 7f354786c740 1 -- 172.21.15.27:0/3063012064 wait complete. 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 1 -- start start 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 1 Processor -- start 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca813580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca813ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca814000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5589ca8191d0 con 0x5589ca816620 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5589ca72bf50 con 0x5589ca80b630 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5589ca819350 con 0x5589ca80bf50 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f354786c740 10 client.0 _ll_register_callbacks cb 0x5589ca767000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f353ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca814000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f353ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca814000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55418/0 (socket says 172.21.15.27:55418) 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f353ed9d640 1 -- 172.21.15.27:0/2564815309 learned_addr learned my addr 172.21.15.27:0/2564815309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f353e59c640 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca813580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.843+0000 7f353dd9b640 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca813ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f353ed9d640 1 -- 172.21.15.27:0/2564815309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 msgr2=0x5589ca813ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f353ed9d640 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca813ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f353ed9d640 1 -- 172.21.15.27:0/2564815309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 msgr2=0x5589ca813580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f353ed9d640 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca813580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f353ed9d640 1 -- 172.21.15.27:0/2564815309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ca4843f0 con 0x5589ca816620 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f353dd9b640 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca813ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f353ed9d640 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca814000 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f353800fc40 tx=0x7f35380094b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc2ab7c3807ae43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f3525ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f3525ffb640 1 -- 172.21.15.27:0/2564815309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353801de90 con 0x5589ca816620 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f3525ffb640 1 -- 172.21.15.27:0/2564815309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f3538028420 con 0x5589ca816620 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f354786c740 1 -- 172.21.15.27:0/2564815309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35300bbf10 con 0x5589ca816620 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f3525ffb640 1 -- 172.21.15.27:0/2564815309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3538028620 con 0x5589ca816620 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f354786c740 10 client.17451 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f354786c740 1 -- 172.21.15.27:0/2564815309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5589ca4440f0 con 0x5589ca816620 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f3525ffb640 1 -- 172.21.15.27:0/2564815309 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35300bbf10 con 0x5589ca816620 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f3525ffb640 1 -- 172.21.15.27:0/2564815309 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f353801e910 con 0x5589ca816620 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f354786c740 10 client.17451 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.844+0000 7f354786c740 10 client.17451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.845+0000 7f354786c740 20 client.17451 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.845+0000 7f354786c740 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5589ca826650 0x5589ca828a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.845+0000 7f354786c740 4 client.17451 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.845+0000 7f354786c740 1 -- 172.21.15.27:0/2564815309 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589ca77dbf0 con 0x5589ca826650 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.845+0000 7f353e59c640 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5589ca826650 0x5589ca828a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.845+0000 7f353e59c640 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5589ca826650 0x5589ca828a70 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27c40435ccd1440d server_cookie=27ec865f132220b6 in_seq=0 out_seq=0 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.845+0000 7f3525ffb640 10 client.17451 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f3525ffb640 1 -- 172.21.15.27:0/2564815309 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5589ca77dbf0 con 0x5589ca826650 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f3525ffb640 10 client.17451 handle_command_reply: tid=0 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 client.17451 shutdown 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 -- 172.21.15.27:0/2564815309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 msgr2=0x5589ca814000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca814000 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f353800fc40 tx=0x7f35380094b0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 -- 172.21.15.27:0/2564815309 shutdown_connections 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5589ca826650 0x5589ca828a70 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f3525ffb640 0 client.17451 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca814000 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca813ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 --2- 172.21.15.27:0/2564815309 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca813580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 -- 172.21.15.27:0/2564815309 >> 172.21.15.27:0/2564815309 conn(0x5589ca807950 msgr2=0x5589ca80d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 -- 172.21.15.27:0/2564815309 shutdown_connections 2024-04-25T01:33:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 1 -- 172.21.15.27:0/2564815309 wait complete. 2024-04-25T01:33:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.846+0000 7f354786c740 20 client.17451 trim_cache size 0 max 16384 2024-04-25T01:33:37.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.863+0000 7f354786c740 1 Processor -- start 2024-04-25T01:33:37.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.863+0000 7f354786c740 1 -- start start 2024-04-25T01:33:37.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f354786c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca80b630 0x5589ca815cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f354786c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca821ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f354786c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca816620 0x5589ca806970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f354786c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f35401b1690 con 0x5589ca80b630 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f354786c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f35401aef60 con 0x5589ca80bf50 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f354786c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35401acdd0 con 0x5589ca816620 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca80b630 0x5589ca815cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca80b630 0x5589ca815cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55428/0 (socket says 172.21.15.27:55428) 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 -- 172.21.15.27:0/1799620404 learned_addr learned my addr 172.21.15.27:0/1799620404 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353dd9b640 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca821ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353ed9d640 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca816620 0x5589ca806970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 -- 172.21.15.27:0/1799620404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca816620 msgr2=0x5589ca806970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca816620 0x5589ca806970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 -- 172.21.15.27:0/1799620404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 msgr2=0x5589ca821ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca821ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 -- 172.21.15.27:0/1799620404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ca828de0 con 0x5589ca80b630 2024-04-25T01:33:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353ed9d640 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca816620 0x5589ca806970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353dd9b640 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca821ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f353e59c640 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca80b630 0x5589ca815cb0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f35300b8370 tx=0x7f35300bbd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b5b5a1787b8fdc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f3527fff640 1 -- 172.21.15.27:0/1799620404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530049040 con 0x5589ca80b630 2024-04-25T01:33:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.864+0000 7f3527fff640 1 -- 172.21.15.27:0/1799620404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35300bbf10 con 0x5589ca80b630 2024-04-25T01:33:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.865+0000 7f3527fff640 1 -- 172.21.15.27:0/1799620404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530003e10 con 0x5589ca80b630 2024-04-25T01:33:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.865+0000 7f354786c740 1 -- 172.21.15.27:0/1799620404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca80b630 msgr2=0x5589ca815cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca80b630 0x5589ca815cb0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f35300b8370 tx=0x7f35300bbd90 comp rx=0 tx=0).stop 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 -- 172.21.15.27:0/1799620404 shutdown_connections 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca816620 0x5589ca806970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80bf50 0x5589ca821ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 --2- 172.21.15.27:0/1799620404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca80b630 0x5589ca815cb0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 -- 172.21.15.27:0/1799620404 >> 172.21.15.27:0/1799620404 conn(0x5589ca807950 msgr2=0x5589ca806260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 -- 172.21.15.27:0/1799620404 shutdown_connections 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 -- 172.21.15.27:0/1799620404 wait complete. 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 -- start start 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.866+0000 7f354786c740 1 Processor -- start 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f354786c740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca810620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f354786c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca810b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f354786c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca88bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f354786c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f35401ad420 con 0x5589ca816620 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f354786c740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f35401ae890 con 0x5589ca80b630 2024-04-25T01:33:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f354786c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35401adaf0 con 0x5589ca80bf50 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f354786c740 10 client.0 _ll_register_callbacks cb 0x7f3538009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca88bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca88bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55430/0 (socket says 172.21.15.27:55430) 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353ed9d640 1 -- 172.21.15.27:0/3515694112 learned_addr learned my addr 172.21.15.27:0/3515694112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353e59c640 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca810620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353dd9b640 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca810b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353ed9d640 1 -- 172.21.15.27:0/3515694112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 msgr2=0x5589ca810b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353ed9d640 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca810b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353ed9d640 1 -- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 msgr2=0x5589ca810620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353ed9d640 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca810620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353ed9d640 1 -- 172.21.15.27:0/3515694112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ca88c0d0 con 0x5589ca816620 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.867+0000 7f353e59c640 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca810620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f353ed9d640 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca88bb90 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f353801ef20 tx=0x7f353801b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59e4eb4820be529 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f3525ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f3525ffb640 1 -- 172.21.15.27:0/3515694112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538039020 con 0x5589ca816620 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f3525ffb640 1 -- 172.21.15.27:0/3515694112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f3538028420 con 0x5589ca816620 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 1 -- 172.21.15.27:0/3515694112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35300bbf10 con 0x5589ca816620 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f3525ffb640 1 -- 172.21.15.27:0/3515694112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3538044050 con 0x5589ca816620 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 10 client.17457 fetch_fsmap learned FSMap version 27 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 1 -- 172.21.15.27:0/3515694112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5589ca88d190 con 0x5589ca816620 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f3525ffb640 1 -- 172.21.15.27:0/3515694112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35300bbf10 con 0x5589ca816620 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f3525ffb640 1 -- 172.21.15.27:0/3515694112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f353801e780 con 0x5589ca816620 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 10 client.17457 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 10 client.17457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 20 client.17457 populate_metadata read hostname 'smithi027' 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5589ca826650 0x5589ca88f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 4 client.17457 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.868+0000 7f354786c740 1 -- 172.21.15.27:0/3515694112 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5589ca777f40 con 0x5589ca826650 2024-04-25T01:33:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.869+0000 7f353e59c640 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5589ca826650 0x5589ca88f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.869+0000 7f3525ffb640 10 client.17457 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.869+0000 7f353e59c640 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5589ca826650 0x5589ca88f910 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed1db66abef2ffb server_cookie=2e911638b45008e5 in_seq=0 out_seq=0 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.869+0000 7f3525ffb640 1 -- 172.21.15.27:0/3515694112 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5589ca777f40 con 0x5589ca826650 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.869+0000 7f3525ffb640 10 client.17457 handle_command_reply: tid=0 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.869+0000 7f354786c740 1 client.17457 shutdown 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 -- 172.21.15.27:0/3515694112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 msgr2=0x5589ca88bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca88bb90 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f353801ef20 tx=0x7f353801b5d0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 -- 172.21.15.27:0/3515694112 shutdown_connections 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5589ca826650 0x5589ca88f910 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589ca816620 0x5589ca88bb90 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f3525ffb640 0 client.17457 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5589ca80bf50 0x5589ca810b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 --2- 172.21.15.27:0/3515694112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5589ca80b630 0x5589ca810620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 -- 172.21.15.27:0/3515694112 >> 172.21.15.27:0/3515694112 conn(0x5589ca807950 msgr2=0x5589ca813dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 -- 172.21.15.27:0/3515694112 shutdown_connections 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 1 -- 172.21.15.27:0/3515694112 wait complete. 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f354786c740 20 client.17457 trim_cache size 0 max 16384 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stdout: "ff5b681a-1e55-4712-a6c1-e293bfef6c80": { 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "ff5b681a-1e55-4712-a6c1-e293bfef6c80", 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:33:37.852 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:33:37.853 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:33:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:33:37.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f3544b61640 1 -- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f352c0a2d00 msgr2=0x7f352c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f3544b61640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f352c0a2d00 0x7f352c0a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3538009370 tx=0x7f35380097b0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f3544b61640 1 -- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540069dc0 msgr2=0x7f3540198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f3544b61640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540069dc0 0x7f3540198080 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3530009230 tx=0x7f3530004270 comp rx=0 tx=0).stop 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f3544b61640 1 -- 172.21.15.27:0/3738364091 shutdown_connections 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f3544b61640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3540107730 0x7f354019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.870+0000 7f3544b61640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f354006a780 0x7f35401985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.871+0000 7f3544b61640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f352c0a2d00 0x7f352c0a51c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.871+0000 7f3544b61640 1 --2- 172.21.15.27:0/3738364091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3540069dc0 0x7f3540198080 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.871+0000 7f3544b61640 1 -- 172.21.15.27:0/3738364091 >> 172.21.15.27:0/3738364091 conn(0x7f35400776a0 msgr2=0x7f35400fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.871+0000 7f3544b61640 1 -- 172.21.15.27:0/3738364091 shutdown_connections 2024-04-25T01:33:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:33:37.871+0000 7f3544b61640 1 -- 172.21.15.27:0/3738364091 wait complete. 2024-04-25T01:33:37.857 INFO:tasks.cephfs.filesystem:scrub status for tag:ff5b681a-1e55-4712-a6c1-e293bfef6c80 - {'path': '/', 'tag': 'ff5b681a-1e55-4712-a6c1-e293bfef6c80', 'options': 'recursive,force'} 2024-04-25T01:33:38.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:38 smithi153 ceph-mon[32621]: pgmap v253: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:33:38.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:38 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:33:38.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:38 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:38.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:38 smithi139 ceph-mon[32760]: pgmap v253: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:33:38.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:38 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:33:38.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:38 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:38.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:38 smithi027 ceph-mon[29197]: pgmap v253: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:33:38.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:38 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:33:38.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:38 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:39 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:33:39.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:39 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:39 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:33:39.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:39 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:39 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:33:39.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:39 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:39 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:33:39.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:39 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:39 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:33:39.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:39 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:39 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:33:39.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:39 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:40.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:40 smithi153 ceph-mon[32621]: pgmap v254: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.3 KiB/s wr, 9 op/s 2024-04-25T01:33:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:40 smithi139 ceph-mon[32760]: pgmap v254: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.3 KiB/s wr, 9 op/s 2024-04-25T01:33:40.843 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:40 smithi027 ceph-mon[29197]: pgmap v254: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.3 KiB/s wr, 9 op/s 2024-04-25T01:33:42.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:42 smithi153 ceph-mon[32621]: pgmap v255: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:33:42.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:42 smithi139 ceph-mon[32760]: pgmap v255: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:33:42.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:42 smithi027 ceph-mon[29197]: pgmap v255: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:33:44.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:44 smithi153 ceph-mon[32621]: pgmap v256: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:33:44.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:44 smithi139 ceph-mon[32760]: pgmap v256: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:33:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:44 smithi027 ceph-mon[29197]: pgmap v256: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:33:46.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:46 smithi153 ceph-mon[32621]: pgmap v257: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:33:46.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:46 smithi139 ceph-mon[32760]: pgmap v257: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:33:46.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:46 smithi027 ceph-mon[29197]: pgmap v257: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:33:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:48 smithi153 ceph-mon[32621]: pgmap v258: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:33:48.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:48 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:33:48.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:48 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:33:48.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:48 smithi139 ceph-mon[32760]: pgmap v258: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:33:48.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:48 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:33:48.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:48 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:33:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:48 smithi027 ceph-mon[29197]: pgmap v258: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:33:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:48 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:33:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:48 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:33:50.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:50 smithi153 ceph-mon[32621]: pgmap v259: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 9 op/s 2024-04-25T01:33:50.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:50 smithi139 ceph-mon[32760]: pgmap v259: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 9 op/s 2024-04-25T01:33:50.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:50 smithi027 ceph-mon[29197]: pgmap v259: 129 pgs: 129 active+clean; 4.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 9 op/s 2024-04-25T01:33:52.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:52 smithi153 ceph-mon[32621]: pgmap v260: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:33:52.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:52 smithi139 ceph-mon[32760]: pgmap v260: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:33:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:52 smithi027 ceph-mon[29197]: pgmap v260: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:33:54.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:54 smithi153 ceph-mon[32621]: pgmap v261: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:33:54.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:54 smithi139 ceph-mon[32760]: pgmap v261: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:33:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:54 smithi027 ceph-mon[29197]: pgmap v261: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:33:56.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:56 smithi153 ceph-mon[32621]: pgmap v262: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:33:56.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:56 smithi139 ceph-mon[32760]: pgmap v262: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:33:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:56 smithi027 ceph-mon[29197]: pgmap v262: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:33:58.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:33:58 smithi153 ceph-mon[32621]: pgmap v263: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:33:58.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:33:58 smithi139 ceph-mon[32760]: pgmap v263: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:33:58.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:33:58 smithi027 ceph-mon[29197]: pgmap v263: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:34:00.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:00 smithi153 ceph-mon[32621]: pgmap v264: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:34:00.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:00 smithi139 ceph-mon[32760]: pgmap v264: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:34:00.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:00 smithi027 ceph-mon[29197]: pgmap v264: 129 pgs: 129 active+clean; 4.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.1 KiB/s wr, 9 op/s 2024-04-25T01:34:02.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:02 smithi153 ceph-mon[32621]: pgmap v265: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 31 op/s 2024-04-25T01:34:02.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:02 smithi139 ceph-mon[32760]: pgmap v265: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 31 op/s 2024-04-25T01:34:02.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:02 smithi027 ceph-mon[29197]: pgmap v265: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 31 op/s 2024-04-25T01:34:04.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:04 smithi153 ceph-mon[32621]: pgmap v266: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:34:04.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:04 smithi139 ceph-mon[32760]: pgmap v266: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:34:04.819 INFO:tasks.workunit.client.0.smithi027.stdout:375/600: open fd = 3 2024-04-25T01:34:04.819 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.819 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.819 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.819 INFO:tasks.workunit.client.0.smithi027.stdout:376/600: open fd = 3 2024-04-25T01:34:04.819 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:377/600: open fd = 3 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:378/600: open fd = 3 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:379/600: open fd = 3 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.820 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:380/600: open fd = 3 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:381/600: open fd = 3 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:382/600: open fd = 3 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:383/600: open fd = 3 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.821 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:384/600: open fd = 3 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:385/600: open fd = 3 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:386/600: open fd = 3 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:387/600: open fd = 3 2024-04-25T01:34:04.822 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:388/600: open fd = 3 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:389/600: open fd = 3 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:390/600: open fd = 3 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.823 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:391/600: open fd = 3 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:392/600: open fd = 3 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:393/600: open fd = 3 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:394/600: open fd = 3 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.824 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:395/600: open fd = 3 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:396/600: open fd = 3 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:397/600: open fd = 3 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:398/600: open fd = 3 2024-04-25T01:34:04.825 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:399/600: open fd = 3 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:400/600: open fd = 3 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:401/600: open fd = 3 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.826 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:402/600: open fd = 3 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:403/600: open fd = 3 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:404/600: open fd = 3 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:405/600: open fd = 3 2024-04-25T01:34:04.827 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:406/600: open fd = 3 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:407/600: open fd = 3 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:408/600: open fd = 3 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.828 INFO:tasks.workunit.client.0.smithi027.stdout:409/600: open fd = 3 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:410/600: open fd = 3 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:411/600: open fd = 3 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:412/600: open fd = 3 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.829 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:413/600: open fd = 3 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:414/600: open fd = 3 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:415/600: open fd = 3 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:416/600: open fd = 3 2024-04-25T01:34:04.830 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:417/600: open fd = 3 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:418/600: open fd = 3 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:419/600: open fd = 3 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.831 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:420/600: open fd = 3 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:421/600: open fd = 3 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:422/600: open fd = 3 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:423/600: open fd = 3 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.832 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:424/600: open fd = 3 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:425/600: open fd = 3 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.833 INFO:tasks.workunit.client.0.smithi027.stdout:426/600: open fd = 3 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:427/600: open fd = 3 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:428/600: open fd = 3 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:429/600: open fd = 3 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.834 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:430/600: open fd = 3 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:431/600: open fd = 3 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:432/600: open fd = 3 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:433/600: open fd = 3 2024-04-25T01:34:04.835 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:434/600: open fd = 3 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:435/600: open fd = 3 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:436/600: open fd = 3 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:34:04.836 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:34:04.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:04 smithi027 ceph-mon[29197]: pgmap v266: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:34:06.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:06 smithi153 ceph-mon[32621]: pgmap v267: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:34:06.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:06 smithi139 ceph-mon[32760]: pgmap v267: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:34:06.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:06 smithi027 ceph-mon[29197]: pgmap v267: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:34:07.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:34:07.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:34:07.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 -- 172.21.15.27:0/1741368647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1032e0 msgr2=0x7fea0c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 --2- 172.21.15.27:0/1741368647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1032e0 0x7fea0c10b7c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fea00009900 tx=0x7fea0002a510 comp rx=0 tx=0).stop 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 -- 172.21.15.27:0/1741368647 shutdown_connections 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 --2- 172.21.15.27:0/1741368647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea0c10bd00 0x7fea0c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 --2- 172.21.15.27:0/1741368647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1032e0 0x7fea0c10b7c0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 --2- 172.21.15.27:0/1741368647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea0c1029c0 0x7fea0c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 -- 172.21.15.27:0/1741368647 >> 172.21.15.27:0/1741368647 conn(0x7fea0c077740 msgr2=0x7fea0c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 -- 172.21.15.27:0/1741368647 shutdown_connections 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 -- 172.21.15.27:0/1741368647 wait complete. 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 Processor -- start 2024-04-25T01:34:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.997+0000 7fea129c8640 1 -- start start 2024-04-25T01:34:07.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea129c8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1029c0 0x7fea0c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea129c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea0c1032e0 0x7fea0c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea129c8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea0c10bd00 0x7fea0c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea129c8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c111b00 con 0x7fea0c1029c0 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea129c8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c111980 con 0x7fea0c10bd00 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea129c8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c111c80 con 0x7fea0c1032e0 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1029c0 0x7fea0c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1029c0 0x7fea0c19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52752/0 (socket says 172.21.15.27:52752) 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 -- 172.21.15.27:0/1675155768 learned_addr learned my addr 172.21.15.27:0/1675155768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea10f65640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea0c10bd00 0x7fea0c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea0c1032e0 0x7fea0c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 -- 172.21.15.27:0/1675155768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea0c1032e0 msgr2=0x7fea0c19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea0c1032e0 0x7fea0c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 -- 172.21.15.27:0/1675155768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea0c10bd00 msgr2=0x7fea0c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:07.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea0c10bd00 0x7fea0c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 -- 172.21.15.27:0/1675155768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0c19f210 con 0x7fea0c1029c0 2024-04-25T01:34:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea10f65640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea0c10bd00 0x7fea0c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.998+0000 7fea0bfff640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1029c0 0x7fea0c19a220 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc00d950 tx=0x7fe9fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.999+0000 7fea097fa640 1 -- 172.21.15.27:0/1675155768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc017070 con 0x7fea0c1029c0 2024-04-25T01:34:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.999+0000 7fea097fa640 1 -- 172.21.15.27:0/1675155768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9fc009c30 con 0x7fea0c1029c0 2024-04-25T01:34:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.999+0000 7fea129c8640 1 -- 172.21.15.27:0/1675155768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea0c19f500 con 0x7fea0c1029c0 2024-04-25T01:34:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.999+0000 7fea097fa640 1 -- 172.21.15.27:0/1675155768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc00f6b0 con 0x7fea0c1029c0 2024-04-25T01:34:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:07.999+0000 7fea129c8640 1 -- 172.21.15.27:0/1675155768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea0c1a6d90 con 0x7fea0c1029c0 2024-04-25T01:34:07.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.001+0000 7fea097fa640 1 -- 172.21.15.27:0/1675155768 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe9fc0109d0 con 0x7fea0c1029c0 2024-04-25T01:34:07.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.001+0000 7fea157b1740 1 Processor -- start 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.001+0000 7fea157b1740 1 -- start start 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.001+0000 7fea097fa640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea157b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 0x564a5e5faa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea157b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e6050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea157b1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e607a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea157b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a5e55f6b0 con 0x564a5e5faf50 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea157b1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564a5e507880 con 0x564a5e605620 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea157b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a5e55b670 con 0x564a5e5fa630 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea10f65640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e6050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea097fa640 1 -- 172.21.15.27:0/1675155768 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fe9fc0cf590 con 0x7fea0c1029c0 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e6050e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52762/0 (socket says 172.21.15.27:52762) 2024-04-25T01:34:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 -- 172.21.15.27:0/475115324 learned_addr learned my addr 172.21.15.27:0/475115324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0bfff640 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 0x564a5e5faa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea10f65640 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e607a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 -- 172.21.15.27:0/475115324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 msgr2=0x564a5e5faa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 0x564a5e5faa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 -- 172.21.15.27:0/475115324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 msgr2=0x564a5e607a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e607a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 -- 172.21.15.27:0/475115324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a5e33d7d0 con 0x564a5e5faf50 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea10f65640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fea04005cc0 tx=0x7fea04012a30 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e6050e0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fea00006010 tx=0x7fea00005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e789def57af5d31a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.002+0000 7fe9f27fc640 1 -- 172.21.15.27:0/475115324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0003f070 con 0x564a5e5faf50 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fe9f27fc640 1 -- 172.21.15.27:0/475115324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea00004030 con 0x564a5e5faf50 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fe9f27fc640 1 -- 172.21.15.27:0/475115324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00035070 con 0x564a5e5faf50 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 -- 172.21.15.27:0/475115324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 msgr2=0x564a5e6050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e6050e0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fea00006010 tx=0x7fea00005e80 comp rx=0 tx=0).stop 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 -- 172.21.15.27:0/475115324 shutdown_connections 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e607a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e6050e0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 --2- 172.21.15.27:0/475115324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 0x564a5e5faa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 -- 172.21.15.27:0/475115324 >> 172.21.15.27:0/475115324 conn(0x564a5e5f6950 msgr2=0x564a5e5f6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 -- 172.21.15.27:0/475115324 shutdown_connections 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.003+0000 7fea157b1740 1 -- 172.21.15.27:0/475115324 wait complete. 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 1 -- start start 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 1 Processor -- start 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5fa630 0x564a5e602540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5faf50 0x564a5e602a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 0x564a5e602fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a5e6083b0 con 0x564a5e5fa630 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564a5e51af50 con 0x564a5e5faf50 2024-04-25T01:34:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a5e608530 con 0x564a5e605620 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea157b1740 10 client.0 _ll_register_callbacks cb 0x564a5e556000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea0bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5fa630 0x564a5e602540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea0bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5fa630 0x564a5e602540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52768/0 (socket says 172.21.15.27:52768) 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea0bfff640 1 -- 172.21.15.27:0/4160074558 learned_addr learned my addr 172.21.15.27:0/4160074558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.004+0000 7fea10f65640 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 0x564a5e602fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5faf50 0x564a5e602a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea0bfff640 1 -- 172.21.15.27:0/4160074558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 msgr2=0x564a5e602fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea0bfff640 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 0x564a5e602fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea0bfff640 1 -- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5faf50 msgr2=0x564a5e602a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea0bfff640 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5faf50 0x564a5e602a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea0bfff640 1 -- 172.21.15.27:0/4160074558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a5e2733f0 con 0x564a5e5fa630 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5faf50 0x564a5e602a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea0bfff640 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5fa630 0x564a5e602540 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc098040 tx=0x7fe9fc09b290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15582302c564ff6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fe9eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/4160074558 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc020040 con 0x564a5e5fa630 2024-04-25T01:34:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/4160074558 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fe9fc0bcad0 con 0x564a5e5fa630 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 1 -- 172.21.15.27:0/4160074558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea00004030 con 0x564a5e5fa630 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/4160074558 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe9fc09b670 con 0x564a5e5fa630 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 10 client.17469 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 1 -- 172.21.15.27:0/4160074558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564a5e2330f0 con 0x564a5e5fa630 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/4160074558 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea00004030 con 0x564a5e5fa630 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/4160074558 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe9fc097ce0 con 0x564a5e5fa630 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 10 client.17469 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 10 client.17469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 20 client.17469 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a5e615800 0x564a5e617c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 4 client.17469 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.005+0000 7fea157b1740 1 -- 172.21.15.27:0/4160074558 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a5e56cbf0 con 0x564a5e615800 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.006+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a5e615800 0x564a5e617c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.007+0000 7fe9eb7fe640 10 client.17469 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.006+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a5e615800 0x564a5e617c70 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da0f4dc05d396b4f server_cookie=ca349bb1dee4775b in_seq=0 out_seq=0 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.007+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/4160074558 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a5e56cbf0 con 0x564a5e615800 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.007+0000 7fe9eb7fe640 10 client.17469 handle_command_reply: tid=0 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.007+0000 7fea157b1740 1 client.17469 shutdown 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 -- 172.21.15.27:0/4160074558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5fa630 msgr2=0x564a5e602540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5fa630 0x564a5e602540 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc098040 tx=0x7fe9fc09b290 comp rx=0 tx=0).stop 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 -- 172.21.15.27:0/4160074558 shutdown_connections 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a5e615800 0x564a5e617c70 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 0x564a5e602fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5faf50 0x564a5e602a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fe9eb7fe640 0 client.17469 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 --2- 172.21.15.27:0/4160074558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5fa630 0x564a5e602540 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 -- 172.21.15.27:0/4160074558 >> 172.21.15.27:0/4160074558 conn(0x564a5e5f6950 msgr2=0x564a5e5fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 -- 172.21.15.27:0/4160074558 shutdown_connections 2024-04-25T01:34:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 1 -- 172.21.15.27:0/4160074558 wait complete. 2024-04-25T01:34:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.008+0000 7fea157b1740 20 client.17469 trim_cache size 0 max 16384 2024-04-25T01:34:08.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.025+0000 7fea157b1740 1 Processor -- start 2024-04-25T01:34:08.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.025+0000 7fea157b1740 1 -- start start 2024-04-25T01:34:08.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea157b1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5fa630 0x564a5e6112a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea157b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e604c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea157b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 0x564a5e601570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea157b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c1b3820 con 0x564a5e5faf50 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea157b1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c1b10f0 con 0x564a5e5fa630 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea157b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c1aef60 con 0x564a5e605620 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e604c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e604c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52778/0 (socket says 172.21.15.27:52778) 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 -- 172.21.15.27:0/3095150785 learned_addr learned my addr 172.21.15.27:0/3095150785 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0bfff640 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5fa630 0x564a5e6112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 -- 172.21.15.27:0/3095150785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 msgr2=0x564a5e601570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea10f65640 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 0x564a5e601570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 0x564a5e601570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 -- 172.21.15.27:0/3095150785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5fa630 msgr2=0x564a5e6112a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5fa630 0x564a5e6112a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 -- 172.21.15.27:0/3095150785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a5e5fb8c0 con 0x564a5e5faf50 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0bfff640 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5fa630 0x564a5e6112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e604c00 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fea00000c00 tx=0x7fea0002a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39f1416b3470d9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fe9f1ffb640 1 -- 172.21.15.27:0/3095150785 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00035070 con 0x564a5e5faf50 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fe9f1ffb640 1 -- 172.21.15.27:0/3095150785 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea00032db0 con 0x564a5e5faf50 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.026+0000 7fe9f1ffb640 1 -- 172.21.15.27:0/3095150785 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0003f040 con 0x564a5e5faf50 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 -- 172.21.15.27:0/3095150785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 msgr2=0x564a5e604c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e604c00 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fea00000c00 tx=0x7fea0002a510 comp rx=0 tx=0).stop 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 -- 172.21.15.27:0/3095150785 shutdown_connections 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e605620 0x564a5e601570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e604c00 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 --2- 172.21.15.27:0/3095150785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e5fa630 0x564a5e6112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 -- 172.21.15.27:0/3095150785 >> 172.21.15.27:0/3095150785 conn(0x564a5e5f6950 msgr2=0x564a5e5f6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 -- 172.21.15.27:0/3095150785 shutdown_connections 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 -- 172.21.15.27:0/3095150785 wait complete. 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.027+0000 7fea157b1740 1 -- start start 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea157b1740 1 Processor -- start 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea157b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 0x564a5e60d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea157b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e608d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea157b1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e609270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea157b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c1af5b0 con 0x564a5e5faf50 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea157b1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c1b0a20 con 0x564a5e605620 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea157b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c1afc80 con 0x564a5e5fa630 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea157b1740 10 client.0 _ll_register_callbacks cb 0x7fe9fc09b530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e608d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e608d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52792/0 (socket says 172.21.15.27:52792) 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 -- 172.21.15.27:0/1879733348 learned_addr learned my addr 172.21.15.27:0/1879733348 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea10f65640 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e609270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0bfff640 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 0x564a5e60d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 -- 172.21.15.27:0/1879733348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 msgr2=0x564a5e60d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 0x564a5e60d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 -- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 msgr2=0x564a5e609270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e609270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 -- 172.21.15.27:0/1879733348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a5e6098b0 con 0x564a5e5faf50 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea10f65640 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e609270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fea0b7fe640 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e608d30 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fea00006010 tx=0x7fea0004c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e68ac7ada98277 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.028+0000 7fe9eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/1879733348 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00059060 con 0x564a5e5faf50 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 1 -- 172.21.15.27:0/1879733348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564a5e566f40 con 0x564a5e5faf50 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/1879733348 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fea0004c8c0 con 0x564a5e5faf50 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/1879733348 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fea00060050 con 0x564a5e5faf50 2024-04-25T01:34:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 10 client.17475 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 1 -- 172.21.15.27:0/1879733348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564a5e67bb80 con 0x564a5e5faf50 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/1879733348 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564a5e566f40 con 0x564a5e5faf50 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/1879733348 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fea00003bd0 con 0x564a5e5faf50 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 10 client.17475 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 10 client.17475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 20 client.17475 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a5e67beb0 0x564a5e67e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 4 client.17475 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.029+0000 7fea157b1740 1 -- 172.21.15.27:0/1879733348 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564a5e4e46f0 con 0x564a5e67beb0 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea0bfff640 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a5e67beb0 0x564a5e67e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea0bfff640 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a5e67beb0 0x564a5e67e2a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76dd430b954605fe server_cookie=5fc860ad39662380 in_seq=0 out_seq=0 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fe9eb7fe640 10 client.17475 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/1879733348 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564a5e4e46f0 con 0x564a5e67beb0 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fe9eb7fe640 10 client.17475 handle_command_reply: tid=0 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 client.17475 shutdown 2024-04-25T01:34:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 -- 172.21.15.27:0/1879733348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 msgr2=0x564a5e608d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e608d30 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fea00006010 tx=0x7fea0004c2e0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 -- 172.21.15.27:0/1879733348 shutdown_connections 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a5e67beb0 0x564a5e67e2a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a5e605620 0x564a5e609270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a5e5faf50 0x564a5e608d30 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fe9eb7fe640 0 client.17475 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 --2- 172.21.15.27:0/1879733348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a5e5fa630 0x564a5e60d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.030+0000 7fea157b1740 1 -- 172.21.15.27:0/1879733348 >> 172.21.15.27:0/1879733348 conn(0x564a5e5f6950 msgr2=0x564a5e602d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea157b1740 1 -- 172.21.15.27:0/1879733348 shutdown_connections 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea157b1740 1 -- 172.21.15.27:0/1879733348 wait complete. 2024-04-25T01:34:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea157b1740 20 client.17475 trim_cache size 0 max 16384 2024-04-25T01:34:08.015 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:34:08.015 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:34:08.015 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:34:08.015 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 -- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9f40a2d00 msgr2=0x7fe9f40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fea04005cc0 tx=0x7fea04012a30 comp rx=0 tx=0).stop 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 -- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1029c0 msgr2=0x7fea0c19a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1029c0 0x7fea0c19a220 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc00d950 tx=0x7fe9fc00de10 comp rx=0 tx=0).stop 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 -- 172.21.15.27:0/1675155768 shutdown_connections 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fea0c10bd00 0x7fea0c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea0c1032e0 0x7fea0c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 --2- 172.21.15.27:0/1675155768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea0c1029c0 0x7fea0c19a220 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 -- 172.21.15.27:0/1675155768 >> 172.21.15.27:0/1675155768 conn(0x7fea0c077740 msgr2=0x7fea0c107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 -- 172.21.15.27:0/1675155768 shutdown_connections 2024-04-25T01:34:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.031+0000 7fea129c8640 1 -- 172.21.15.27:0/1675155768 wait complete. 2024-04-25T01:34:08.018 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:34:08.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:34:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 -- 172.21.15.27:0/2272795147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181010e0 msgr2=0x7fe8181055a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 --2- 172.21.15.27:0/2272795147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181010e0 0x7fe8181055a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fe808009900 tx=0x7fe80802a510 comp rx=0 tx=0).stop 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 -- 172.21.15.27:0/2272795147 shutdown_connections 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 --2- 172.21.15.27:0/2272795147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe818105ae0 0x7fe818105f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 --2- 172.21.15.27:0/2272795147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181010e0 0x7fe8181055a0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 --2- 172.21.15.27:0/2272795147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8181007c0 0x7fe818100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 -- 172.21.15.27:0/2272795147 >> 172.21.15.27:0/2272795147 conn(0x7fe818077740 msgr2=0x7fe818077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 -- 172.21.15.27:0/2272795147 shutdown_connections 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.144+0000 7fe82014f640 1 -- 172.21.15.27:0/2272795147 wait complete. 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe82014f640 1 Processor -- start 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe82014f640 1 -- start start 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe82014f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181007c0 0x7fe81819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe82014f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe8181010e0 0x7fe81819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe82014f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe818105ae0 0x7fe8181a2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe82014f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8181180e0 con 0x7fe8181007c0 2024-04-25T01:34:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe82014f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe818117f60 con 0x7fe8181010e0 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe82014f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe818118260 con 0x7fe818105ae0 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81deeb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181007c0 0x7fe81819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81deeb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181007c0 0x7fe81819e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52806/0 (socket says 172.21.15.27:52806) 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81deeb640 1 -- 172.21.15.27:0/2143538891 learned_addr learned my addr 172.21.15.27:0/2143538891 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81d6ea640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe8181010e0 0x7fe81819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81deeb640 1 -- 172.21.15.27:0/2143538891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe818105ae0 msgr2=0x7fe8181a2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81deeb640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe818105ae0 0x7fe8181a2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81deeb640 1 -- 172.21.15.27:0/2143538891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe8181010e0 msgr2=0x7fe81819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81deeb640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe8181010e0 0x7fe81819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81deeb640 1 -- 172.21.15.27:0/2143538891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8181a36b0 con 0x7fe8181007c0 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.145+0000 7fe81d6ea640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe8181010e0 0x7fe81819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.146+0000 7fe81deeb640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181007c0 0x7fe81819e6c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fe81400d950 tx=0x7fe81400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.146+0000 7fe80effd640 1 -- 172.21.15.27:0/2143538891 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814017030 con 0x7fe8181007c0 2024-04-25T01:34:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.146+0000 7fe80effd640 1 -- 172.21.15.27:0/2143538891 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe814004020 con 0x7fe8181007c0 2024-04-25T01:34:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.146+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8181a39a0 con 0x7fe8181007c0 2024-04-25T01:34:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.146+0000 7fe80effd640 1 -- 172.21.15.27:0/2143538891 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8140047a0 con 0x7fe8181007c0 2024-04-25T01:34:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.146+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8181ab230 con 0x7fe8181007c0 2024-04-25T01:34:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.147+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe818068560 con 0x7fe8181007c0 2024-04-25T01:34:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.148+0000 7fe80effd640 1 -- 172.21.15.27:0/2143538891 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe8140041d0 con 0x7fe8181007c0 2024-04-25T01:34:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.148+0000 7fe80effd640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe8040a2d80 0x7fe8040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.149+0000 7fe80effd640 1 -- 172.21.15.27:0/2143538891 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fe8140cf2b0 con 0x7fe8181007c0 2024-04-25T01:34:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.149+0000 7fe81d6ea640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe8040a2d80 0x7fe8040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.149+0000 7fe81d6ea640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe8040a2d80 0x7fe8040a5240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe8180fe870 tx=0x7fe808005a20 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.153+0000 7fe80effd640 1 -- 172.21.15.27:0/2143538891 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe81409ba90 con 0x7fe8181007c0 2024-04-25T01:34:08.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.381+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8181ab550 con 0x7fe8181007c0 2024-04-25T01:34:08.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.382+0000 7fe80effd640 1 -- 172.21.15.27:0/2143538891 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7fe81409b390 con 0x7fe8181007c0 2024-04-25T01:34:08.358 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:34:08.358 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:34:08.358 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:34:08.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe8040a2d80 msgr2=0x7fe8040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe8040a2d80 0x7fe8040a5240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe8180fe870 tx=0x7fe808005a20 comp rx=0 tx=0).stop 2024-04-25T01:34:08.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181007c0 msgr2=0x7fe81819e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181007c0 0x7fe81819e6c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fe81400d950 tx=0x7fe81400de10 comp rx=0 tx=0).stop 2024-04-25T01:34:08.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 shutdown_connections 2024-04-25T01:34:08.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe818105ae0 0x7fe8181a2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe8181010e0 0x7fe81819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe8040a2d80 0x7fe8040a5240 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 --2- 172.21.15.27:0/2143538891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8181007c0 0x7fe81819e6c0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 >> 172.21.15.27:0/2143538891 conn(0x7fe818077740 msgr2=0x7fe818103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 shutdown_connections 2024-04-25T01:34:08.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.384+0000 7fe82014f640 1 -- 172.21.15.27:0/2143538891 wait complete. 2024-04-25T01:34:08.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:34:08.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 -- 172.21.15.27:0/1027068524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100fe0 msgr2=0x7f52c410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 --2- 172.21.15.27:0/1027068524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100fe0 0x7f52c410d4f0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009cd0 tx=0x7f52bc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 -- 172.21.15.27:0/1027068524 shutdown_connections 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 --2- 172.21.15.27:0/1027068524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100fe0 0x7f52c410d4f0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 --2- 172.21.15.27:0/1027068524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c4100620 0x7f52c4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 --2- 172.21.15.27:0/1027068524 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52c4106630 0x7f52c4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 -- 172.21.15.27:0/1027068524 >> 172.21.15.27:0/1027068524 conn(0x7f52c40776a0 msgr2=0x7f52c4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 -- 172.21.15.27:0/1027068524 shutdown_connections 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.498+0000 7f52caedd640 1 -- 172.21.15.27:0/1027068524 wait complete. 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52caedd640 1 Processor -- start 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52caedd640 1 -- start start 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52caedd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100620 0x7f52c4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52caedd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c4100fe0 0x7f52c4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52caedd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52c4106630 0x7f52c419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52caedd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52c4110a40 con 0x7f52c4100620 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52caedd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41108c0 con 0x7f52c4106630 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52caedd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52c4110bc0 con 0x7f52c4100fe0 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c8c79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100620 0x7f52c4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c8c79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100620 0x7f52c4197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52818/0 (socket says 172.21.15.27:52818) 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c8c79640 1 -- 172.21.15.27:0/144123161 learned_addr learned my addr 172.21.15.27:0/144123161 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c3fff640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c4100fe0 0x7f52c4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c8c79640 1 -- 172.21.15.27:0/144123161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c4100fe0 msgr2=0x7f52c4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c8c79640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c4100fe0 0x7f52c4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c8c79640 1 -- 172.21.15.27:0/144123161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52c4106630 msgr2=0x7f52c419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c947a640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52c4106630 0x7f52c419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.499+0000 7f52c8c79640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52c4106630 0x7f52c419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.500+0000 7f52c8c79640 1 -- 172.21.15.27:0/144123161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c419cfe0 con 0x7f52c4100620 2024-04-25T01:34:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.500+0000 7f52c8c79640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100620 0x7f52c4197ff0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f52b400d950 tx=0x7f52b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.500+0000 7f52c1ffb640 1 -- 172.21.15.27:0/144123161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4017030 con 0x7f52c4100620 2024-04-25T01:34:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.500+0000 7f52c1ffb640 1 -- 172.21.15.27:0/144123161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b4004020 con 0x7f52c4100620 2024-04-25T01:34:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.500+0000 7f52caedd640 1 -- 172.21.15.27:0/144123161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c419d2d0 con 0x7f52c4100620 2024-04-25T01:34:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.500+0000 7f52c1ffb640 1 -- 172.21.15.27:0/144123161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b40047a0 con 0x7f52c4100620 2024-04-25T01:34:08.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.500+0000 7f52caedd640 1 -- 172.21.15.27:0/144123161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c41a4b60 con 0x7f52c4100620 2024-04-25T01:34:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.502+0000 7f52c1ffb640 1 -- 172.21.15.27:0/144123161 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f52b40041d0 con 0x7f52c4100620 2024-04-25T01:34:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52cdbea740 1 Processor -- start 2024-04-25T01:34:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52cdbea740 1 -- start start 2024-04-25T01:34:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c1ffb640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52ac0a2d00 0x7f52ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798580 0x558456798960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798ea0 0x5584567a3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584567a3570 0x5584567a5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52cdbea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5584566fd670 con 0x558456798580 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52cdbea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5584566a5970 con 0x5584567a3570 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52cdbea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5584566f9630 con 0x558456798ea0 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798580 0x558456798960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c947a640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52ac0a2d00 0x7f52ac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798580 0x558456798960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52832/0 (socket says 172.21.15.27:52832) 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 -- 172.21.15.27:0/2134205034 learned_addr learned my addr 172.21.15.27:0/2134205034 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c1ffb640 1 -- 172.21.15.27:0/144123161 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f52b40cf2a0 con 0x7f52c4100620 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c3fff640 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798ea0 0x5584567a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c947a640 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584567a3570 0x5584567a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 -- 172.21.15.27:0/2134205034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798ea0 msgr2=0x5584567a3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798ea0 0x5584567a3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 -- 172.21.15.27:0/2134205034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584567a3570 msgr2=0x5584567a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584567a3570 0x5584567a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 -- 172.21.15.27:0/2134205034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584566c9400 con 0x558456798580 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c947a640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52ac0a2d00 0x7f52ac0a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009ca0 tx=0x7f52bc031630 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.503+0000 7f52c8c79640 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798580 0x558456798960 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f52b409b310 tx=0x7f52b40976e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b090f782b8921586 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52aaffd640 1 -- 172.21.15.27:0/2134205034 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b409ace0 con 0x558456798580 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52aaffd640 1 -- 172.21.15.27:0/2134205034 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b409ba80 con 0x558456798580 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52aaffd640 1 -- 172.21.15.27:0/2134205034 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b40b4580 con 0x558456798580 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 -- 172.21.15.27:0/2134205034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798580 msgr2=0x558456798960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798580 0x558456798960 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f52b409b310 tx=0x7f52b40976e0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 -- 172.21.15.27:0/2134205034 shutdown_connections 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5584567a3570 0x5584567a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798ea0 0x5584567a3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 --2- 172.21.15.27:0/2134205034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798580 0x558456798960 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 -- 172.21.15.27:0/2134205034 >> 172.21.15.27:0/2134205034 conn(0x5584567948a0 msgr2=0x558456794c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 -- 172.21.15.27:0/2134205034 shutdown_connections 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.504+0000 7f52cdbea740 1 -- 172.21.15.27:0/2134205034 wait complete. 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 1 -- start start 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 1 Processor -- start 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 0x5584567a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567a0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567a0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5584567a62a0 con 0x558456798ea0 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5584566b9050 con 0x558456798580 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5584567a6420 con 0x5584567a3570 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52cdbea740 10 client.0 _ll_register_callbacks cb 0x558456410be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567a0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52840/0 (socket says 172.21.15.27:52840) 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.505+0000 7f52c3fff640 1 -- 172.21.15.27:0/25237252 learned_addr learned my addr 172.21.15.27:0/25237252 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c8c79640 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 0x5584567a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c947a640 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c3fff640 1 -- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 msgr2=0x5584567a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c3fff640 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c3fff640 1 -- 172.21.15.27:0/25237252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 msgr2=0x5584567a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c3fff640 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 0x5584567a2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c3fff640 1 -- 172.21.15.27:0/25237252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845668adc0 con 0x558456798ea0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c947a640 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52c3fff640 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567a0250 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f52b8011740 tx=0x7f52b8011c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a6d9d67bb61c73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52a3fff640 1 -- 172.21.15.27:0/25237252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b800fdb0 con 0x558456798ea0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52a3fff640 1 -- 172.21.15.27:0/25237252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f52b8004020 con 0x558456798ea0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52cdbea740 1 -- 172.21.15.27:0/25237252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52b409ba80 con 0x558456798ea0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52a3fff640 1 -- 172.21.15.27:0/25237252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f52b8024050 con 0x558456798ea0 2024-04-25T01:34:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52cdbea740 10 client.17493 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52cdbea740 1 -- 172.21.15.27:0/25237252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5584564113f0 con 0x558456798ea0 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.506+0000 7f52a3fff640 1 -- 172.21.15.27:0/25237252 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b409ba80 con 0x558456798ea0 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52a3fff640 1 -- 172.21.15.27:0/25237252 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f52b8018030 con 0x558456798ea0 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52cdbea740 10 client.17493 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52cdbea740 10 client.17493 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52cdbea740 20 client.17493 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52cdbea740 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5584567b3680 0x5584567b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52cdbea740 4 client.17493 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52cdbea740 1 -- 172.21.15.27:0/25237252 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55845670abb0 con 0x5584567b3680 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52c8c79640 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5584567b3680 0x5584567b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52c8c79640 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5584567b3680 0x5584567b5ac0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dab7295ef321034c server_cookie=411de7a1a02de293 in_seq=0 out_seq=0 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.507+0000 7f52a3fff640 10 client.17493 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52a3fff640 1 -- 172.21.15.27:0/25237252 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55845670abb0 con 0x5584567b3680 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52a3fff640 10 client.17493 handle_command_reply: tid=0 2024-04-25T01:34:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 client.17493 shutdown 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 -- 172.21.15.27:0/25237252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 msgr2=0x5584567a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567a0250 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f52b8011740 tx=0x7f52b8011c00 comp rx=0 tx=0).stop 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 -- 172.21.15.27:0/25237252 shutdown_connections 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x5584567b3680 0x5584567b5ac0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567a0250 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52a3fff640 0 client.17493 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 --2- 172.21.15.27:0/25237252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 0x5584567a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 -- 172.21.15.27:0/25237252 >> 172.21.15.27:0/25237252 conn(0x5584567948a0 msgr2=0x55845679a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 -- 172.21.15.27:0/25237252 shutdown_connections 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 1 -- 172.21.15.27:0/25237252 wait complete. 2024-04-25T01:34:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.508+0000 7f52cdbea740 20 client.17493 trim_cache size 0 max 16384 2024-04-25T01:34:08.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52cdbea740 1 Processor -- start 2024-04-25T01:34:08.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52cdbea740 1 -- start start 2024-04-25T01:34:08.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 0x5584567af160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567bb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567bb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52cdbea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41b17c0 con 0x558456798ea0 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52cdbea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41af090 con 0x558456798580 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52cdbea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41acf00 con 0x5584567a3570 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567bb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567bb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52844/0 (socket says 172.21.15.27:52844) 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 -- 172.21.15.27:0/896717456 learned_addr learned my addr 172.21.15.27:0/896717456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c8c79640 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 0x5584567af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c947a640 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567bb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 -- 172.21.15.27:0/896717456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 msgr2=0x5584567bb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567bb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 -- 172.21.15.27:0/896717456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 msgr2=0x5584567af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 0x5584567af160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 -- 172.21.15.27:0/896717456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584567a0bd0 con 0x558456798ea0 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c947a640 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567bb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52c3fff640 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567bb390 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f52b800cc40 tx=0x7f52b800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60178a88d2b38fc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.522+0000 7f52aa7fc640 1 -- 172.21.15.27:0/896717456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b8019860 con 0x558456798ea0 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52aa7fc640 1 -- 172.21.15.27:0/896717456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b8011e50 con 0x558456798ea0 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52aa7fc640 1 -- 172.21.15.27:0/896717456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b8004280 con 0x558456798ea0 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 -- 172.21.15.27:0/896717456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 msgr2=0x5584567bb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567bb390 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f52b800cc40 tx=0x7f52b800cf70 comp rx=0 tx=0).stop 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 -- 172.21.15.27:0/896717456 shutdown_connections 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584567a3570 0x5584567bb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558456798ea0 0x5584567bb390 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 --2- 172.21.15.27:0/896717456 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798580 0x5584567af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 -- 172.21.15.27:0/896717456 >> 172.21.15.27:0/896717456 conn(0x5584567948a0 msgr2=0x5584567932b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 -- 172.21.15.27:0/896717456 shutdown_connections 2024-04-25T01:34:08.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 -- 172.21.15.27:0/896717456 wait complete. 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 -- start start 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.523+0000 7f52cdbea740 1 Processor -- start 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798580 0x5584567a90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798ea0 0x5584567a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52cdbea740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584567a3570 0x5584567a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52cdbea740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41ad550 con 0x5584567a3570 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52cdbea740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41ae9c0 con 0x558456798ea0 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52cdbea740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41adc20 con 0x558456798580 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52cdbea740 10 client.0 _ll_register_callbacks cb 0x7f52b800c690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584567a3570 0x5584567a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584567a3570 0x5584567a9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52856/0 (socket says 172.21.15.27:52856) 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 -- 172.21.15.27:0/1557266233 learned_addr learned my addr 172.21.15.27:0/1557266233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c8c79640 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798580 0x5584567a90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c3fff640 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798ea0 0x5584567a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 -- 172.21.15.27:0/1557266233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798580 msgr2=0x5584567a90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798580 0x5584567a90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 -- 172.21.15.27:0/1557266233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798ea0 msgr2=0x5584567a95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798ea0 0x5584567a95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 -- 172.21.15.27:0/1557266233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584564db7d0 con 0x5584567a3570 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52c947a640 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584567a3570 0x5584567a9b30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f52bc02ae00 tx=0x7f52bc042040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a511a8a4406961ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.524+0000 7f52a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52a3fff640 1 -- 172.21.15.27:0/1557266233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc050030 con 0x5584567a3570 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52a3fff640 1 -- 172.21.15.27:0/1557266233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f52bc004790 con 0x5584567a3570 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 1 -- 172.21.15.27:0/1557266233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52b8011e50 con 0x5584567a3570 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52a3fff640 1 -- 172.21.15.27:0/1557266233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f52bc0305a0 con 0x5584567a3570 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 10 client.17499 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 1 -- 172.21.15.27:0/1557266233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558456819c50 con 0x5584567a3570 2024-04-25T01:34:08.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52a3fff640 1 -- 172.21.15.27:0/1557266233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b8011e50 con 0x5584567a3570 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52a3fff640 1 -- 172.21.15.27:0/1557266233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f52bc049050 con 0x5584567a3570 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 10 client.17499 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 10 client.17499 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 20 client.17499 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55845681a420 0x55845681c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 4 client.17499 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52cdbea740 1 -- 172.21.15.27:0/1557266233 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558456704f00 con 0x55845681a420 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.525+0000 7f52c8c79640 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55845681a420 0x55845681c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52a3fff640 10 client.17499 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52c8c79640 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55845681a420 0x55845681c840 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0934592672fdcc1 server_cookie=18c22f630661b6aa in_seq=0 out_seq=0 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52a3fff640 1 -- 172.21.15.27:0/1557266233 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558456704f00 con 0x55845681a420 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52a3fff640 10 client.17499 handle_command_reply: tid=0 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 client.17499 shutdown 2024-04-25T01:34:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 -- 172.21.15.27:0/1557266233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584567a3570 msgr2=0x5584567a9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584567a3570 0x5584567a9b30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f52bc02ae00 tx=0x7f52bc042040 comp rx=0 tx=0).stop 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 -- 172.21.15.27:0/1557266233 shutdown_connections 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55845681a420 0x55845681c840 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584567a3570 0x5584567a9b30 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558456798ea0 0x5584567a95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52a3fff640 0 client.17499 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 --2- 172.21.15.27:0/1557266233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558456798580 0x5584567a90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 -- 172.21.15.27:0/1557266233 >> 172.21.15.27:0/1557266233 conn(0x5584567948a0 msgr2=0x558456793b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 -- 172.21.15.27:0/1557266233 shutdown_connections 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 1 -- 172.21.15.27:0/1557266233 wait complete. 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.526+0000 7f52cdbea740 20 client.17499 trim_cache size 0 max 16384 2024-04-25T01:34:08.509 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:34:08.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 -- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52ac0a2d00 msgr2=0x7f52ac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52ac0a2d00 0x7f52ac0a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009ca0 tx=0x7f52bc031630 comp rx=0 tx=0).stop 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 -- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100620 msgr2=0x7f52c4197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100620 0x7f52c4197ff0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f52b400d950 tx=0x7f52b400de10 comp rx=0 tx=0).stop 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 -- 172.21.15.27:0/144123161 shutdown_connections 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52c4106630 0x7f52c419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c4100fe0 0x7f52c4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f52ac0a2d00 0x7f52ac0a51c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 --2- 172.21.15.27:0/144123161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c4100620 0x7f52c4197ff0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 -- 172.21.15.27:0/144123161 >> 172.21.15.27:0/144123161 conn(0x7f52c40776a0 msgr2=0x7f52c4069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 -- 172.21.15.27:0/144123161 shutdown_connections 2024-04-25T01:34:08.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.527+0000 7f52caedd640 1 -- 172.21.15.27:0/144123161 wait complete. 2024-04-25T01:34:08.514 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:34:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 -- 172.21.15.27:0/4115446955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 msgr2=0x7efe9410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 --2- 172.21.15.27:0/4115446955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 0x7efe9410e5f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009cd0 tx=0x7efe8c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 -- 172.21.15.27:0/4115446955 shutdown_connections 2024-04-25T01:34:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 --2- 172.21.15.27:0/4115446955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 0x7efe9410e5f0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 --2- 172.21.15.27:0/4115446955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94069dc0 0x7efe9406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 --2- 172.21.15.27:0/4115446955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe94107730 0x7efe94107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 -- 172.21.15.27:0/4115446955 >> 172.21.15.27:0/4115446955 conn(0x7efe940776a0 msgr2=0x7efe94077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 -- 172.21.15.27:0/4115446955 shutdown_connections 2024-04-25T01:34:08.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.642+0000 7efe98d16640 1 -- 172.21.15.27:0/4115446955 wait complete. 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe98d16640 1 Processor -- start 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe98d16640 1 -- start start 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe98d16640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94069dc0 0x7efe9419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe98d16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 0x7efe9419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe98d16640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe94107730 0x7efe9419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe98d16640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efe94111b90 con 0x7efe9406a780 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe98d16640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe94111a10 con 0x7efe94107730 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe98d16640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe94111d10 con 0x7efe94069dc0 2024-04-25T01:34:08.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 0x7efe9419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 0x7efe9419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52878/0 (socket says 172.21.15.27:52878) 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 -- 172.21.15.27:0/2453559006 learned_addr learned my addr 172.21.15.27:0/2453559006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe92d9d640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe94107730 0x7efe9419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe9259c640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94069dc0 0x7efe9419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 -- 172.21.15.27:0/2453559006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94069dc0 msgr2=0x7efe9419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94069dc0 0x7efe9419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 -- 172.21.15.27:0/2453559006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe94107730 msgr2=0x7efe9419eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe94107730 0x7efe9419eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 -- 172.21.15.27:0/2453559006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9419f240 con 0x7efe9406a780 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.643+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 0x7efe9419a790 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7efe8800d770 tx=0x7efe8800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.644+0000 7efe837fe640 1 -- 172.21.15.27:0/2453559006 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88017070 con 0x7efe9406a780 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.644+0000 7efe837fe640 1 -- 172.21.15.27:0/2453559006 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe88005500 con 0x7efe9406a780 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.644+0000 7efe98d16640 1 -- 172.21.15.27:0/2453559006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe9419f530 con 0x7efe9406a780 2024-04-25T01:34:08.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.644+0000 7efe837fe640 1 -- 172.21.15.27:0/2453559006 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88004e00 con 0x7efe9406a780 2024-04-25T01:34:08.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.644+0000 7efe98d16640 1 -- 172.21.15.27:0/2453559006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe941a6dc0 con 0x7efe9406a780 2024-04-25T01:34:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.646+0000 7efe837fe640 1 -- 172.21.15.27:0/2453559006 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efe88004fa0 con 0x7efe9406a780 2024-04-25T01:34:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.646+0000 7efe9ba1b740 1 Processor -- start 2024-04-25T01:34:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.646+0000 7efe9ba1b740 1 -- start start 2024-04-25T01:34:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a4e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a4eea0 0x557e38a59030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a59570 0x557e38a5b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557e389b3670 con 0x557e38a59570 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557e3895b970 con 0x557e38a4e580 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557e389af630 con 0x557e38a4eea0 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe837fe640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efe7c0a2d00 0x7efe7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a59570 0x557e38a5b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe92d9d640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efe7c0a2d00 0x7efe7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a59570 0x557e38a5b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52886/0 (socket says 172.21.15.27:52886) 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 -- 172.21.15.27:0/2020430177 learned_addr learned my addr 172.21.15.27:0/2020430177 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe9259c640 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a4eea0 0x557e38a59030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe92d9d640 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a4e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe837fe640 1 -- 172.21.15.27:0/2453559006 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7efe880cf3c0 con 0x7efe9406a780 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 -- 172.21.15.27:0/2020430177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a4eea0 msgr2=0x557e38a59030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a4eea0 0x557e38a59030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 -- 172.21.15.27:0/2020430177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 msgr2=0x557e38a4e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a4e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 -- 172.21.15.27:0/2020430177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e3897f400 con 0x557e38a59570 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe92d9d640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efe7c0a2d00 0x7efe7c0a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efe8c005e30 tx=0x7efe8c0099c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.647+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a59570 0x557e38a5b960 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7efe880978b0 tx=0x7efe8806e510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4cf166adc8ffa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe80ff9640 1 -- 172.21.15.27:0/2020430177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88020070 con 0x557e38a59570 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe80ff9640 1 -- 172.21.15.27:0/2020430177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe8809f850 con 0x557e38a59570 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe80ff9640 1 -- 172.21.15.27:0/2020430177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8806a0b0 con 0x557e38a59570 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2020430177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a59570 msgr2=0x557e38a5b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a59570 0x557e38a5b960 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7efe880978b0 tx=0x7efe8806e510 comp rx=0 tx=0).stop 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2020430177 shutdown_connections 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a59570 0x557e38a5b960 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a4eea0 0x557e38a59030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/2020430177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a4e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.648+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2020430177 >> 172.21.15.27:0/2020430177 conn(0x557e38a4a8a0 msgr2=0x557e38a4ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2020430177 shutdown_connections 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2020430177 wait complete. 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 -- start start 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 Processor -- start 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a58f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a56770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557e38a5c300 con 0x557e38a4eea0 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x557e3896f050 con 0x557e38a4e580 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557e38a5c480 con 0x557e38a59570 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9ba1b740 10 client.0 _ll_register_callbacks cb 0x557e386c6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52898/0 (socket says 172.21.15.27:52898) 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe91d9b640 1 -- 172.21.15.27:0/1283845287 learned_addr learned my addr 172.21.15.27:0/1283845287 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe9259c640 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a58f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.649+0000 7efe92d9d640 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a56770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe91d9b640 1 -- 172.21.15.27:0/1283845287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 msgr2=0x557e38a56770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe91d9b640 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a56770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe91d9b640 1 -- 172.21.15.27:0/1283845287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 msgr2=0x557e38a58f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe91d9b640 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a58f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe91d9b640 1 -- 172.21.15.27:0/1283845287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e38940dc0 con 0x557e38a4eea0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe91d9b640 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a561f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7efe8809b430 tx=0x7efe88009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df810aafa3eaf99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe79ffb640 1 -- 172.21.15.27:0/1283845287 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88020040 con 0x557e38a4eea0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe79ffb640 1 -- 172.21.15.27:0/1283845287 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7efe880cc950 con 0x557e38a4eea0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 1 -- 172.21.15.27:0/1283845287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe8809f850 con 0x557e38a4eea0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe79ffb640 1 -- 172.21.15.27:0/1283845287 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efe880a4110 con 0x557e38a4eea0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 10 client.17511 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 1 -- 172.21.15.27:0/1283845287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557e386c73f0 con 0x557e38a4eea0 2024-04-25T01:34:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe79ffb640 1 -- 172.21.15.27:0/1283845287 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe8809f850 con 0x557e38a4eea0 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe79ffb640 1 -- 172.21.15.27:0/1283845287 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7efe880b41d0 con 0x557e38a4eea0 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 10 client.17511 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 10 client.17511 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 20 client.17511 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x557e38a69440 0x557e38a6b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 4 client.17511 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.650+0000 7efe9ba1b740 1 -- 172.21.15.27:0/1283845287 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e389c0bb0 con 0x557e38a69440 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.651+0000 7efe9259c640 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x557e38a69440 0x557e38a6b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9259c640 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x557e38a69440 0x557e38a6b830 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7eea6f17269eedb9 server_cookie=f2fb89aaf42c17ce in_seq=0 out_seq=0 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe79ffb640 10 client.17511 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe79ffb640 1 -- 172.21.15.27:0/1283845287 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557e389c0bb0 con 0x557e38a69440 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe79ffb640 10 client.17511 handle_command_reply: tid=0 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 client.17511 shutdown 2024-04-25T01:34:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 -- 172.21.15.27:0/1283845287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 msgr2=0x557e38a561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a561f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7efe8809b430 tx=0x7efe88009c30 comp rx=0 tx=0).stop 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 -- 172.21.15.27:0/1283845287 shutdown_connections 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x557e38a69440 0x557e38a6b830 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a56770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe79ffb640 0 client.17511 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a561f0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/1283845287 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a58f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 -- 172.21.15.27:0/1283845287 >> 172.21.15.27:0/1283845287 conn(0x557e38a4a8a0 msgr2=0x557e38a49b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 -- 172.21.15.27:0/1283845287 shutdown_connections 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.652+0000 7efe9ba1b740 1 -- 172.21.15.27:0/1283845287 wait complete. 2024-04-25T01:34:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.653+0000 7efe9ba1b740 20 client.17511 trim_cache size 0 max 16384 2024-04-25T01:34:08.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9ba1b740 1 Processor -- start 2024-04-25T01:34:08.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9ba1b740 1 -- start start 2024-04-25T01:34:08.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a6e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a6eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a53450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efe941b3950 con 0x557e38a4eea0 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe941b1220 con 0x557e38a4e580 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe941af090 con 0x557e38a59570 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a6eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a6eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52908/0 (socket says 172.21.15.27:52908) 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe91d9b640 1 -- 172.21.15.27:0/2000943668 learned_addr learned my addr 172.21.15.27:0/2000943668 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe91d9b640 1 -- 172.21.15.27:0/2000943668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 msgr2=0x557e38a53450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2000943668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a53450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe91d9b640 1 -- 172.21.15.27:0/2000943668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 msgr2=0x557e38a6e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe9259c640 1 --2- 172.21.15.27:0/2000943668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a6e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2000943668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a6e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.666+0000 7efe91d9b640 1 -- 172.21.15.27:0/2000943668 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e387917d0 con 0x557e38a4eea0 2024-04-25T01:34:08.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe91d9b640 1 --2- 172.21.15.27:0/2000943668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a6eda0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efe880cc780 tx=0x7efe88009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e7f432290553b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe7bfff640 1 -- 172.21.15.27:0/2000943668 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88020040 con 0x557e38a4eea0 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe7bfff640 1 -- 172.21.15.27:0/2000943668 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe880cf170 con 0x557e38a4eea0 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe7bfff640 1 -- 172.21.15.27:0/2000943668 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8809aa10 con 0x557e38a4eea0 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2000943668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 msgr2=0x557e38a6eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/2000943668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a6eda0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efe880cc780 tx=0x7efe88009ed0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2000943668 shutdown_connections 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/2000943668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a53450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/2000943668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38a6eda0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/2000943668 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38a6e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2000943668 >> 172.21.15.27:0/2000943668 conn(0x557e38a4a8a0 msgr2=0x557e38a56310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2000943668 shutdown_connections 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.667+0000 7efe9ba1b740 1 -- 172.21.15.27:0/2000943668 wait complete. 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 1 -- start start 2024-04-25T01:34:08.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 1 Processor -- start 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38ac7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38ac8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a6e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efe941af6e0 con 0x557e38a4eea0 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efe941b0b50 con 0x557e38a4e580 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe941afdb0 con 0x557e38a59570 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe9ba1b740 10 client.0 _ll_register_callbacks cb 0x7efe880b4030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38ac8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38ac8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52924/0 (socket says 172.21.15.27:52924) 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 -- 172.21.15.27:0/3096759366 learned_addr learned my addr 172.21.15.27:0/3096759366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 -- 172.21.15.27:0/3096759366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 msgr2=0x557e38a6e0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a6e0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 -- 172.21.15.27:0/3096759366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 msgr2=0x557e38ac7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38ac7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 -- 172.21.15.27:0/3096759366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e38a6e7d0 con 0x557e38a4eea0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe91d9b640 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38ac8450 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7efe88097900 tx=0x7efe88004780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28a23c0ac3c8b0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.668+0000 7efe79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe79ffb640 1 -- 172.21.15.27:0/3096759366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88020070 con 0x557e38a4eea0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe79ffb640 1 -- 172.21.15.27:0/3096759366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7efe8809a9c0 con 0x557e38a4eea0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 1 -- 172.21.15.27:0/3096759366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe880cf170 con 0x557e38a4eea0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe79ffb640 1 -- 172.21.15.27:0/3096759366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efe8809f850 con 0x557e38a4eea0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 10 client.17517 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 1 -- 172.21.15.27:0/3096759366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557e38a6e7d0 con 0x557e38a4eea0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe79ffb640 1 -- 172.21.15.27:0/3096759366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe880cf170 con 0x557e38a4eea0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe79ffb640 1 -- 172.21.15.27:0/3096759366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7efe8809a360 con 0x557e38a4eea0 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 10 client.17517 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 10 client.17517 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 20 client.17517 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x557e38acf9f0 0x557e38ad1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 4 client.17517 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.669+0000 7efe9ba1b740 1 -- 172.21.15.27:0/3096759366 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557e389baf00 con 0x557e38acf9f0 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9259c640 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x557e38acf9f0 0x557e38ad1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe79ffb640 10 client.17517 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9259c640 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x557e38acf9f0 0x557e38ad1de0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e2d996e46b2004bb server_cookie=f16598e5449f6c97 in_seq=0 out_seq=0 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe79ffb640 1 -- 172.21.15.27:0/3096759366 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557e389baf00 con 0x557e38acf9f0 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe79ffb640 10 client.17517 handle_command_reply: tid=0 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 client.17517 shutdown 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 -- 172.21.15.27:0/3096759366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 msgr2=0x557e38ac8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38ac8450 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7efe88097900 tx=0x7efe88004780 comp rx=0 tx=0).stop 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 -- 172.21.15.27:0/3096759366 shutdown_connections 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x557e38acf9f0 0x557e38ad1de0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557e38a59570 0x557e38a6e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557e38a4eea0 0x557e38ac8450 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe79ffb640 0 client.17517 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:34:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 --2- 172.21.15.27:0/3096759366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x557e38a4e580 0x557e38ac7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 -- 172.21.15.27:0/3096759366 >> 172.21.15.27:0/3096759366 conn(0x557e38a4a8a0 msgr2=0x557e38a55a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 -- 172.21.15.27:0/3096759366 shutdown_connections 2024-04-25T01:34:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.670+0000 7efe9ba1b740 1 -- 172.21.15.27:0/3096759366 wait complete. 2024-04-25T01:34:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe9ba1b740 20 client.17517 trim_cache size 0 max 16384 2024-04-25T01:34:08.653 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:34:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 -- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efe7c0a2d00 msgr2=0x7efe7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efe7c0a2d00 0x7efe7c0a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efe8c005e30 tx=0x7efe8c0099c0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 -- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 msgr2=0x7efe9419a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 0x7efe9419a790 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7efe8800d770 tx=0x7efe8800dc30 comp rx=0 tx=0).stop 2024-04-25T01:34:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 -- 172.21.15.27:0/2453559006 shutdown_connections 2024-04-25T01:34:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efe94107730 0x7efe9419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7efe7c0a2d00 0x7efe7c0a51c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9406a780 0x7efe9419a790 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 --2- 172.21.15.27:0/2453559006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94069dc0 0x7efe9419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 -- 172.21.15.27:0/2453559006 >> 172.21.15.27:0/2453559006 conn(0x7efe940776a0 msgr2=0x7efe9410cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 -- 172.21.15.27:0/2453559006 shutdown_connections 2024-04-25T01:34:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.671+0000 7efe98d16640 1 -- 172.21.15.27:0/2453559006 wait complete. 2024-04-25T01:34:08.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:08 smithi027 ceph-mon[29197]: pgmap v268: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-04-25T01:34:08.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:08 smithi027 ceph-mon[29197]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T01:34:08.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:08 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2143538891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:08.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:34:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 -- 172.21.15.27:0/3785901023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c069dc0 msgr2=0x7fa62c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 --2- 172.21.15.27:0/3785901023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c069dc0 0x7fa62c06a240 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fa628009930 tx=0x7fa62802a420 comp rx=0 tx=0).stop 2024-04-25T01:34:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 -- 172.21.15.27:0/3785901023 shutdown_connections 2024-04-25T01:34:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 --2- 172.21.15.27:0/3785901023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62c06a780 0x7fa62c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 --2- 172.21.15.27:0/3785901023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c069dc0 0x7fa62c06a240 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 --2- 172.21.15.27:0/3785901023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa62c107730 0x7fa62c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 -- 172.21.15.27:0/3785901023 >> 172.21.15.27:0/3785901023 conn(0x7fa62c0776a0 msgr2=0x7fa62c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 -- 172.21.15.27:0/3785901023 shutdown_connections 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 -- 172.21.15.27:0/3785901023 wait complete. 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 Processor -- start 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.785+0000 7fa6347f9640 1 -- start start 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa6347f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62c069dc0 0x7fa62c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa6347f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa62c06a780 0x7fa62c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa6347f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c107730 0x7fa62c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa6347f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c111b90 con 0x7fa62c107730 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa6347f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c111a10 con 0x7fa62c06a780 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa6347f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c111d10 con 0x7fa62c069dc0 2024-04-25T01:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c107730 0x7fa62c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c107730 0x7fa62c19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52950/0 (socket says 172.21.15.27:52950) 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 -- 172.21.15.27:0/33511349 learned_addr learned my addr 172.21.15.27:0/33511349 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632595640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62c069dc0 0x7fa62c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa631d94640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa62c06a780 0x7fa62c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 -- 172.21.15.27:0/33511349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62c069dc0 msgr2=0x7fa62c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62c069dc0 0x7fa62c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 -- 172.21.15.27:0/33511349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa62c06a780 msgr2=0x7fa62c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa62c06a780 0x7fa62c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 -- 172.21.15.27:0/33511349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c19cf70 con 0x7fa62c107730 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa632d96640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c107730 0x7fa62c19c840 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fa62400eb10 tx=0x7fa62400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa61f7fe640 1 -- 172.21.15.27:0/33511349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624004b20 con 0x7fa62c107730 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa61f7fe640 1 -- 172.21.15.27:0/33511349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa624004cc0 con 0x7fa62c107730 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa6347f9640 1 -- 172.21.15.27:0/33511349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c19d260 con 0x7fa62c107730 2024-04-25T01:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa61f7fe640 1 -- 172.21.15.27:0/33511349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624010760 con 0x7fa62c107730 2024-04-25T01:34:08.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.786+0000 7fa6347f9640 1 -- 172.21.15.27:0/33511349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1a4af0 con 0x7fa62c107730 2024-04-25T01:34:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa61f7fe640 1 -- 172.21.15.27:0/33511349 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa624005020 con 0x7fa62c107730 2024-04-25T01:34:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa6375e6740 1 Processor -- start 2024-04-25T01:34:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa6375e6740 1 -- start start 2024-04-25T01:34:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 0x55ff3e316960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316ea0 0x55ff3e321030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e323960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa6375e6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e27b670 con 0x55ff3e321570 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa6375e6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e223970 con 0x55ff3e316ea0 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.789+0000 7fa6375e6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e277630 con 0x55ff3e316580 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e323960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e323960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52964/0 (socket says 172.21.15.27:52964) 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 -- 172.21.15.27:0/3902422476 learned_addr learned my addr 172.21.15.27:0/3902422476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa61f7fe640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6180a2d80 0x7fa6180a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 0x55ff3e316960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa632d96640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6180a2d80 0x7fa6180a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa632595640 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316ea0 0x55ff3e321030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 -- 172.21.15.27:0/3902422476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 msgr2=0x55ff3e316960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 0x55ff3e316960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 -- 172.21.15.27:0/3902422476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316ea0 msgr2=0x55ff3e321030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316ea0 0x55ff3e321030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 -- 172.21.15.27:0/3902422476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff3e247400 con 0x55ff3e321570 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa61f7fe640 1 -- 172.21.15.27:0/33511349 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fa6240d0020 con 0x7fa62c107730 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa631d94640 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e323960 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fa6280096c0 tx=0x7fa628005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3431dc3c9d5f8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa632d96640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6180a2d80 0x7fa6180a5240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa62409c090 tx=0x7fa62406b030 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa61cff9640 1 -- 172.21.15.27:0/3902422476 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628035040 con 0x55ff3e321570 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa61cff9640 1 -- 172.21.15.27:0/3902422476 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6280046c0 con 0x55ff3e321570 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.790+0000 7fa61cff9640 1 -- 172.21.15.27:0/3902422476 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6280038a0 con 0x55ff3e321570 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 -- 172.21.15.27:0/3902422476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 msgr2=0x55ff3e323960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e323960 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fa6280096c0 tx=0x7fa628005730 comp rx=0 tx=0).stop 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 -- 172.21.15.27:0/3902422476 shutdown_connections 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e323960 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316ea0 0x55ff3e321030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3902422476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 0x55ff3e316960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 -- 172.21.15.27:0/3902422476 >> 172.21.15.27:0/3902422476 conn(0x55ff3e3128a0 msgr2=0x55ff3e312c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 -- 172.21.15.27:0/3902422476 shutdown_connections 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 -- 172.21.15.27:0/3902422476 wait complete. 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.791+0000 7fa6375e6740 1 -- start start 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa6375e6740 1 Processor -- start 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 0x55ff3e320e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e31e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e31e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa6375e6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e324300 con 0x55ff3e321570 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa6375e6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e237050 con 0x55ff3e316580 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa6375e6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e324480 con 0x55ff3e316ea0 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa6375e6740 10 client.0 _ll_register_callbacks cb 0x55ff3df8ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e31e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e31e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52966/0 (socket says 172.21.15.27:52966) 2024-04-25T01:34:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 -- 172.21.15.27:0/1734970687 learned_addr learned my addr 172.21.15.27:0/1734970687 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 0x55ff3e320e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa632595640 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e31e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 -- 172.21.15.27:0/1734970687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 msgr2=0x55ff3e31e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e31e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 -- 172.21.15.27:0/1734970687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 msgr2=0x55ff3e320e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 0x55ff3e320e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 -- 172.21.15.27:0/1734970687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff3e208dc0 con 0x55ff3e321570 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa631d94640 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e31e6b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa628004e70 tx=0x7fa628004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25413d18a71fdc18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.792+0000 7fa615ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa615ffb640 1 -- 172.21.15.27:0/1734970687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628035040 con 0x55ff3e321570 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa615ffb640 1 -- 172.21.15.27:0/1734970687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fa6280038a0 con 0x55ff3e321570 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 1 -- 172.21.15.27:0/1734970687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6280046c0 con 0x55ff3e321570 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa615ffb640 1 -- 172.21.15.27:0/1734970687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa62805a050 con 0x55ff3e321570 2024-04-25T01:34:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 10 client.17529 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 1 -- 172.21.15.27:0/1734970687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ff3df8f3f0 con 0x55ff3e321570 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa615ffb640 1 -- 172.21.15.27:0/1734970687 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6280046c0 con 0x55ff3e321570 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa615ffb640 1 -- 172.21.15.27:0/1734970687 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa62803a040 con 0x55ff3e321570 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 10 client.17529 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 10 client.17529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 20 client.17529 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ff3e3313d0 0x55ff3e3337c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 4 client.17529 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa6375e6740 1 -- 172.21.15.27:0/1734970687 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff3e288bb0 con 0x55ff3e3313d0 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.793+0000 7fa632595640 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ff3e3313d0 0x55ff3e3337c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.794+0000 7fa615ffb640 10 client.17529 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.794+0000 7fa632595640 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ff3e3313d0 0x55ff3e3337c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fa6ea9c504fb1ec server_cookie=c7a419132e4755c3 in_seq=0 out_seq=0 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa615ffb640 1 -- 172.21.15.27:0/1734970687 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff3e288bb0 con 0x55ff3e3313d0 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa615ffb640 10 client.17529 handle_command_reply: tid=0 2024-04-25T01:34:08.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 client.17529 shutdown 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 -- 172.21.15.27:0/1734970687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 msgr2=0x55ff3e31e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e31e6b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa628004e70 tx=0x7fa628004610 comp rx=0 tx=0).stop 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 -- 172.21.15.27:0/1734970687 shutdown_connections 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ff3e3313d0 0x55ff3e3337c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e31e6b0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e31e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 --2- 172.21.15.27:0/1734970687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 0x55ff3e320e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa615ffb640 0 client.17529 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 -- 172.21.15.27:0/1734970687 >> 172.21.15.27:0/1734970687 conn(0x55ff3e3128a0 msgr2=0x55ff3e3186a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 -- 172.21.15.27:0/1734970687 shutdown_connections 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 1 -- 172.21.15.27:0/1734970687 wait complete. 2024-04-25T01:34:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.795+0000 7fa6375e6740 20 client.17529 trim_cache size 0 max 16384 2024-04-25T01:34:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:08 smithi153 ceph-mon[32621]: pgmap v268: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-04-25T01:34:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:08 smithi153 ceph-mon[32621]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T01:34:08.779 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:08 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2143538891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:08.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:08 smithi139 ceph-mon[32760]: pgmap v268: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-04-25T01:34:08.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:08 smithi139 ceph-mon[32760]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T01:34:08.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:08 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2143538891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa6375e6740 1 Processor -- start 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa6375e6740 1 -- start start 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 0x55ff3e3119c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e316ea0 0x55ff3e3209b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e321570 0x55ff3e31b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa6375e6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1b1710 con 0x55ff3e316ea0 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa6375e6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1aefe0 con 0x55ff3e321570 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa6375e6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1ace50 con 0x55ff3e316580 2024-04-25T01:34:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa632595640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e316ea0 0x55ff3e3209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa632595640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e316ea0 0x55ff3e3209b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52974/0 (socket says 172.21.15.27:52974) 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.808+0000 7fa632595640 1 -- 172.21.15.27:0/3469245476 learned_addr learned my addr 172.21.15.27:0/3469245476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa631d94640 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e321570 0x55ff3e31b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa632595640 1 -- 172.21.15.27:0/3469245476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 msgr2=0x55ff3e3119c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa632595640 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 0x55ff3e3119c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa632595640 1 -- 172.21.15.27:0/3469245476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e321570 msgr2=0x55ff3e31b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa632595640 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e321570 0x55ff3e31b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa632595640 1 -- 172.21.15.27:0/3469245476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff3e0597d0 con 0x55ff3e316ea0 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa631d94640 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e321570 0x55ff3e31b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa632595640 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e316ea0 0x55ff3e3209b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x55ff3e324520 tx=0x7fa620005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de00b8ac8acde12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa617fff640 1 -- 172.21.15.27:0/3469245476 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620003e40 con 0x55ff3e316ea0 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa617fff640 1 -- 172.21.15.27:0/3469245476 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa62001d530 con 0x55ff3e316ea0 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa617fff640 1 -- 172.21.15.27:0/3469245476 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620011a60 con 0x55ff3e316ea0 2024-04-25T01:34:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa6375e6740 1 -- 172.21.15.27:0/3469245476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e316ea0 msgr2=0x55ff3e3209b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.809+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e316ea0 0x55ff3e3209b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x55ff3e324520 tx=0x7fa620005ec0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 -- 172.21.15.27:0/3469245476 shutdown_connections 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e321570 0x55ff3e31b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e316ea0 0x55ff3e3209b0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3469245476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316580 0x55ff3e3119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 -- 172.21.15.27:0/3469245476 >> 172.21.15.27:0/3469245476 conn(0x55ff3e3128a0 msgr2=0x55ff3e3112b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 -- 172.21.15.27:0/3469245476 shutdown_connections 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 -- 172.21.15.27:0/3469245476 wait complete. 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 -- start start 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 Processor -- start 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 0x55ff3e394140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e394680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e38e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1ad4a0 con 0x55ff3e321570 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1ae910 con 0x55ff3e316580 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1adb70 con 0x55ff3e316ea0 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa6375e6740 10 client.0 _ll_register_callbacks cb 0x7fa628009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e38e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e38e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52978/0 (socket says 172.21.15.27:52978) 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 -- 172.21.15.27:0/3530935831 learned_addr learned my addr 172.21.15.27:0/3530935831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 0x55ff3e394140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 -- 172.21.15.27:0/3530935831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 msgr2=0x55ff3e394680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa632595640 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e394680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e394680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 -- 172.21.15.27:0/3530935831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 msgr2=0x55ff3e394140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 0x55ff3e394140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.810+0000 7fa631d94640 1 -- 172.21.15.27:0/3530935831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff3e38ec40 con 0x55ff3e321570 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa632595640 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e394680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa631d94640 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e38e550 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa628031bc0 tx=0x7fa628001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c0b702af5ce2f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa615ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa615ffb640 1 -- 172.21.15.27:0/3530935831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628035040 con 0x55ff3e321570 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa615ffb640 1 -- 172.21.15.27:0/3530935831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fa628031620 con 0x55ff3e321570 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 1 -- 172.21.15.27:0/3530935831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa62001d530 con 0x55ff3e321570 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa615ffb640 1 -- 172.21.15.27:0/3530935831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa628005d50 con 0x55ff3e321570 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 10 client.17535 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 1 -- 172.21.15.27:0/3530935831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ff3e39b8b0 con 0x55ff3e321570 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa615ffb640 1 -- 172.21.15.27:0/3530935831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa62001d530 con 0x55ff3e321570 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa615ffb640 1 -- 172.21.15.27:0/3530935831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa628003b80 con 0x55ff3e321570 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 10 client.17535 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 10 client.17535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 20 client.17535 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ff3e39bdc0 0x55ff3e39e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 4 client.17535 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.811+0000 7fa6375e6740 1 -- 172.21.15.27:0/3530935831 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff3e31ea40 con 0x55ff3e39bdc0 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.812+0000 7fa632595640 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ff3e39bdc0 0x55ff3e39e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.812+0000 7fa632595640 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ff3e39bdc0 0x55ff3e39e1e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c366443dd07e8b32 server_cookie=4f3cd89e8a711131 in_seq=0 out_seq=0 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.812+0000 7fa615ffb640 10 client.17535 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.812+0000 7fa615ffb640 1 -- 172.21.15.27:0/3530935831 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff3e31ea40 con 0x55ff3e39bdc0 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.812+0000 7fa615ffb640 10 client.17535 handle_command_reply: tid=0 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.812+0000 7fa6375e6740 1 client.17535 shutdown 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 -- 172.21.15.27:0/3530935831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 msgr2=0x55ff3e38e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e38e550 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa628031bc0 tx=0x7fa628001e50 comp rx=0 tx=0).stop 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 -- 172.21.15.27:0/3530935831 shutdown_connections 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ff3e39bdc0 0x55ff3e39e1e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ff3e321570 0x55ff3e38e550 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa615ffb640 0 client.17535 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ff3e316ea0 0x55ff3e394680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 --2- 172.21.15.27:0/3530935831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ff3e316580 0x55ff3e394140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 -- 172.21.15.27:0/3530935831 >> 172.21.15.27:0/3530935831 conn(0x55ff3e3128a0 msgr2=0x55ff3e311b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 -- 172.21.15.27:0/3530935831 shutdown_connections 2024-04-25T01:34:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 1 -- 172.21.15.27:0/3530935831 wait complete. 2024-04-25T01:34:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6375e6740 20 client.17535 trim_cache size 0 max 16384 2024-04-25T01:34:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:34:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 -- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6180a2d80 msgr2=0x7fa6180a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6180a2d80 0x7fa6180a5240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa62409c090 tx=0x7fa62406b030 comp rx=0 tx=0).stop 2024-04-25T01:34:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 -- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c107730 msgr2=0x7fa62c19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c107730 0x7fa62c19c840 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fa62400eb10 tx=0x7fa62400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 -- 172.21.15.27:0/33511349 shutdown_connections 2024-04-25T01:34:08.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6180a2d80 0x7fa6180a5240 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c107730 0x7fa62c19c840 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa62c06a780 0x7fa62c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 --2- 172.21.15.27:0/33511349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62c069dc0 0x7fa62c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:08.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 -- 172.21.15.27:0/33511349 >> 172.21.15.27:0/33511349 conn(0x7fa62c0776a0 msgr2=0x7fa62c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:08.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 -- 172.21.15.27:0/33511349 shutdown_connections 2024-04-25T01:34:08.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:08.813+0000 7fa6347f9640 1 -- 172.21.15.27:0/33511349 wait complete. 2024-04-25T01:34:09.800 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:34:09.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:34:09.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:34:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1276349002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda406a780 msgr2=0x7fbda410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1276349002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda406a780 0x7fbda410e5f0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009cd0 tx=0x7fbd9c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1276349002 shutdown_connections 2024-04-25T01:34:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1276349002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda406a780 0x7fbda410e5f0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1276349002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbda4069dc0 0x7fbda406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1276349002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda4107730 0x7fbda4107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1276349002 >> 172.21.15.27:0/1276349002 conn(0x7fbda40776a0 msgr2=0x7fbda4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1276349002 shutdown_connections 2024-04-25T01:34:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.945+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1276349002 wait complete. 2024-04-25T01:34:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbdaa9e3640 1 Processor -- start 2024-04-25T01:34:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbdaa9e3640 1 -- start start 2024-04-25T01:34:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbdaa9e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda4069dc0 0x7fbda4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbdaa9e3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda406a780 0x7fbda41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbdaa9e3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbda4107730 0x7fbda419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbdaa9e3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4111b40 con 0x7fbda4069dc0 2024-04-25T01:34:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbdaa9e3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41119c0 con 0x7fbda406a780 2024-04-25T01:34:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbdaa9e3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4111cc0 con 0x7fbda4107730 2024-04-25T01:34:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbda3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda4069dc0 0x7fbda4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbda3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda4069dc0 0x7fbda4197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52992/0 (socket says 172.21.15.27:52992) 2024-04-25T01:34:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.946+0000 7fbda3fff640 1 -- 172.21.15.27:0/1951804162 learned_addr learned my addr 172.21.15.27:0/1951804162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda8f80640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbda4107730 0x7fbda419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda37fe640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda406a780 0x7fbda41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda3fff640 1 -- 172.21.15.27:0/1951804162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbda4107730 msgr2=0x7fbda419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda3fff640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbda4107730 0x7fbda419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda3fff640 1 -- 172.21.15.27:0/1951804162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda406a780 msgr2=0x7fbda41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda3fff640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda406a780 0x7fbda41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda3fff640 1 -- 172.21.15.27:0/1951804162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda419cf90 con 0x7fbda4069dc0 2024-04-25T01:34:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda3fff640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda4069dc0 0x7fbda4197fa0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400e9f0 tx=0x7fbd9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda17fa640 1 -- 172.21.15.27:0/1951804162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9400cd80 con 0x7fbda4069dc0 2024-04-25T01:34:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda17fa640 1 -- 172.21.15.27:0/1951804162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd94010910 con 0x7fbda4069dc0 2024-04-25T01:34:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1951804162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda419d280 con 0x7fbda4069dc0 2024-04-25T01:34:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbda17fa640 1 -- 172.21.15.27:0/1951804162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd94018990 con 0x7fbda4069dc0 2024-04-25T01:34:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.947+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1951804162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda41a4b10 con 0x7fbda4069dc0 2024-04-25T01:34:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbda17fa640 1 -- 172.21.15.27:0/1951804162 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbd94010ac0 con 0x7fbda4069dc0 2024-04-25T01:34:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbdad7cd740 1 Processor -- start 2024-04-25T01:34:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbdad7cd740 1 -- start start 2024-04-25T01:34:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f97980 0x5623b9f95cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f96290 0x5623b9f96710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9fa1010 0x5623b9fa3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5623b9efa2c0 con 0x5623b9f97980 2024-04-25T01:34:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5623b9ea2910 con 0x5623b9fa1010 2024-04-25T01:34:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5623b9ef6280 con 0x5623b9f96290 2024-04-25T01:34:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbda8f80640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f97980 0x5623b9f95cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbda8f80640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f97980 0x5623b9f95cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53002/0 (socket says 172.21.15.27:53002) 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbda8f80640 1 -- 172.21.15.27:0/1719801549 learned_addr learned my addr 172.21.15.27:0/1719801549 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbda3fff640 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f96290 0x5623b9f96710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbda17fa640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd8c0a2d00 0x7fbd8c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.950+0000 7fbda37fe640 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9fa1010 0x5623b9fa3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda8f80640 1 -- 172.21.15.27:0/1719801549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f96290 msgr2=0x5623b9f96710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda8f80640 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f96290 0x5623b9f96710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda8f80640 1 -- 172.21.15.27:0/1719801549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9fa1010 msgr2=0x5623b9fa3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda8f80640 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9fa1010 0x5623b9fa3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda8f80640 1 -- 172.21.15.27:0/1719801549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623b9eb3fb0 con 0x5623b9f97980 2024-04-25T01:34:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda3fff640 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f96290 0x5623b9f96710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda8f80640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd8c0a2d00 0x7fbd8c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda17fa640 1 -- 172.21.15.27:0/1951804162 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fbd940cfc80 con 0x7fbda4069dc0 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbda8f80640 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f97980 0x5623b9f95cc0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009a60 tx=0x7fbd9c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a42efb49804aefa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbd8a7fc640 1 -- 172.21.15.27:0/1719801549 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0315b0 con 0x5623b9f97980 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbd8a7fc640 1 -- 172.21.15.27:0/1719801549 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd9c036040 con 0x5623b9f97980 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.951+0000 7fbd8a7fc640 1 -- 172.21.15.27:0/1719801549 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0307c0 con 0x5623b9f97980 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 -- 172.21.15.27:0/1719801549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f97980 msgr2=0x5623b9f95cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbda8f80640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd8c0a2d00 0x7fbd8c0a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009430 tx=0x7fbd9c048000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f97980 0x5623b9f95cc0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009a60 tx=0x7fbd9c005f70 comp rx=0 tx=0).stop 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 -- 172.21.15.27:0/1719801549 shutdown_connections 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9fa1010 0x5623b9fa3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f96290 0x5623b9f96710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/1719801549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f97980 0x5623b9f95cc0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 -- 172.21.15.27:0/1719801549 >> 172.21.15.27:0/1719801549 conn(0x5623b9f5e950 msgr2=0x5623b9f927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 -- 172.21.15.27:0/1719801549 shutdown_connections 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 -- 172.21.15.27:0/1719801549 wait complete. 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.952+0000 7fbdad7cd740 1 -- start start 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbdad7cd740 1 Processor -- start 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f96290 0x5623b9f9df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f97980 0x5623b9f9e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9fa1010 0x5623b9f9e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5623b9fa3ea0 con 0x5623b9f96290 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5623b9eb5e90 con 0x5623b9f97980 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5623b9fa4020 con 0x5623b9fa1010 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbdad7cd740 10 client.0 _ll_register_callbacks cb 0x5623b9ef1970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:09.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f96290 0x5623b9f9df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f96290 0x5623b9f9df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53008/0 (socket says 172.21.15.27:53008) 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda3fff640 1 -- 172.21.15.27:0/498224111 learned_addr learned my addr 172.21.15.27:0/498224111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda37fe640 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f97980 0x5623b9f9e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda8f80640 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9fa1010 0x5623b9f9e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda3fff640 1 -- 172.21.15.27:0/498224111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9fa1010 msgr2=0x5623b9f9e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda3fff640 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9fa1010 0x5623b9f9e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda3fff640 1 -- 172.21.15.27:0/498224111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f97980 msgr2=0x5623b9f9e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda3fff640 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f97980 0x5623b9f9e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.953+0000 7fbda3fff640 1 -- 172.21.15.27:0/498224111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623b9cd67e0 con 0x5623b9f96290 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbda3fff640 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f96290 0x5623b9f9df10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fbd9409bcf0 tx=0x7fbd9409b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478ed58cc691f01c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbd837fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbd837fe640 1 -- 172.21.15.27:0/498224111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd940a0110 con 0x5623b9f96290 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbd837fe640 1 -- 172.21.15.27:0/498224111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fbd94018b30 con 0x5623b9f96290 2024-04-25T01:34:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 1 -- 172.21.15.27:0/498224111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd9c036040 con 0x5623b9f96290 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbd837fe640 1 -- 172.21.15.27:0/498224111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd94018d30 con 0x5623b9f96290 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 10 client.17547 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 1 -- 172.21.15.27:0/498224111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5623b9d36f70 con 0x5623b9f96290 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbd837fe640 1 -- 172.21.15.27:0/498224111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd9c036040 con 0x5623b9f96290 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbd837fe640 1 -- 172.21.15.27:0/498224111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbd94097ab0 con 0x5623b9f96290 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 10 client.17547 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 10 client.17547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 20 client.17547 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5623b9fb1080 0x5623b9fb34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 4 client.17547 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.954+0000 7fbdad7cd740 1 -- 172.21.15.27:0/498224111 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623b9ece5a0 con 0x5623b9fb1080 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.955+0000 7fbda37fe640 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5623b9fb1080 0x5623b9fb34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.955+0000 7fbda37fe640 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5623b9fb1080 0x5623b9fb34f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d00f153fdc46fbe server_cookie=12ba4c823f013a16 in_seq=0 out_seq=0 2024-04-25T01:34:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.955+0000 7fbd837fe640 10 client.17547 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbd837fe640 1 -- 172.21.15.27:0/498224111 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623b9ece5a0 con 0x5623b9fb1080 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbd837fe640 10 client.17547 handle_command_reply: tid=0 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 client.17547 shutdown 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 -- 172.21.15.27:0/498224111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f96290 msgr2=0x5623b9f9df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f96290 0x5623b9f9df10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fbd9409bcf0 tx=0x7fbd9409b780 comp rx=0 tx=0).stop 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 -- 172.21.15.27:0/498224111 shutdown_connections 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5623b9fb1080 0x5623b9fb34f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9fa1010 0x5623b9f9e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f97980 0x5623b9f9e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbd837fe640 0 client.17547 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/498224111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9f96290 0x5623b9f9df10 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 -- 172.21.15.27:0/498224111 >> 172.21.15.27:0/498224111 conn(0x5623b9f5e950 msgr2=0x5623b9f985f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 -- 172.21.15.27:0/498224111 shutdown_connections 2024-04-25T01:34:09.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 1 -- 172.21.15.27:0/498224111 wait complete. 2024-04-25T01:34:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.956+0000 7fbdad7cd740 20 client.17547 trim_cache size 0 max 16384 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbdad7cd740 1 Processor -- start 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbdad7cd740 1 -- start start 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 0x5623b9facda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9fa9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f917d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41b16d0 con 0x5623b9fa1010 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41aefa0 con 0x5623b9f96290 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41ace10 con 0x5623b9f97980 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbda8f80640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f917d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbda8f80640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f917d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53018/0 (socket says 172.21.15.27:53018) 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.974+0000 7fbda8f80640 1 -- 172.21.15.27:0/3019959783 learned_addr learned my addr 172.21.15.27:0/3019959783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbda37fe640 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9fa9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbda3fff640 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 0x5623b9facda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbda8f80640 1 -- 172.21.15.27:0/3019959783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 msgr2=0x5623b9fa9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbda8f80640 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9fa9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbda8f80640 1 -- 172.21.15.27:0/3019959783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 msgr2=0x5623b9facda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbda8f80640 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 0x5623b9facda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbda8f80640 1 -- 172.21.15.27:0/3019959783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623b9bce170 con 0x5623b9fa1010 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbda8f80640 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f917d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0039c0 tx=0x7fbd9c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7fbae5d79110a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbd89ffb640 1 -- 172.21.15.27:0/3019959783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c056040 con 0x5623b9fa1010 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbd89ffb640 1 -- 172.21.15.27:0/3019959783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd9c036030 con 0x5623b9fa1010 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.975+0000 7fbd89ffb640 1 -- 172.21.15.27:0/3019959783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0305a0 con 0x5623b9fa1010 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3019959783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 msgr2=0x5623b9f917d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f917d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0039c0 tx=0x7fbd9c03d000 comp rx=0 tx=0).stop 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3019959783 shutdown_connections 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f917d0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9fa9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3019959783 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 0x5623b9facda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3019959783 >> 172.21.15.27:0/3019959783 conn(0x5623b9f5e950 msgr2=0x5623b9f910c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3019959783 shutdown_connections 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3019959783 wait complete. 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 -- start start 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.976+0000 7fbdad7cd740 1 Processor -- start 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 0x5623b9f9bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9f9a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbdad7cd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f9a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41ad460 con 0x5623b9fa1010 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41ae8d0 con 0x5623b9f96290 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbdad7cd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41adb30 con 0x5623b9f97980 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbdad7cd740 10 client.0 _ll_register_callbacks cb 0x7fbd94009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:09.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f9a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f9a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53026/0 (socket says 172.21.15.27:53026) 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 -- 172.21.15.27:0/3636238468 learned_addr learned my addr 172.21.15.27:0/3636238468 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda37fe640 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9f9a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda3fff640 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 0x5623b9f9bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 -- 172.21.15.27:0/3636238468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 msgr2=0x5623b9f9a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9f9a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 -- 172.21.15.27:0/3636238468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 msgr2=0x5623b9f9bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 0x5623b9f9bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 -- 172.21.15.27:0/3636238468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623b9f9af10 con 0x5623b9fa1010 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda37fe640 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9f9a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbda8f80640 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f9a6e0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009a60 tx=0x7fbd9c0042a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35cdac7bccb80a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbd837fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbd837fe640 1 -- 172.21.15.27:0/3636238468 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c003ab0 con 0x5623b9fa1010 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbd837fe640 1 -- 172.21.15.27:0/3636238468 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7fbd9c003c50 con 0x5623b9fa1010 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.977+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3636238468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd9c036030 con 0x5623b9fa1010 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbd837fe640 1 -- 172.21.15.27:0/3636238468 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd9c003e50 con 0x5623b9fa1010 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbdad7cd740 10 client.17553 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3636238468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5623ba01c640 con 0x5623b9fa1010 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbd837fe640 1 -- 172.21.15.27:0/3636238468 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd9c036030 con 0x5623b9fa1010 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbd837fe640 1 -- 172.21.15.27:0/3636238468 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbd9c030070 con 0x5623b9fa1010 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbdad7cd740 10 client.17553 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbdad7cd740 10 client.17553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbdad7cd740 20 client.17553 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5623ba01c8b0 0x5623ba01eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbdad7cd740 4 client.17553 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3636238468 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5623b9f07800 con 0x5623ba01c8b0 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.978+0000 7fbda3fff640 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5623ba01c8b0 0x5623ba01eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.979+0000 7fbda3fff640 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5623ba01c8b0 0x5623ba01eca0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee7e12c4f41002b7 server_cookie=aa59be512a573257 in_seq=0 out_seq=0 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.979+0000 7fbd837fe640 10 client.17553 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbd837fe640 1 -- 172.21.15.27:0/3636238468 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5623b9f07800 con 0x5623ba01c8b0 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbd837fe640 10 client.17553 handle_command_reply: tid=0 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 client.17553 shutdown 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3636238468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 msgr2=0x5623b9f9a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f9a6e0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009a60 tx=0x7fbd9c0042a0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3636238468 shutdown_connections 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5623ba01c8b0 0x5623ba01eca0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623b9fa1010 0x5623b9f9a6e0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623b9f97980 0x5623b9f9a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 --2- 172.21.15.27:0/3636238468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5623b9f96290 0x5623b9f9bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbd837fe640 0 client.17553 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3636238468 >> 172.21.15.27:0/3636238468 conn(0x5623b9f5e950 msgr2=0x5623b9f97d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3636238468 shutdown_connections 2024-04-25T01:34:09.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 1 -- 172.21.15.27:0/3636238468 wait complete. 2024-04-25T01:34:09.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.980+0000 7fbdad7cd740 20 client.17553 trim_cache size 0 max 16384 2024-04-25T01:34:09.965 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:34:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:34:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "8eeed922-9b44-463d-a270-4d5b7bfd8fe4", 2024-04-25T01:34:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:34:09.965 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:34:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd8c0a2d00 msgr2=0x7fbd8c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd8c0a2d00 0x7fbd8c0a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009430 tx=0x7fbd9c048000 comp rx=0 tx=0).stop 2024-04-25T01:34:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda4069dc0 msgr2=0x7fbda4197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda4069dc0 0x7fbda4197fa0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400e9f0 tx=0x7fbd9400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1951804162 shutdown_connections 2024-04-25T01:34:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbda4107730 0x7fbda419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbda406a780 0x7fbda41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd8c0a2d00 0x7fbd8c0a51c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 --2- 172.21.15.27:0/1951804162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbda4069dc0 0x7fbda4197fa0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1951804162 >> 172.21.15.27:0/1951804162 conn(0x7fbda40776a0 msgr2=0x7fbda40fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1951804162 shutdown_connections 2024-04-25T01:34:09.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:09.981+0000 7fbdaa9e3640 1 -- 172.21.15.27:0/1951804162 wait complete. 2024-04-25T01:34:09.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:34:09.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:34:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 -- 172.21.15.27:0/3280676709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 msgr2=0x7f0998100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3280676709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 0x7f0998100aa0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f0980009930 tx=0x7f098002a420 comp rx=0 tx=0).stop 2024-04-25T01:34:10.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 -- 172.21.15.27:0/3280676709 shutdown_connections 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3280676709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0998100fe0 0x7f099810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3280676709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 0x7f0998100aa0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3280676709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0998106630 0x7f0998106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 -- 172.21.15.27:0/3280676709 >> 172.21.15.27:0/3280676709 conn(0x7f09980776a0 msgr2=0x7f0998077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 -- 172.21.15.27:0/3280676709 shutdown_connections 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 -- 172.21.15.27:0/3280676709 wait complete. 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.096+0000 7f099ed4f640 1 Processor -- start 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099ed4f640 1 -- start start 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099ed4f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 0x7f0998196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099ed4f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0998100fe0 0x7f0998197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099ed4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0998106630 0x7f099819b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099ed4f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0998110a40 con 0x7f0998100620 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099ed4f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09981108c0 con 0x7f0998100fe0 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099ed4f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0998110bc0 con 0x7f0998106630 2024-04-25T01:34:10.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 0x7f0998196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 0x7f0998196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53042/0 (socket says 172.21.15.27:53042) 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 -- 172.21.15.27:0/3938998478 learned_addr learned my addr 172.21.15.27:0/3938998478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099d2ec640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0998106630 0x7f099819b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 -- 172.21.15.27:0/3938998478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0998106630 msgr2=0x7f099819b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0998106630 0x7f099819b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f0997fff640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0998100fe0 0x7f0998197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 -- 172.21.15.27:0/3938998478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0998100fe0 msgr2=0x7f0998197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0998100fe0 0x7f0998197400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 -- 172.21.15.27:0/3938998478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099819be60 con 0x7f0998100620 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099caeb640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 0x7f0998196ec0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f098800d770 tx=0x7f098800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f0995ffb640 1 -- 172.21.15.27:0/3938998478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988017070 con 0x7f0998100620 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f0995ffb640 1 -- 172.21.15.27:0/3938998478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0988005500 con 0x7f0998100620 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f099ed4f640 1 -- 172.21.15.27:0/3938998478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099819c150 con 0x7f0998100620 2024-04-25T01:34:10.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.097+0000 7f0995ffb640 1 -- 172.21.15.27:0/3938998478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988004e00 con 0x7f0998100620 2024-04-25T01:34:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.098+0000 7f099ed4f640 1 -- 172.21.15.27:0/3938998478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09981a39e0 con 0x7f0998100620 2024-04-25T01:34:10.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f0995ffb640 1 -- 172.21.15.27:0/3938998478 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0988004fa0 con 0x7f0998100620 2024-04-25T01:34:10.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f09a1a59740 1 Processor -- start 2024-04-25T01:34:10.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f09a1a59740 1 -- start start 2024-04-25T01:34:10.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 0x55c7deb0ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 0x55c7deb160e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb18a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f09a1a59740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c7dea706b0 con 0x55c7deb16620 2024-04-25T01:34:10.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f09a1a59740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c7dea18880 con 0x55c7deb0b630 2024-04-25T01:34:10.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.100+0000 7f09a1a59740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c7dea6c670 con 0x55c7deb0bf50 2024-04-25T01:34:10.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0995ffb640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09840a2d80 0x7f09840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb18a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb18a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53054/0 (socket says 172.21.15.27:53054) 2024-04-25T01:34:10.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 -- 172.21.15.27:0/4136990762 learned_addr learned my addr 172.21.15.27:0/4136990762 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:10.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f099d2ec640 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 0x55c7deb0ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0995ffb640 1 -- 172.21.15.27:0/3938998478 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f09880cf2a0 con 0x7f0998100620 2024-04-25T01:34:10.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f099d2ec640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09840a2d80 0x7f09840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f099caeb640 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 0x55c7deb160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 -- 172.21.15.27:0/4136990762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 msgr2=0x55c7deb160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 0x55c7deb160e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 -- 172.21.15.27:0/4136990762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 msgr2=0x55c7deb0ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 0x55c7deb0ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 -- 172.21.15.27:0/4136990762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7de84e7d0 con 0x55c7deb16620 2024-04-25T01:34:10.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f0997fff640 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb18a10 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f0980006010 tx=0x7f0980005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec0538c6e5cd326 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f099d2ec640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09840a2d80 0x7f09840a5240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0990010fd0 tx=0x7f0990004a40 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f098effd640 1 -- 172.21.15.27:0/4136990762 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098003f070 con 0x55c7deb16620 2024-04-25T01:34:10.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f098effd640 1 -- 172.21.15.27:0/4136990762 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0980004030 con 0x55c7deb16620 2024-04-25T01:34:10.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.101+0000 7f098effd640 1 -- 172.21.15.27:0/4136990762 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980035070 con 0x55c7deb16620 2024-04-25T01:34:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 -- 172.21.15.27:0/4136990762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 msgr2=0x55c7deb18a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb18a10 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f0980006010 tx=0x7f0980005e80 comp rx=0 tx=0).stop 2024-04-25T01:34:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 -- 172.21.15.27:0/4136990762 shutdown_connections 2024-04-25T01:34:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb18a10 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 0x55c7deb160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 --2- 172.21.15.27:0/4136990762 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 0x55c7deb0ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 -- 172.21.15.27:0/4136990762 >> 172.21.15.27:0/4136990762 conn(0x55c7deb07950 msgr2=0x55c7deb07d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 -- 172.21.15.27:0/4136990762 shutdown_connections 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 -- 172.21.15.27:0/4136990762 wait complete. 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.102+0000 7f09a1a59740 1 -- start start 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f09a1a59740 1 Processor -- start 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 0x55c7deb13530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0bf50 0x55c7deb13a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb13fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f09a1a59740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c7deb193b0 con 0x55c7deb16620 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f09a1a59740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55c7dea2bf50 con 0x55c7deb0bf50 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f09a1a59740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c7deb19530 con 0x55c7deb0b630 2024-04-25T01:34:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f09a1a59740 10 client.0 _ll_register_callbacks cb 0x55c7dea67000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099d2ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb13fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099d2ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb13fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53068/0 (socket says 172.21.15.27:53068) 2024-04-25T01:34:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099d2ec640 1 -- 172.21.15.27:0/1323847939 learned_addr learned my addr 172.21.15.27:0/1323847939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f0997fff640 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0bf50 0x55c7deb13a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099caeb640 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 0x55c7deb13530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099d2ec640 1 -- 172.21.15.27:0/1323847939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 msgr2=0x55c7deb13530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099d2ec640 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 0x55c7deb13530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099d2ec640 1 -- 172.21.15.27:0/1323847939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0bf50 msgr2=0x55c7deb13a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099d2ec640 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0bf50 0x55c7deb13a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.103+0000 7f099d2ec640 1 -- 172.21.15.27:0/1323847939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7de7843f0 con 0x55c7deb16620 2024-04-25T01:34:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f099d2ec640 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb13fb0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0990007fe0 tx=0x7f0990004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64199c560a7d25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f0977fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f0977fff640 1 -- 172.21.15.27:0/1323847939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099000ebd0 con 0x55c7deb16620 2024-04-25T01:34:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f0977fff640 1 -- 172.21.15.27:0/1323847939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f099000ed70 con 0x55c7deb16620 2024-04-25T01:34:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 1 -- 172.21.15.27:0/1323847939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0980004030 con 0x55c7deb16620 2024-04-25T01:34:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f0977fff640 1 -- 172.21.15.27:0/1323847939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0990021440 con 0x55c7deb16620 2024-04-25T01:34:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 10 client.17565 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 1 -- 172.21.15.27:0/1323847939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c7de7440f0 con 0x55c7deb16620 2024-04-25T01:34:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f0977fff640 1 -- 172.21.15.27:0/1323847939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0980004030 con 0x55c7deb16620 2024-04-25T01:34:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f0977fff640 1 -- 172.21.15.27:0/1323847939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0990029630 con 0x55c7deb16620 2024-04-25T01:34:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 10 client.17565 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 10 client.17565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 20 client.17565 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c7deb268e0 0x55c7deb28d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 4 client.17565 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.104+0000 7f09a1a59740 1 -- 172.21.15.27:0/1323847939 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7dea7dbf0 con 0x55c7deb268e0 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.105+0000 7f099caeb640 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c7deb268e0 0x55c7deb28d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.105+0000 7f099caeb640 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c7deb268e0 0x55c7deb28d30 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d00ea94ba50289a server_cookie=5a841dbbc56a367 in_seq=0 out_seq=0 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.105+0000 7f0977fff640 10 client.17565 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f0977fff640 1 -- 172.21.15.27:0/1323847939 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7dea7dbf0 con 0x55c7deb268e0 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f0977fff640 10 client.17565 handle_command_reply: tid=0 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 client.17565 shutdown 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 -- 172.21.15.27:0/1323847939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 msgr2=0x55c7deb13fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb13fb0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0990007fe0 tx=0x7f0990004ab0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 -- 172.21.15.27:0/1323847939 shutdown_connections 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c7deb268e0 0x55c7deb28d30 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb13fb0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0bf50 0x55c7deb13a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f0977fff640 0 client.17565 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1323847939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 0x55c7deb13530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 -- 172.21.15.27:0/1323847939 >> 172.21.15.27:0/1323847939 conn(0x55c7deb07950 msgr2=0x55c7deb0d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 -- 172.21.15.27:0/1323847939 shutdown_connections 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 1 -- 172.21.15.27:0/1323847939 wait complete. 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.106+0000 7f09a1a59740 20 client.17565 trim_cache size 0 max 16384 2024-04-25T01:34:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.123+0000 7f09a1a59740 1 Processor -- start 2024-04-25T01:34:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f09a1a59740 1 -- start start 2024-04-25T01:34:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 0x55c7deb22270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 0x55c7deb0dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb126d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f09a1a59740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09981af3c0 con 0x55c7deb16620 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f09a1a59740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09981acc90 con 0x55c7deb0b630 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f09a1a59740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09981aab00 con 0x55c7deb0bf50 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb126d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53070/0 (socket says 172.21.15.27:53070) 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 -- 172.21.15.27:0/2616729716 learned_addr learned my addr 172.21.15.27:0/2616729716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 -- 172.21.15.27:0/2616729716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 msgr2=0x55c7deb0dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 --2- 172.21.15.27:0/2616729716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 0x55c7deb0dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 -- 172.21.15.27:0/2616729716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 msgr2=0x55c7deb22270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 --2- 172.21.15.27:0/2616729716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 0x55c7deb22270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 -- 172.21.15.27:0/2616729716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7deaf9090 con 0x55c7deb16620 2024-04-25T01:34:10.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f099d2ec640 1 --2- 172.21.15.27:0/2616729716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb126d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f099000bb20 tx=0x7f09900074d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247a5e18a986654c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f098e7fc640 1 -- 172.21.15.27:0/2616729716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990007a10 con 0x55c7deb16620 2024-04-25T01:34:10.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f098e7fc640 1 -- 172.21.15.27:0/2616729716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0990029400 con 0x55c7deb16620 2024-04-25T01:34:10.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.124+0000 7f098e7fc640 1 -- 172.21.15.27:0/2616729716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099000eca0 con 0x55c7deb16620 2024-04-25T01:34:10.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 -- 172.21.15.27:0/2616729716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 msgr2=0x55c7deb126d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 --2- 172.21.15.27:0/2616729716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb126d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f099000bb20 tx=0x7f09900074d0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 -- 172.21.15.27:0/2616729716 shutdown_connections 2024-04-25T01:34:10.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 --2- 172.21.15.27:0/2616729716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb16620 0x55c7deb126d0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 --2- 172.21.15.27:0/2616729716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0bf50 0x55c7deb0dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 --2- 172.21.15.27:0/2616729716 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb0b630 0x55c7deb22270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 -- 172.21.15.27:0/2616729716 >> 172.21.15.27:0/2616729716 conn(0x55c7deb07950 msgr2=0x55c7deb06260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 -- 172.21.15.27:0/2616729716 shutdown_connections 2024-04-25T01:34:10.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 -- 172.21.15.27:0/2616729716 wait complete. 2024-04-25T01:34:10.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 -- start start 2024-04-25T01:34:10.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.125+0000 7f09a1a59740 1 Processor -- start 2024-04-25T01:34:10.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 0x55c7deb1e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb0bf50 0x55c7deb1ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f09a1a59740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb16620 0x55c7deb1b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f09a1a59740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09981ab150 con 0x55c7deb0bf50 2024-04-25T01:34:10.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f09a1a59740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09981ac5c0 con 0x55c7deb16620 2024-04-25T01:34:10.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f09a1a59740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09981ab820 con 0x55c7deb0b630 2024-04-25T01:34:10.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f09a1a59740 10 client.0 _ll_register_callbacks cb 0x7f0990007800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb0bf50 0x55c7deb1ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb0bf50 0x55c7deb1ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53078/0 (socket says 172.21.15.27:53078) 2024-04-25T01:34:10.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 -- 172.21.15.27:0/1908429696 learned_addr learned my addr 172.21.15.27:0/1908429696 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:10.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f099d2ec640 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb16620 0x55c7deb1b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f099caeb640 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 0x55c7deb1e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 -- 172.21.15.27:0/1908429696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 msgr2=0x55c7deb1e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 0x55c7deb1e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 -- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb16620 msgr2=0x55c7deb1b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb16620 0x55c7deb1b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 -- 172.21.15.27:0/1908429696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7deb1bd10 con 0x55c7deb0bf50 2024-04-25T01:34:10.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f099d2ec640 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb16620 0x55c7deb1b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:10.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0997fff640 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb0bf50 0x55c7deb1ecc0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f0980006010 tx=0x7f098002ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e10eec4db1641a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.126+0000 7f0977fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:10.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f0977fff640 1 -- 172.21.15.27:0/1908429696 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980035070 con 0x55c7deb0bf50 2024-04-25T01:34:10.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 1 -- 172.21.15.27:0/1908429696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0990029400 con 0x55c7deb0bf50 2024-04-25T01:34:10.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f0977fff640 1 -- 172.21.15.27:0/1908429696 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f09800315b0 con 0x55c7deb0bf50 2024-04-25T01:34:10.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f0977fff640 1 -- 172.21.15.27:0/1908429696 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0980056050 con 0x55c7deb0bf50 2024-04-25T01:34:10.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 10 client.17571 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:10.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 1 -- 172.21.15.27:0/1908429696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c7deb8d7c0 con 0x55c7deb0bf50 2024-04-25T01:34:10.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f0977fff640 1 -- 172.21.15.27:0/1908429696 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0990029400 con 0x55c7deb0bf50 2024-04-25T01:34:10.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f0977fff640 1 -- 172.21.15.27:0/1908429696 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0980003a60 con 0x55c7deb0bf50 2024-04-25T01:34:10.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 10 client.17571 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:10.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 10 client.17571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:10.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 20 client.17571 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:10.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c7deb8dc10 0x55c7deb90000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 4 client.17571 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:34:10.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.127+0000 7f09a1a59740 1 -- 172.21.15.27:0/1908429696 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c7dea77f40 con 0x55c7deb8dc10 2024-04-25T01:34:10.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f099caeb640 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c7deb8dc10 0x55c7deb90000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f099caeb640 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c7deb8dc10 0x55c7deb90000 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8098007e7432edde server_cookie=3069f311b602ec84 in_seq=0 out_seq=0 2024-04-25T01:34:10.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f0977fff640 10 client.17571 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:10.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f0977fff640 1 -- 172.21.15.27:0/1908429696 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c7dea77f40 con 0x55c7deb8dc10 2024-04-25T01:34:10.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f0977fff640 10 client.17571 handle_command_reply: tid=0 2024-04-25T01:34:10.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 client.17571 shutdown 2024-04-25T01:34:10.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 -- 172.21.15.27:0/1908429696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb0bf50 msgr2=0x55c7deb1ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb0bf50 0x55c7deb1ecc0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f0980006010 tx=0x7f098002ab50 comp rx=0 tx=0).stop 2024-04-25T01:34:10.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 -- 172.21.15.27:0/1908429696 shutdown_connections 2024-04-25T01:34:10.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55c7deb8dc10 0x55c7deb90000 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55c7deb16620 0x55c7deb1b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7deb0bf50 0x55c7deb1ecc0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 --2- 172.21.15.27:0/1908429696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7deb0b630 0x55c7deb1e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f0977fff640 0 client.17571 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:10.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 -- 172.21.15.27:0/1908429696 >> 172.21.15.27:0/1908429696 conn(0x55c7deb07950 msgr2=0x55c7deb06af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 -- 172.21.15.27:0/1908429696 shutdown_connections 2024-04-25T01:34:10.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 1 -- 172.21.15.27:0/1908429696 wait complete. 2024-04-25T01:34:10.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.128+0000 7f09a1a59740 20 client.17571 trim_cache size 0 max 16384 2024-04-25T01:34:10.126 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:34:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:34:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:34:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: "8eeed922-9b44-463d-a270-4d5b7bfd8fe4": { 2024-04-25T01:34:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:34:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "8eeed922-9b44-463d-a270-4d5b7bfd8fe4", 2024-04-25T01:34:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:34:10.128 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:34:10.128 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:34:10.128 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:34:10.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 -- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09840a2d80 msgr2=0x7f09840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09840a2d80 0x7f09840a5240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0990010fd0 tx=0x7f0990004a40 comp rx=0 tx=0).stop 2024-04-25T01:34:10.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 -- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 msgr2=0x7f0998196ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 0x7f0998196ec0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f098800d770 tx=0x7f098800dc30 comp rx=0 tx=0).stop 2024-04-25T01:34:10.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 -- 172.21.15.27:0/3938998478 shutdown_connections 2024-04-25T01:34:10.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0998106630 0x7f099819b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0998100fe0 0x7f0998197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09840a2d80 0x7f09840a5240 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 --2- 172.21.15.27:0/3938998478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0998100620 0x7f0998196ec0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 -- 172.21.15.27:0/3938998478 >> 172.21.15.27:0/3938998478 conn(0x7f09980776a0 msgr2=0x7f0998103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 -- 172.21.15.27:0/3938998478 shutdown_connections 2024-04-25T01:34:10.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:10.129+0000 7f099ed4f640 1 -- 172.21.15.27:0/3938998478 wait complete. 2024-04-25T01:34:10.133 INFO:tasks.cephfs.filesystem:scrub status for tag:8eeed922-9b44-463d-a270-4d5b7bfd8fe4 - {'path': '/', 'tag': '8eeed922-9b44-463d-a270-4d5b7bfd8fe4', 'options': 'recursive,force'} 2024-04-25T01:34:10.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:10 smithi153 ceph-mon[32621]: pgmap v270: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:34:10.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:10 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:34:10.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:10 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:10.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:10 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:34:10.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:10 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:10.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:10 smithi139 ceph-mon[32760]: pgmap v270: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:34:10.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:10 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:34:10.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:10 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:10.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:10 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:34:10.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:10 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:10 smithi027 ceph-mon[29197]: pgmap v270: 129 pgs: 129 active+clean; 4.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:34:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:10 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:34:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:10 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:10 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:34:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:10 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:11.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:11 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:34:11.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:11 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:11.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:11 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:34:11.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:11 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:11.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:11 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:34:11.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:11 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:12.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:12 smithi153 ceph-mon[32621]: pgmap v271: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:12.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:12 smithi139 ceph-mon[32760]: pgmap v271: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:12 smithi027 ceph-mon[29197]: pgmap v271: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:14.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:14 smithi153 ceph-mon[32621]: pgmap v272: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:14.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:14 smithi139 ceph-mon[32760]: pgmap v272: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:14.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:14 smithi027 ceph-mon[29197]: pgmap v272: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:16 smithi027 ceph-mon[29197]: pgmap v273: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:16.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:16 smithi027 ceph-mon[29197]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T01:34:16.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:16 smithi153 ceph-mon[32621]: pgmap v273: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:16.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:16 smithi153 ceph-mon[32621]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T01:34:16.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:16 smithi139 ceph-mon[32760]: pgmap v273: 129 pgs: 129 active+clean; 4.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:16.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:16 smithi139 ceph-mon[32760]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T01:34:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:18 smithi027 ceph-mon[29197]: pgmap v275: 129 pgs: 129 active+clean; 4.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:18.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:18 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:34:18.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:18 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:34:18.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:18 smithi153 ceph-mon[32621]: pgmap v275: 129 pgs: 129 active+clean; 4.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:18.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:18 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:34:18.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:18 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:34:18.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:18 smithi139 ceph-mon[32760]: pgmap v275: 129 pgs: 129 active+clean; 4.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:18.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:18 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:34:18.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:18 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:34:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:20 smithi027 ceph-mon[29197]: pgmap v276: 129 pgs: 129 active+clean; 4.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:20 smithi153 ceph-mon[32621]: pgmap v276: 129 pgs: 129 active+clean; 4.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:20.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:20 smithi139 ceph-mon[32760]: pgmap v276: 129 pgs: 129 active+clean; 4.3 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:34:22.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:22 smithi027 ceph-mon[29197]: pgmap v277: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:22.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:22 smithi153 ceph-mon[32621]: pgmap v277: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:22.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:22 smithi139 ceph-mon[32760]: pgmap v277: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:24.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:24 smithi027 ceph-mon[29197]: pgmap v278: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:24.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:24 smithi153 ceph-mon[32621]: pgmap v278: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:24.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:24 smithi139 ceph-mon[32760]: pgmap v278: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:26.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:26 smithi027 ceph-mon[29197]: pgmap v279: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:26.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:26 smithi153 ceph-mon[32621]: pgmap v279: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:26.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:26 smithi139 ceph-mon[32760]: pgmap v279: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 KiB/s wr, 11 op/s 2024-04-25T01:34:28.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:28 smithi153 ceph-mon[32621]: pgmap v280: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:34:28.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:28 smithi139 ceph-mon[32760]: pgmap v280: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:34:28.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:28 smithi027 ceph-mon[29197]: pgmap v280: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:34:30.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:30 smithi153 ceph-mon[32621]: pgmap v281: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 9 op/s 2024-04-25T01:34:30.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:30 smithi139 ceph-mon[32760]: pgmap v281: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 9 op/s 2024-04-25T01:34:30.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:30 smithi027 ceph-mon[29197]: pgmap v281: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 KiB/s wr, 9 op/s 2024-04-25T01:34:32.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:32 smithi153 ceph-mon[32621]: pgmap v282: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:32.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:32 smithi139 ceph-mon[32760]: pgmap v282: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:32.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:32 smithi027 ceph-mon[29197]: pgmap v282: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:34.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:34 smithi153 ceph-mon[32621]: pgmap v283: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:34.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:34 smithi139 ceph-mon[32760]: pgmap v283: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:34.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:34 smithi027 ceph-mon[29197]: pgmap v283: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:36.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:36 smithi153 ceph-mon[32621]: pgmap v284: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:36.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:36 smithi139 ceph-mon[32760]: pgmap v284: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:36.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:36 smithi027 ceph-mon[29197]: pgmap v284: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:37.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:37 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:37.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:37 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:37.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:37 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:38.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:38 smithi153 ceph-mon[32621]: pgmap v285: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:34:38.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:38 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:38.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:38 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:38.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:38 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:34:38.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:38 smithi139 ceph-mon[32760]: pgmap v285: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:34:38.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:38 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:38.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:38 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:38.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:38 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:34:38.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:38 smithi027 ceph-mon[29197]: pgmap v285: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:34:38.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:38 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:38.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:38 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:38.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:38 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:34:40.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:34:40.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:34:40.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.279+0000 7f35300bc640 1 -- 172.21.15.27:0/2341816039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 msgr2=0x7f352810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.279+0000 7f35300bc640 1 --2- 172.21.15.27:0/2341816039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 0x7f352810e5e0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f3520009cd0 tx=0x7f352002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.279+0000 7f35300bc640 1 -- 172.21.15.27:0/2341816039 shutdown_connections 2024-04-25T01:34:40.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.279+0000 7f35300bc640 1 --2- 172.21.15.27:0/2341816039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 0x7f352810e5e0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.279+0000 7f35300bc640 1 --2- 172.21.15.27:0/2341816039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3528101710 0x7f3528101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.279+0000 7f35300bc640 1 --2- 172.21.15.27:0/2341816039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3528107720 0x7f3528107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.279+0000 7f35300bc640 1 -- 172.21.15.27:0/2341816039 >> 172.21.15.27:0/2341816039 conn(0x7f35280fd180 msgr2=0x7f35280ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 -- 172.21.15.27:0/2341816039 shutdown_connections 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 -- 172.21.15.27:0/2341816039 wait complete. 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 Processor -- start 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 -- start start 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3528101710 0x7f3528197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 0x7f35281984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3528107720 0x7f352819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3528111b30 con 0x7f35281020d0 2024-04-25T01:34:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f35281119b0 con 0x7f3528101710 2024-04-25T01:34:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f35300bc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3528111cb0 con 0x7f3528107720 2024-04-25T01:34:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.280+0000 7f352d657640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 0x7f35281984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352d657640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 0x7f35281984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54760/0 (socket says 172.21.15.27:54760) 2024-04-25T01:34:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352d657640 1 -- 172.21.15.27:0/769728911 learned_addr learned my addr 172.21.15.27:0/769728911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352de58640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3528101710 0x7f3528197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352e659640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3528107720 0x7f352819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352d657640 1 -- 172.21.15.27:0/769728911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3528107720 msgr2=0x7f352819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352d657640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3528107720 0x7f352819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352d657640 1 -- 172.21.15.27:0/769728911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3528101710 msgr2=0x7f3528197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352d657640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3528101710 0x7f3528197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352d657640 1 -- 172.21.15.27:0/769728911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352819cf80 con 0x7f35281020d0 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f352d657640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 0x7f35281984d0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f351800c920 tx=0x7f351800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f351effd640 1 -- 172.21.15.27:0/769728911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3518007c70 con 0x7f35281020d0 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f351effd640 1 -- 172.21.15.27:0/769728911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3518007e10 con 0x7f35281020d0 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f35300bc640 1 -- 172.21.15.27:0/769728911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352819d270 con 0x7f35281020d0 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f351effd640 1 -- 172.21.15.27:0/769728911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3518018890 con 0x7f35281020d0 2024-04-25T01:34:40.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.281+0000 7f35300bc640 1 -- 172.21.15.27:0/769728911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35281a4b00 con 0x7f35281020d0 2024-04-25T01:34:40.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f351effd640 1 -- 172.21.15.27:0/769728911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3518018a70 con 0x7f35281020d0 2024-04-25T01:34:40.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f3532dc5740 1 Processor -- start 2024-04-25T01:34:40.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f3532dc5740 1 -- start start 2024-04-25T01:34:40.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 0x55fe7b9f8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 0x55fe7ba09e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f3532dc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b95d6b0 con 0x55fe7b9f8d10 2024-04-25T01:34:40.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f3532dc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b905880 con 0x55fe7b9f8360 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.284+0000 7f3532dc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b959670 con 0x55fe7ba07a30 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f351effd640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35140a2d80 0x7f35140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba074f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54766/0 (socket says 172.21.15.27:54766) 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 -- 172.21.15.27:0/866358363 learned_addr learned my addr 172.21.15.27:0/866358363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352e659640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35140a2d80 0x7f35140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352d657640 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 0x55fe7ba09e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352e659640 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 0x55fe7b9f8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f351effd640 1 -- 172.21.15.27:0/769728911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f35180cf9a0 con 0x7f35281020d0 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 -- 172.21.15.27:0/866358363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 msgr2=0x55fe7ba09e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 0x55fe7ba09e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 -- 172.21.15.27:0/866358363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 msgr2=0x55fe7b9f8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 0x55fe7b9f8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 -- 172.21.15.27:0/866358363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe7b73b7d0 con 0x55fe7b9f8d10 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352de58640 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba074f0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f352400b820 tx=0x7f352400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3174d3f3146561a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f352e659640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35140a2d80 0x7f35140a5240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3520009a60 tx=0x7f3520032fe0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.285+0000 7f3513fff640 1 -- 172.21.15.27:0/866358363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352400d950 con 0x55fe7b9f8d10 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3513fff640 1 -- 172.21.15.27:0/866358363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3524019070 con 0x55fe7b9f8d10 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3513fff640 1 -- 172.21.15.27:0/866358363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35240123e0 con 0x55fe7b9f8d10 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 -- 172.21.15.27:0/866358363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 msgr2=0x55fe7ba074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba074f0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f352400b820 tx=0x7f352400bce0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 -- 172.21.15.27:0/866358363 shutdown_connections 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 0x55fe7ba09e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba074f0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 --2- 172.21.15.27:0/866358363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 0x55fe7b9f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 -- 172.21.15.27:0/866358363 >> 172.21.15.27:0/866358363 conn(0x55fe7b9f4950 msgr2=0x55fe7b9f4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 -- 172.21.15.27:0/866358363 shutdown_connections 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.286+0000 7f3532dc5740 1 -- 172.21.15.27:0/866358363 wait complete. 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 1 -- start start 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 1 Processor -- start 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 0x55fe7ba04920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba04e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba053a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fe7ba0a7f0 con 0x55fe7ba07a30 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b918f50 con 0x55fe7b9f8d10 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fe7ba0a970 con 0x55fe7b9f8360 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f3532dc5740 10 client.0 _ll_register_callbacks cb 0x55fe7b954000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f352e659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f352e659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba053a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54768/0 (socket says 172.21.15.27:54768) 2024-04-25T01:34:40.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f352e659640 1 -- 172.21.15.27:0/1384055018 learned_addr learned my addr 172.21.15.27:0/1384055018 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f352de58640 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 0x55fe7ba04920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.287+0000 7f352d657640 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba04e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f352e659640 1 -- 172.21.15.27:0/1384055018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 msgr2=0x55fe7ba04920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f352e659640 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 0x55fe7ba04920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f352e659640 1 -- 172.21.15.27:0/1384055018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 msgr2=0x55fe7ba04e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f352e659640 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba04e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f352e659640 1 -- 172.21.15.27:0/1384055018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe7b6713f0 con 0x55fe7ba07a30 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f352e659640 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba053a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f35200039c0 tx=0x7f35200095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf9ccbd53fad8b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f35117fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f35117fa640 1 -- 172.21.15.27:0/1384055018 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352004e070 con 0x55fe7ba07a30 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f35117fa640 1 -- 172.21.15.27:0/1384055018 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f3520004650 con 0x55fe7ba07a30 2024-04-25T01:34:40.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f3532dc5740 1 -- 172.21.15.27:0/1384055018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3524019070 con 0x55fe7ba07a30 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f35117fa640 1 -- 172.21.15.27:0/1384055018 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3520004850 con 0x55fe7ba07a30 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f3532dc5740 10 client.17583 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f3532dc5740 1 -- 172.21.15.27:0/1384055018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fe7b6310f0 con 0x55fe7ba07a30 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.288+0000 7f35117fa640 1 -- 172.21.15.27:0/1384055018 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3524019070 con 0x55fe7ba07a30 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f35117fa640 1 -- 172.21.15.27:0/1384055018 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3520049040 con 0x55fe7ba07a30 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f3532dc5740 10 client.17583 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f3532dc5740 10 client.17583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f3532dc5740 20 client.17583 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f3532dc5740 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fe7ba17c10 0x55fe7ba1a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f3532dc5740 4 client.17583 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:40.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f3532dc5740 1 -- 172.21.15.27:0/1384055018 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe7b96abf0 con 0x55fe7ba17c10 2024-04-25T01:34:40.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f352de58640 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fe7ba17c10 0x55fe7ba1a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f352de58640 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fe7ba17c10 0x55fe7ba1a060 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b59210a83ae576d server_cookie=1ce53ffdaa03d8b7 in_seq=0 out_seq=0 2024-04-25T01:34:40.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.289+0000 7f35117fa640 10 client.17583 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:40.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f35117fa640 1 -- 172.21.15.27:0/1384055018 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe7b96abf0 con 0x55fe7ba17c10 2024-04-25T01:34:40.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f35117fa640 10 client.17583 handle_command_reply: tid=0 2024-04-25T01:34:40.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 client.17583 shutdown 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 -- 172.21.15.27:0/1384055018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 msgr2=0x55fe7ba053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba053a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f35200039c0 tx=0x7f35200095c0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 -- 172.21.15.27:0/1384055018 shutdown_connections 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fe7ba17c10 0x55fe7ba1a060 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba053a0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba04e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 --2- 172.21.15.27:0/1384055018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 0x55fe7ba04920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f35117fa640 0 client.17583 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 -- 172.21.15.27:0/1384055018 >> 172.21.15.27:0/1384055018 conn(0x55fe7b9f4950 msgr2=0x55fe7b9fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 -- 172.21.15.27:0/1384055018 shutdown_connections 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.290+0000 7f3532dc5740 1 -- 172.21.15.27:0/1384055018 wait complete. 2024-04-25T01:34:40.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.291+0000 7f3532dc5740 20 client.17583 trim_cache size 0 max 16384 2024-04-25T01:34:40.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f3532dc5740 1 Processor -- start 2024-04-25T01:34:40.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f3532dc5740 1 -- start start 2024-04-25T01:34:40.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 0x55fe7ba07050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba03860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 0x55fe7ba136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f3532dc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f35281af350 con 0x55fe7b9f8d10 2024-04-25T01:34:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f3532dc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f35281acc20 con 0x55fe7b9f8360 2024-04-25T01:34:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f3532dc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35281aaa90 con 0x55fe7ba07a30 2024-04-25T01:34:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f352d657640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba03860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f352d657640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba03860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54770/0 (socket says 172.21.15.27:54770) 2024-04-25T01:34:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.308+0000 7f352d657640 1 -- 172.21.15.27:0/645910841 learned_addr learned my addr 172.21.15.27:0/645910841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f352d657640 1 -- 172.21.15.27:0/645910841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 msgr2=0x55fe7ba136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f352d657640 1 --2- 172.21.15.27:0/645910841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 0x55fe7ba136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f352d657640 1 -- 172.21.15.27:0/645910841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 msgr2=0x55fe7ba07050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f352d657640 1 --2- 172.21.15.27:0/645910841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 0x55fe7ba07050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f352d657640 1 -- 172.21.15.27:0/645910841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe7b9f4780 con 0x55fe7b9f8d10 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f352d657640 1 --2- 172.21.15.27:0/645910841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba03860 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f351809b7d0 tx=0x7f3518013720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40370f66bcccee0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f35137fe640 1 -- 172.21.15.27:0/645910841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351809bad0 con 0x55fe7b9f8d10 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f35137fe640 1 -- 172.21.15.27:0/645910841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35180cfe50 con 0x55fe7b9f8d10 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f35137fe640 1 -- 172.21.15.27:0/645910841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351800fb30 con 0x55fe7b9f8d10 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f3532dc5740 1 -- 172.21.15.27:0/645910841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 msgr2=0x55fe7ba03860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f3532dc5740 1 --2- 172.21.15.27:0/645910841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba03860 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f351809b7d0 tx=0x7f3518013720 comp rx=0 tx=0).stop 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f3532dc5740 1 -- 172.21.15.27:0/645910841 shutdown_connections 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f3532dc5740 1 --2- 172.21.15.27:0/645910841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7ba07a30 0x55fe7ba136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.309+0000 7f3532dc5740 1 --2- 172.21.15.27:0/645910841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba03860 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 --2- 172.21.15.27:0/645910841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8360 0x55fe7ba07050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 -- 172.21.15.27:0/645910841 >> 172.21.15.27:0/645910841 conn(0x55fe7b9f4950 msgr2=0x55fe7b9f3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 -- 172.21.15.27:0/645910841 shutdown_connections 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 -- 172.21.15.27:0/645910841 wait complete. 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 -- start start 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 Processor -- start 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 0x55fe7ba0b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba0bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba0c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f35281ab0e0 con 0x55fe7ba07a30 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f35281ac550 con 0x55fe7b9f8d10 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35281ab7b0 con 0x55fe7b9f8360 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f3532dc5740 10 client.0 _ll_register_callbacks cb 0x7f3520009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.310+0000 7f352e659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba0c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352e659640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba0c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54774/0 (socket says 172.21.15.27:54774) 2024-04-25T01:34:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352e659640 1 -- 172.21.15.27:0/3365142650 learned_addr learned my addr 172.21.15.27:0/3365142650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352de58640 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 0x55fe7ba0b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352d657640 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba0bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352e659640 1 -- 172.21.15.27:0/3365142650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 msgr2=0x55fe7ba0b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352e659640 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 0x55fe7ba0b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352e659640 1 -- 172.21.15.27:0/3365142650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 msgr2=0x55fe7ba0bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352e659640 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba0bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352e659640 1 -- 172.21.15.27:0/3365142650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe7ba7d1a0 con 0x55fe7ba07a30 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f352e659640 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba0c070 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f352002ae00 tx=0x7f3520009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d23b32d7103dc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f35117fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f35117fa640 1 -- 172.21.15.27:0/3365142650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520004450 con 0x55fe7ba07a30 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f35117fa640 1 -- 172.21.15.27:0/3365142650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f35200045f0 con 0x55fe7ba07a30 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f3532dc5740 1 -- 172.21.15.27:0/3365142650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35180cfe50 con 0x55fe7ba07a30 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f35117fa640 1 -- 172.21.15.27:0/3365142650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f35200047f0 con 0x55fe7ba07a30 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f3532dc5740 10 client.17589 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.311+0000 7f3532dc5740 1 -- 172.21.15.27:0/3365142650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fe7ba7e480 con 0x55fe7ba07a30 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f35117fa640 1 -- 172.21.15.27:0/3365142650 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35180cfe50 con 0x55fe7ba07a30 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f35117fa640 1 -- 172.21.15.27:0/3365142650 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3520049050 con 0x55fe7ba07a30 2024-04-25T01:34:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f3532dc5740 10 client.17589 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f3532dc5740 10 client.17589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f3532dc5740 20 client.17589 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f3532dc5740 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fe7ba7ec10 0x55fe7ba81030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f3532dc5740 4 client.17589 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f3532dc5740 1 -- 172.21.15.27:0/3365142650 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fe7b964f40 con 0x55fe7ba7ec10 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f352de58640 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fe7ba7ec10 0x55fe7ba81030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f352de58640 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fe7ba7ec10 0x55fe7ba81030 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=339425ae381be826 server_cookie=4a15237dbb46cd5d in_seq=0 out_seq=0 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.312+0000 7f35117fa640 10 client.17589 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f35117fa640 1 -- 172.21.15.27:0/3365142650 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fe7b964f40 con 0x55fe7ba7ec10 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f35117fa640 10 client.17589 handle_command_reply: tid=0 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 client.17589 shutdown 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 -- 172.21.15.27:0/3365142650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 msgr2=0x55fe7ba0c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba0c070 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f352002ae00 tx=0x7f3520009890 comp rx=0 tx=0).stop 2024-04-25T01:34:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 -- 172.21.15.27:0/3365142650 shutdown_connections 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55fe7ba7ec10 0x55fe7ba81030 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe7ba07a30 0x55fe7ba0c070 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55fe7b9f8d10 0x55fe7ba0bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f35117fa640 0 client.17589 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 --2- 172.21.15.27:0/3365142650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe7b9f8360 0x55fe7ba0b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 -- 172.21.15.27:0/3365142650 >> 172.21.15.27:0/3365142650 conn(0x55fe7b9f4950 msgr2=0x55fe7b9f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 -- 172.21.15.27:0/3365142650 shutdown_connections 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 1 -- 172.21.15.27:0/3365142650 wait complete. 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f3532dc5740 20 client.17589 trim_cache size 0 max 16384 2024-04-25T01:34:40.299 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:34:40.300 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:34:40.300 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:34:40.300 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:34:40.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f35300bc640 1 -- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35140a2d80 msgr2=0x7f35140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f35300bc640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35140a2d80 0x7f35140a5240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3520009a60 tx=0x7f3520032fe0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f35300bc640 1 -- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 msgr2=0x7f35281984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.313+0000 7f35300bc640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 0x7f35281984d0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f351800c920 tx=0x7f351800cde0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.314+0000 7f35300bc640 1 -- 172.21.15.27:0/769728911 shutdown_connections 2024-04-25T01:34:40.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.314+0000 7f35300bc640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3528107720 0x7f352819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.314+0000 7f35300bc640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f35140a2d80 0x7f35140a5240 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.314+0000 7f35300bc640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35281020d0 0x7f35281984d0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.314+0000 7f35300bc640 1 --2- 172.21.15.27:0/769728911 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3528101710 0x7f3528197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.314+0000 7f35300bc640 1 -- 172.21.15.27:0/769728911 >> 172.21.15.27:0/769728911 conn(0x7f35280fd180 msgr2=0x7f35280fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.314+0000 7f35300bc640 1 -- 172.21.15.27:0/769728911 shutdown_connections 2024-04-25T01:34:40.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.314+0000 7f35300bc640 1 -- 172.21.15.27:0/769728911 wait complete. 2024-04-25T01:34:40.302 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:34:40.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:34:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 -- 172.21.15.27:0/4050457199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c01017c0 msgr2=0x7ff3c0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 --2- 172.21.15.27:0/4050457199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c01017c0 0x7ff3c0101c20 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009900 tx=0x7ff3b402a480 comp rx=0 tx=0).stop 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 -- 172.21.15.27:0/4050457199 shutdown_connections 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 --2- 172.21.15.27:0/4050457199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0102160 0x7ff3c010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 --2- 172.21.15.27:0/4050457199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c01017c0 0x7ff3c0101c20 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 --2- 172.21.15.27:0/4050457199 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3c0107750 0x7ff3c0107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 -- 172.21.15.27:0/4050457199 >> 172.21.15.27:0/4050457199 conn(0x7ff3c00788d0 msgr2=0x7ff3c0078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 -- 172.21.15.27:0/4050457199 shutdown_connections 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.429+0000 7ff3c5283640 1 -- 172.21.15.27:0/4050457199 wait complete. 2024-04-25T01:34:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3c5283640 1 Processor -- start 2024-04-25T01:34:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3c5283640 1 -- start start 2024-04-25T01:34:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3c5283640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3c01017c0 0x7ff3c0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3c5283640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0102160 0x7ff3c0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3c5283640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c0107750 0x7ff3c019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3c5283640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c0111b10 con 0x7ff3c0107750 2024-04-25T01:34:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3c5283640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c0111990 con 0x7ff3c01017c0 2024-04-25T01:34:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3c5283640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c0111c90 con 0x7ff3c0102160 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c0107750 0x7ff3c019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c0107750 0x7ff3c019c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54792/0 (socket says 172.21.15.27:54792) 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3bf7fe640 1 -- 172.21.15.27:0/3088279674 learned_addr learned my addr 172.21.15.27:0/3088279674 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.430+0000 7ff3beffd640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3c01017c0 0x7ff3c0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3be7fc640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0102160 0x7ff3c0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3bf7fe640 1 -- 172.21.15.27:0/3088279674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0102160 msgr2=0x7ff3c0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3bf7fe640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0102160 0x7ff3c0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3bf7fe640 1 -- 172.21.15.27:0/3088279674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3c01017c0 msgr2=0x7ff3c0198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3bf7fe640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3c01017c0 0x7ff3c0198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3bf7fe640 1 -- 172.21.15.27:0/3088279674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c019d020 con 0x7ff3c0107750 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3bf7fe640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c0107750 0x7ff3c019c8f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800cad0 tx=0x7ff3b800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3a7fff640 1 -- 172.21.15.27:0/3088279674 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b800ae20 con 0x7ff3c0107750 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c019d310 con 0x7ff3c0107750 2024-04-25T01:34:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3a7fff640 1 -- 172.21.15.27:0/3088279674 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b8005020 con 0x7ff3c0107750 2024-04-25T01:34:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3a7fff640 1 -- 172.21.15.27:0/3088279674 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b800f6f0 con 0x7ff3c0107750 2024-04-25T01:34:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.431+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c01a4ba0 con 0x7ff3c0107750 2024-04-25T01:34:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.432+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c00689b0 con 0x7ff3c0107750 2024-04-25T01:34:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.433+0000 7ff3a7fff640 1 -- 172.21.15.27:0/3088279674 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff3b80051d0 con 0x7ff3c0107750 2024-04-25T01:34:40.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.434+0000 7ff3a7fff640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff3a80a2d00 0x7ff3a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.434+0000 7ff3beffd640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff3a80a2d00 0x7ff3a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.434+0000 7ff3a7fff640 1 -- 172.21.15.27:0/3088279674 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7ff3b80cfba0 con 0x7ff3c0107750 2024-04-25T01:34:40.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.434+0000 7ff3beffd640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff3a80a2d00 0x7ff3a80a51c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009b50 tx=0x7ff3b0005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.438+0000 7ff3a7fff640 1 -- 172.21.15.27:0/3088279674 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff3b8014030 con 0x7ff3c0107750 2024-04-25T01:34:40.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.662+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3c002d170 con 0x7ff3c0107750 2024-04-25T01:34:40.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.662+0000 7ff3a7fff640 1 -- 172.21.15.27:0/3088279674 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7ff3b809c3d0 con 0x7ff3c0107750 2024-04-25T01:34:40.639 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:34:40.640 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:34:40.640 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:34:40.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff3a80a2d00 msgr2=0x7ff3a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff3a80a2d00 0x7ff3a80a51c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009b50 tx=0x7ff3b0005f90 comp rx=0 tx=0).stop 2024-04-25T01:34:40.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c0107750 msgr2=0x7ff3c019c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c0107750 0x7ff3c019c8f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800cad0 tx=0x7ff3b800cf90 comp rx=0 tx=0).stop 2024-04-25T01:34:40.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 shutdown_connections 2024-04-25T01:34:40.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff3a80a2d00 0x7ff3a80a51c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c0107750 0x7ff3c019c8f0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0102160 0x7ff3c0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 --2- 172.21.15.27:0/3088279674 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3c01017c0 0x7ff3c0198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 >> 172.21.15.27:0/3088279674 conn(0x7ff3c00788d0 msgr2=0x7ff3c01005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 shutdown_connections 2024-04-25T01:34:40.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.665+0000 7ff3c5283640 1 -- 172.21.15.27:0/3088279674 wait complete. 2024-04-25T01:34:40.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:34:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 -- 172.21.15.27:0/419934162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64106630 msgr2=0x7f8c64106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 --2- 172.21.15.27:0/419934162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64106630 0x7f8c64106a10 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009930 tx=0x7f8c4c02a420 comp rx=0 tx=0).stop 2024-04-25T01:34:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 -- 172.21.15.27:0/419934162 shutdown_connections 2024-04-25T01:34:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 --2- 172.21.15.27:0/419934162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c64100fe0 0x7f8c6410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 --2- 172.21.15.27:0/419934162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c64100620 0x7f8c64100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 --2- 172.21.15.27:0/419934162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64106630 0x7f8c64106a10 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 -- 172.21.15.27:0/419934162 >> 172.21.15.27:0/419934162 conn(0x7f8c640776a0 msgr2=0x7f8c64077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 -- 172.21.15.27:0/419934162 shutdown_connections 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 -- 172.21.15.27:0/419934162 wait complete. 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.781+0000 7f8c69db0640 1 Processor -- start 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c69db0640 1 -- start start 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c69db0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64100620 0x7f8c64196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c69db0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c64100fe0 0x7f8c64197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c69db0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c64106630 0x7f8c6419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c69db0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64110a00 con 0x7f8c64100620 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c69db0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64110880 con 0x7f8c64100fe0 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c69db0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64110b80 con 0x7f8c64106630 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64100620 0x7f8c64196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64100620 0x7f8c64196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54818/0 (socket says 172.21.15.27:54818) 2024-04-25T01:34:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 -- 172.21.15.27:0/2577488842 learned_addr learned my addr 172.21.15.27:0/2577488842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c64100fe0 0x7f8c64197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c63fff640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c64106630 0x7f8c6419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 -- 172.21.15.27:0/2577488842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c64106630 msgr2=0x7f8c6419b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c64106630 0x7f8c6419b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 -- 172.21.15.27:0/2577488842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c64100fe0 msgr2=0x7f8c64197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c64100fe0 0x7f8c64197480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 -- 172.21.15.27:0/2577488842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6419bee0 con 0x7f8c64100620 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c64100fe0 0x7f8c64197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.782+0000 7f8c637fe640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64100620 0x7f8c64196f40 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009230 tx=0x7f8c4c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.783+0000 7f8c60ff9640 1 -- 172.21.15.27:0/2577488842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c03f030 con 0x7f8c64100620 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.783+0000 7f8c60ff9640 1 -- 172.21.15.27:0/2577488842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c4c003ad0 con 0x7f8c64100620 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.783+0000 7f8c60ff9640 1 -- 172.21.15.27:0/2577488842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c035070 con 0x7f8c64100620 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.783+0000 7f8c69db0640 1 -- 172.21.15.27:0/2577488842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c6419c170 con 0x7f8c64100620 2024-04-25T01:34:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.783+0000 7f8c69db0640 1 -- 172.21.15.27:0/2577488842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c641a3a60 con 0x7f8c64100620 2024-04-25T01:34:40.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.785+0000 7f8c60ff9640 1 -- 172.21.15.27:0/2577488842 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8c4c042030 con 0x7f8c64100620 2024-04-25T01:34:40.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.785+0000 7f8c6cba0740 1 Processor -- start 2024-04-25T01:34:40.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.785+0000 7f8c6cba0740 1 -- start start 2024-04-25T01:34:40.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c55960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55ea0 0x558ae3c60030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c60570 0x558ae3c62960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558ae3bba670 con 0x558ae3c55580 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558ae3b62970 con 0x558ae3c55ea0 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558ae3bb6630 con 0x558ae3c60570 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c60ff9640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8c500a2d00 0x7f8c500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c55960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c55960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54822/0 (socket says 172.21.15.27:54822) 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c637fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55ea0 0x558ae3c60030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 -- 172.21.15.27:0/1868236369 learned_addr learned my addr 172.21.15.27:0/1868236369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c60570 0x558ae3c62960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8c500a2d00 0x7f8c500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c60ff9640 1 -- 172.21.15.27:0/2577488842 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f8c4c0efdf0 con 0x7f8c64100620 2024-04-25T01:34:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 -- 172.21.15.27:0/1868236369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c60570 msgr2=0x558ae3c62960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c60570 0x558ae3c62960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 -- 172.21.15.27:0/1868236369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55ea0 msgr2=0x558ae3c60030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55ea0 0x558ae3c60030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c63fff640 1 -- 172.21.15.27:0/1868236369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ae3b86400 con 0x558ae3c55580 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.786+0000 7f8c637fe640 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55ea0 0x558ae3c60030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c63fff640 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c55960 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c012cd0 tx=0x7f8c5c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed41703b115c678e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c63fff640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8c500a2d00 0x7f8c500a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00fc30 tx=0x7f8c5c01a000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c59ffb640 1 -- 172.21.15.27:0/1868236369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c020070 con 0x558ae3c55580 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c59ffb640 1 -- 172.21.15.27:0/1868236369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c5c00aaa0 con 0x558ae3c55580 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c59ffb640 1 -- 172.21.15.27:0/1868236369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0113e0 con 0x558ae3c55580 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c6cba0740 1 -- 172.21.15.27:0/1868236369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 msgr2=0x558ae3c55960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c55960 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c012cd0 tx=0x7f8c5c00f730 comp rx=0 tx=0).stop 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c6cba0740 1 -- 172.21.15.27:0/1868236369 shutdown_connections 2024-04-25T01:34:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c60570 0x558ae3c62960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55ea0 0x558ae3c60030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/1868236369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c55960 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.787+0000 7f8c6cba0740 1 -- 172.21.15.27:0/1868236369 >> 172.21.15.27:0/1868236369 conn(0x558ae3c518a0 msgr2=0x558ae3c51c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 -- 172.21.15.27:0/1868236369 shutdown_connections 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 -- 172.21.15.27:0/1868236369 wait complete. 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 -- start start 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 Processor -- start 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55580 0x558ae3c5ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55ea0 0x558ae3c5d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 0x558ae3c5d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558ae3c654b0 con 0x558ae3c55ea0 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558ae3b76050 con 0x558ae3c60570 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558ae3c65630 con 0x558ae3c55580 2024-04-25T01:34:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c6cba0740 10 client.0 _ll_register_callbacks cb 0x558ae38cdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c62ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55ea0 0x558ae3c5d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c62ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55ea0 0x558ae3c5d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54834/0 (socket says 172.21.15.27:54834) 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c62ffd640 1 -- 172.21.15.27:0/3621367521 learned_addr learned my addr 172.21.15.27:0/3621367521 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.788+0000 7f8c637fe640 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55580 0x558ae3c5ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c63fff640 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 0x558ae3c5d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c62ffd640 1 -- 172.21.15.27:0/3621367521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55580 msgr2=0x558ae3c5ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55580 0x558ae3c5ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c62ffd640 1 -- 172.21.15.27:0/3621367521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 msgr2=0x558ae3c5d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 0x558ae3c5d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c62ffd640 1 -- 172.21.15.27:0/3621367521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ae3b47dc0 con 0x558ae3c55ea0 2024-04-25T01:34:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55ea0 0x558ae3c5d230 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400b930 tx=0x7f8c5400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697ba3da42979f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c42ffd640 1 -- 172.21.15.27:0/3621367521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5401b640 con 0x558ae3c55ea0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c6cba0740 1 -- 172.21.15.27:0/3621367521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c5c00aaa0 con 0x558ae3c55ea0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c42ffd640 1 -- 172.21.15.27:0/3621367521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f8c5401b7e0 con 0x558ae3c55ea0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c42ffd640 1 -- 172.21.15.27:0/3621367521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8c54024050 con 0x558ae3c55ea0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c6cba0740 10 client.17607 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c6cba0740 1 -- 172.21.15.27:0/3621367521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558ae38ce3f0 con 0x558ae3c55ea0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.789+0000 7f8c42ffd640 1 -- 172.21.15.27:0/3621367521 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c5c00aaa0 con 0x558ae3c55ea0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c42ffd640 1 -- 172.21.15.27:0/3621367521 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8c54019020 con 0x558ae3c55ea0 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c6cba0740 10 client.17607 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c6cba0740 10 client.17607 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:34:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c6cba0740 20 client.17607 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558ae3c72830 0x558ae3c74c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c6cba0740 4 client.17607 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c6cba0740 1 -- 172.21.15.27:0/3621367521 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ae3bc7bb0 con 0x558ae3c72830 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c637fe640 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558ae3c72830 0x558ae3c74c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c637fe640 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558ae3c72830 0x558ae3c74c70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7674da2d896e1b67 server_cookie=b746afeb267725c3 in_seq=0 out_seq=0 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.790+0000 7f8c42ffd640 10 client.17607 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c42ffd640 1 -- 172.21.15.27:0/3621367521 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ae3bc7bb0 con 0x558ae3c72830 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c42ffd640 10 client.17607 handle_command_reply: tid=0 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 client.17607 shutdown 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 -- 172.21.15.27:0/3621367521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55ea0 msgr2=0x558ae3c5d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55ea0 0x558ae3c5d230 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400b930 tx=0x7f8c5400bdf0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 -- 172.21.15.27:0/3621367521 shutdown_connections 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558ae3c72830 0x558ae3c74c70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 0x558ae3c5d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55ea0 0x558ae3c5d230 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c42ffd640 0 client.17607 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/3621367521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55580 0x558ae3c5ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 -- 172.21.15.27:0/3621367521 >> 172.21.15.27:0/3621367521 conn(0x558ae3c518a0 msgr2=0x558ae3c50b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.791+0000 7f8c6cba0740 1 -- 172.21.15.27:0/3621367521 shutdown_connections 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.792+0000 7f8c6cba0740 1 -- 172.21.15.27:0/3621367521 wait complete. 2024-04-25T01:34:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.792+0000 7f8c6cba0740 20 client.17607 trim_cache size 0 max 16384 2024-04-25T01:34:40.777 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:40 smithi153 ceph-mon[32621]: pgmap v286: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c6cba0740 1 Processor -- start 2024-04-25T01:34:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c6cba0740 1 -- start start 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55580 0x558ae3c512c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 0x558ae3c58980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c60570 0x558ae3c5fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641af310 con 0x558ae3c60570 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641acbe0 con 0x558ae3c55580 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641aaa50 con 0x558ae3c55ea0 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c63fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c60570 0x558ae3c5fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c63fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c60570 0x558ae3c5fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54838/0 (socket says 172.21.15.27:54838) 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c63fff640 1 -- 172.21.15.27:0/2054897241 learned_addr learned my addr 172.21.15.27:0/2054897241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c637fe640 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55580 0x558ae3c512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 0x558ae3c58980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c63fff640 1 -- 172.21.15.27:0/2054897241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 msgr2=0x558ae3c58980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c63fff640 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 0x558ae3c58980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c63fff640 1 -- 172.21.15.27:0/2054897241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55580 msgr2=0x558ae3c512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c63fff640 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55580 0x558ae3c512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c63fff640 1 -- 172.21.15.27:0/2054897241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ae39987d0 con 0x558ae3c60570 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 0x558ae3c58980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.805+0000 7f8c637fe640 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55580 0x558ae3c512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c63fff640 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c60570 0x558ae3c5fa70 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00ffd0 tx=0x7f8c5c00a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82ca484951a25e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c597fa640 1 -- 172.21.15.27:0/2054897241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c032070 con 0x558ae3c60570 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c597fa640 1 -- 172.21.15.27:0/2054897241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c5c019ab0 con 0x558ae3c60570 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c597fa640 1 -- 172.21.15.27:0/2054897241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c020040 con 0x558ae3c60570 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 -- 172.21.15.27:0/2054897241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c60570 msgr2=0x558ae3c5fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c60570 0x558ae3c5fa70 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00ffd0 tx=0x7f8c5c00a490 comp rx=0 tx=0).stop 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 -- 172.21.15.27:0/2054897241 shutdown_connections 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c60570 0x558ae3c5fa70 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 0x558ae3c58980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/2054897241 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c55580 0x558ae3c512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 -- 172.21.15.27:0/2054897241 >> 172.21.15.27:0/2054897241 conn(0x558ae3c518a0 msgr2=0x558ae3c57560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 -- 172.21.15.27:0/2054897241 shutdown_connections 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.806+0000 7f8c6cba0740 1 -- 172.21.15.27:0/2054897241 wait complete. 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 1 -- start start 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 1 Processor -- start 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c665c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 0x558ae3c66b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 0x558ae3c67060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641ab0a0 con 0x558ae3c55580 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641ac510 con 0x558ae3c60570 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641ab770 con 0x558ae3c55ea0 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c6cba0740 10 client.0 _ll_register_callbacks cb 0x7f8c54004330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c665c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54854/0 (socket says 172.21.15.27:54854) 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c637fe640 1 -- 172.21.15.27:0/756513832 learned_addr learned my addr 172.21.15.27:0/756513832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.807+0000 7f8c63fff640 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 0x558ae3c67060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c637fe640 1 -- 172.21.15.27:0/756513832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 msgr2=0x558ae3c66b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c637fe640 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 0x558ae3c66b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c637fe640 1 -- 172.21.15.27:0/756513832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 msgr2=0x558ae3c67060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c637fe640 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 0x558ae3c67060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c637fe640 1 -- 172.21.15.27:0/756513832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ae3cd8080 con 0x558ae3c55580 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c637fe640 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c665c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0b88f0 tx=0x7f8c4c036450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4273d26c7a5ba52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c42ffd640 1 -- 172.21.15.27:0/756513832 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c049040 con 0x558ae3c55580 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c42ffd640 1 -- 172.21.15.27:0/756513832 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f8c4c03b460 con 0x558ae3c55580 2024-04-25T01:34:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 1 -- 172.21.15.27:0/756513832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c5c019ab0 con 0x558ae3c55580 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c42ffd640 1 -- 172.21.15.27:0/756513832 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8c4c0c02d0 con 0x558ae3c55580 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 10 client.17613 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 1 -- 172.21.15.27:0/756513832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558ae3cd9350 con 0x558ae3c55580 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c42ffd640 1 -- 172.21.15.27:0/756513832 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c5c019ab0 con 0x558ae3c55580 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c42ffd640 1 -- 172.21.15.27:0/756513832 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8c4c0ed3d0 con 0x558ae3c55580 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 10 client.17613 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 10 client.17613 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 20 client.17613 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558ae3cd9a70 0x558ae3cdbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 4 client.17613 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.808+0000 7f8c6cba0740 1 -- 172.21.15.27:0/756513832 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ae3bc1f00 con 0x558ae3cd9a70 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558ae3cd9a70 0x558ae3cdbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c62ffd640 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558ae3cd9a70 0x558ae3cdbea0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e39a3febb97528e server_cookie=f7d08e7ddb377ac1 in_seq=0 out_seq=0 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c42ffd640 10 client.17613 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:34:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c42ffd640 1 -- 172.21.15.27:0/756513832 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ae3bc1f00 con 0x558ae3cd9a70 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c42ffd640 10 client.17613 handle_command_reply: tid=0 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 client.17613 shutdown 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 -- 172.21.15.27:0/756513832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 msgr2=0x558ae3c665c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c665c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0b88f0 tx=0x7f8c4c036450 comp rx=0 tx=0).stop 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 -- 172.21.15.27:0/756513832 shutdown_connections 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x558ae3cd9a70 0x558ae3cdbea0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558ae3c60570 0x558ae3c67060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558ae3c55ea0 0x558ae3c66b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c42ffd640 0 client.17613 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 --2- 172.21.15.27:0/756513832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558ae3c55580 0x558ae3c665c0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.809+0000 7f8c6cba0740 1 -- 172.21.15.27:0/756513832 >> 172.21.15.27:0/756513832 conn(0x558ae3c518a0 msgr2=0x558ae3c6d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c6cba0740 1 -- 172.21.15.27:0/756513832 shutdown_connections 2024-04-25T01:34:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c6cba0740 1 -- 172.21.15.27:0/756513832 wait complete. 2024-04-25T01:34:40.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c6cba0740 20 client.17613 trim_cache size 0 max 16384 2024-04-25T01:34:40.793 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 -- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8c500a2d00 msgr2=0x7f8c500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8c500a2d00 0x7f8c500a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00fc30 tx=0x7f8c5c01a000 comp rx=0 tx=0).stop 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 -- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64100620 msgr2=0x7f8c64196f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64100620 0x7f8c64196f40 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009230 tx=0x7f8c4c004270 comp rx=0 tx=0).stop 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 -- 172.21.15.27:0/2577488842 shutdown_connections 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c64106630 0x7f8c6419b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c64100fe0 0x7f8c64197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8c500a2d00 0x7f8c500a51c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 --2- 172.21.15.27:0/2577488842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c64100620 0x7f8c64196f40 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 -- 172.21.15.27:0/2577488842 >> 172.21.15.27:0/2577488842 conn(0x7f8c640776a0 msgr2=0x7f8c64103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 -- 172.21.15.27:0/2577488842 shutdown_connections 2024-04-25T01:34:40.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.810+0000 7f8c69db0640 1 -- 172.21.15.27:0/2577488842 wait complete. 2024-04-25T01:34:40.795 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:40 smithi139 ceph-mon[32760]: pgmap v286: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:40.796 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:40 smithi027 ceph-mon[29197]: pgmap v286: 129 pgs: 129 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:40.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:34:40.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 -- 172.21.15.27:0/3731285654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0106630 msgr2=0x7f29d0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3731285654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0106630 0x7f29d0106a10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009930 tx=0x7f29c402a420 comp rx=0 tx=0).stop 2024-04-25T01:34:40.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 -- 172.21.15.27:0/3731285654 shutdown_connections 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3731285654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0100fe0 0x7f29d010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3731285654 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29d0100620 0x7f29d0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3731285654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0106630 0x7f29d0106a10 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 -- 172.21.15.27:0/3731285654 >> 172.21.15.27:0/3731285654 conn(0x7f29d00776a0 msgr2=0x7f29d0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 -- 172.21.15.27:0/3731285654 shutdown_connections 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.925+0000 7f29d6df9640 1 -- 172.21.15.27:0/3731285654 wait complete. 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d6df9640 1 Processor -- start 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d6df9640 1 -- start start 2024-04-25T01:34:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d6df9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0100620 0x7f29d0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d6df9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0100fe0 0x7f29d0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d6df9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29d0106630 0x7f29d019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d6df9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29d0110a40 con 0x7f29d0100fe0 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d6df9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29d01108c0 con 0x7f29d0106630 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d6df9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29d0110bc0 con 0x7f29d0100620 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0100fe0 0x7f29d0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0100fe0 0x7f29d0198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54866/0 (socket says 172.21.15.27:54866) 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29cffff640 1 -- 172.21.15.27:0/3752797895 learned_addr learned my addr 172.21.15.27:0/3752797895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d5396640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29d0106630 0x7f29d019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29cffff640 1 -- 172.21.15.27:0/3752797895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0100620 msgr2=0x7f29d0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29d4b95640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0100620 0x7f29d0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29cffff640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0100620 0x7f29d0197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29cffff640 1 -- 172.21.15.27:0/3752797895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29d0106630 msgr2=0x7f29d019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29cffff640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29d0106630 0x7f29d019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.926+0000 7f29cffff640 1 -- 172.21.15.27:0/3752797895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d019cfe0 con 0x7f29d0100fe0 2024-04-25T01:34:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.927+0000 7f29cffff640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0100fe0 0x7f29d0198530 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00d950 tx=0x7f29bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.927+0000 7f29cdffb640 1 -- 172.21.15.27:0/3752797895 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc017070 con 0x7f29d0100fe0 2024-04-25T01:34:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.927+0000 7f29cdffb640 1 -- 172.21.15.27:0/3752797895 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29bc009c30 con 0x7f29d0100fe0 2024-04-25T01:34:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.927+0000 7f29d6df9640 1 -- 172.21.15.27:0/3752797895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d019d2d0 con 0x7f29d0100fe0 2024-04-25T01:34:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.927+0000 7f29cdffb640 1 -- 172.21.15.27:0/3752797895 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc00f6b0 con 0x7f29d0100fe0 2024-04-25T01:34:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.927+0000 7f29d6df9640 1 -- 172.21.15.27:0/3752797895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d01a4b60 con 0x7f29d0100fe0 2024-04-25T01:34:40.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29cdffb640 1 -- 172.21.15.27:0/3752797895 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f29bc0109d0 con 0x7f29d0100fe0 2024-04-25T01:34:40.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d9be1740 1 Processor -- start 2024-04-25T01:34:40.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d9be1740 1 -- start start 2024-04-25T01:34:40.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f00960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00ea0 0x558542f0b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f0b570 0x558542f0d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d9be1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558542e65670 con 0x558542f00580 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d9be1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558542e0d970 con 0x558542f00ea0 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d9be1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558542e61630 con 0x558542f0b570 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29cdffb640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f29c00a2d00 0x7f29c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d5396640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f00960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d5396640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f00960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54876/0 (socket says 172.21.15.27:54876) 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d5396640 1 -- 172.21.15.27:0/2701190877 learned_addr learned my addr 172.21.15.27:0/2701190877 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29cffff640 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f0b570 0x558542f0d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.930+0000 7f29d4b95640 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00ea0 0x558542f0b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29d5396640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f29c00a2d00 0x7f29c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29cdffb640 1 -- 172.21.15.27:0/3752797895 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f29bc0cf540 con 0x7f29d0100fe0 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29d5396640 1 -- 172.21.15.27:0/2701190877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f0b570 msgr2=0x558542f0d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29d5396640 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f0b570 0x558542f0d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29d5396640 1 -- 172.21.15.27:0/2701190877 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00ea0 msgr2=0x558542f0b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29d5396640 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00ea0 0x558542f0b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29d5396640 1 -- 172.21.15.27:0/2701190877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558542e31400 con 0x558542f00580 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29d5396640 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f00960 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f29c8014940 tx=0x7f29c8014e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00809427f137d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29baffd640 1 -- 172.21.15.27:0/2701190877 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8014050 con 0x558542f00580 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29baffd640 1 -- 172.21.15.27:0/2701190877 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29c80141f0 con 0x558542f00580 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29baffd640 1 -- 172.21.15.27:0/2701190877 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8013640 con 0x558542f00580 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.931+0000 7f29d5396640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f29c00a2d00 0x7f29c00a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f29c8021000 tx=0x7f29c8020000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 -- 172.21.15.27:0/2701190877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 msgr2=0x558542f00960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f00960 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f29c8014940 tx=0x7f29c8014e00 comp rx=0 tx=0).stop 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 -- 172.21.15.27:0/2701190877 shutdown_connections 2024-04-25T01:34:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f0b570 0x558542f0d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00ea0 0x558542f0b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 --2- 172.21.15.27:0/2701190877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f00960 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 -- 172.21.15.27:0/2701190877 >> 172.21.15.27:0/2701190877 conn(0x558542efc8a0 msgr2=0x558542efcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 -- 172.21.15.27:0/2701190877 shutdown_connections 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 -- 172.21.15.27:0/2701190877 wait complete. 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 -- start start 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.932+0000 7f29d9be1740 1 Processor -- start 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f0aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d9be1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558542f0e360 con 0x558542f00580 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d9be1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x558542e21050 con 0x558542f0b570 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d9be1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558542f0e4e0 con 0x558542f00ea0 2024-04-25T01:34:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d9be1740 10 client.0 _ll_register_callbacks cb 0x558542b78be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f0aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f0aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54886/0 (socket says 172.21.15.27:54886) 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 -- 172.21.15.27:0/3010353269 learned_addr learned my addr 172.21.15.27:0/3010353269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29cffff640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d5396640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 -- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 msgr2=0x558542f081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f081a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 -- 172.21.15.27:0/3010353269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 msgr2=0x558542f086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 -- 172.21.15.27:0/3010353269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558542df2dc0 con 0x558542f00580 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29cffff640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d5396640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29d4b95640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f0aeb0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f29c4005940 tx=0x7f29c4031860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00555f8fc8cbb41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29affff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.933+0000 7f29affff640 1 -- 172.21.15.27:0/3010353269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c403f070 con 0x558542f00580 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 1 -- 172.21.15.27:0/3010353269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29c80141f0 con 0x558542f00580 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29affff640 1 -- 172.21.15.27:0/3010353269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f29c4004450 con 0x558542f00580 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29affff640 1 -- 172.21.15.27:0/3010353269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f29c404c050 con 0x558542f00580 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 10 client.17625 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 1 -- 172.21.15.27:0/3010353269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558542b793f0 con 0x558542f00580 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29affff640 1 -- 172.21.15.27:0/3010353269 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29c80141f0 con 0x558542f00580 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29affff640 1 -- 172.21.15.27:0/3010353269 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f29c4004aa0 con 0x558542f00580 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 10 client.17625 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 10 client.17625 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 20 client.17625 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x558542f1b620 0x558542f1da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 4 client.17625 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29d9be1740 1 -- 172.21.15.27:0/3010353269 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558542e72bb0 con 0x558542f1b620 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.934+0000 7f29cffff640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x558542f1b620 0x558542f1da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.935+0000 7f29cffff640 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x558542f1b620 0x558542f1da70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=91c1120eee0f4663 server_cookie=c8401b18b068bc7a in_seq=0 out_seq=0 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.935+0000 7f29affff640 10 client.17625 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29affff640 1 -- 172.21.15.27:0/3010353269 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558542e72bb0 con 0x558542f1b620 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29affff640 10 client.17625 handle_command_reply: tid=0 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 client.17625 shutdown 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 -- 172.21.15.27:0/3010353269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 msgr2=0x558542f0aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f0aeb0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f29c4005940 tx=0x7f29c4031860 comp rx=0 tx=0).stop 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 -- 172.21.15.27:0/3010353269 shutdown_connections 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x558542f1b620 0x558542f1da70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29affff640 0 client.17625 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3010353269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f0aeb0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 -- 172.21.15.27:0/3010353269 >> 172.21.15.27:0/3010353269 conn(0x558542efc8a0 msgr2=0x558542efbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 -- 172.21.15.27:0/3010353269 shutdown_connections 2024-04-25T01:34:40.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 1 -- 172.21.15.27:0/3010353269 wait complete. 2024-04-25T01:34:40.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.936+0000 7f29d9be1740 20 client.17625 trim_cache size 0 max 16384 2024-04-25T01:34:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.949+0000 7f29d9be1740 1 Processor -- start 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.949+0000 7f29d9be1740 1 -- start start 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.949+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00580 0x558542f17250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.949+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f0b570 0x558542f085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d9be1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29d01b1590 con 0x558542f0b570 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d9be1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29d01aee60 con 0x558542f00580 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d9be1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29d01accd0 con 0x558542f00ea0 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f0b570 0x558542f085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f0b570 0x558542f085c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54898/0 (socket says 172.21.15.27:54898) 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 -- 172.21.15.27:0/3123255727 learned_addr learned my addr 172.21.15.27:0/3123255727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29cffff640 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 -- 172.21.15.27:0/3123255727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 msgr2=0x558542f070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d4b95640 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00580 0x558542f17250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 -- 172.21.15.27:0/3123255727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00580 msgr2=0x558542f17250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00580 0x558542f17250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 -- 172.21.15.27:0/3123255727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558542c437d0 con 0x558542f0b570 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29cffff640 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29d5396640 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f0b570 0x558542f085c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f29c8000f80 tx=0x7f29c8005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6d2cec218c6f09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29ba7fc640 1 -- 172.21.15.27:0/3123255727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c80120c0 con 0x558542f0b570 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29ba7fc640 1 -- 172.21.15.27:0/3123255727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29c8014530 con 0x558542f0b570 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.950+0000 7f29ba7fc640 1 -- 172.21.15.27:0/3123255727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8013c20 con 0x558542f0b570 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 -- 172.21.15.27:0/3123255727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f0b570 msgr2=0x558542f085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f0b570 0x558542f085c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f29c8000f80 tx=0x7f29c8005fc0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 -- 172.21.15.27:0/3123255727 shutdown_connections 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f0b570 0x558542f085c0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 --2- 172.21.15.27:0/3123255727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f00580 0x558542f17250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 -- 172.21.15.27:0/3123255727 >> 172.21.15.27:0/3123255727 conn(0x558542efc8a0 msgr2=0x558542f02560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 -- 172.21.15.27:0/3123255727 shutdown_connections 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 -- 172.21.15.27:0/3123255727 wait complete. 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 -- start start 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.951+0000 7f29d9be1740 1 Processor -- start 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f13910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f0ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d9be1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f0f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d9be1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29d01ad320 con 0x558542f00580 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d9be1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f29d01ae790 con 0x558542f0b570 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d9be1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29d01ad9f0 con 0x558542f00ea0 2024-04-25T01:34:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d9be1740 10 client.0 _ll_register_callbacks cb 0x7f29c4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f13910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f13910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54900/0 (socket says 172.21.15.27:54900) 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 -- 172.21.15.27:0/1734377775 learned_addr learned my addr 172.21.15.27:0/1734377775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29cffff640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f0ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d5396640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f0f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 -- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 msgr2=0x558542f0ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f0ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 -- 172.21.15.27:0/1734377775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 msgr2=0x558542f0f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f0f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 -- 172.21.15.27:0/1734377775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558542f0f7b0 con 0x558542f00580 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d5396640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f0f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29cffff640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f0ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d4b95640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f13910 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f29c40036c0 tx=0x7f29c40317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9bb0ce1a7de8f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29affff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29affff640 1 -- 172.21.15.27:0/1734377775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4054020 con 0x558542f00580 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29affff640 1 -- 172.21.15.27:0/1734377775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f29c4004450 con 0x558542f00580 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.952+0000 7f29d9be1740 1 -- 172.21.15.27:0/1734377775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29c8014530 con 0x558542f00580 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29affff640 1 -- 172.21.15.27:0/1734377775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f29c405f050 con 0x558542f00580 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29d9be1740 10 client.17631 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29d9be1740 1 -- 172.21.15.27:0/1734377775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558542f81900 con 0x558542f00580 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29affff640 1 -- 172.21.15.27:0/1734377775 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29c8014530 con 0x558542f00580 2024-04-25T01:34:40.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29affff640 1 -- 172.21.15.27:0/1734377775 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f29c4004fe0 con 0x558542f00580 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29d9be1740 10 client.17631 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29d9be1740 10 client.17631 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29d9be1740 20 client.17631 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29d9be1740 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x558542f1b620 0x558542f84110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29d9be1740 4 client.17631 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29d9be1740 1 -- 172.21.15.27:0/1734377775 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558542e6cf00 con 0x558542f1b620 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.953+0000 7f29cffff640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x558542f1b620 0x558542f84110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29cffff640 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x558542f1b620 0x558542f84110 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e2834172a4723d2 server_cookie=d9f4bc623d284c08 in_seq=0 out_seq=0 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29affff640 10 client.17631 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29affff640 1 -- 172.21.15.27:0/1734377775 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558542e6cf00 con 0x558542f1b620 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29affff640 10 client.17631 handle_command_reply: tid=0 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 client.17631 shutdown 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 -- 172.21.15.27:0/1734377775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 msgr2=0x558542f13910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f13910 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f29c40036c0 tx=0x7f29c40317c0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 -- 172.21.15.27:0/1734377775 shutdown_connections 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x558542f1b620 0x558542f84110 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558542f0b570 0x558542f0f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558542f00ea0 0x558542f0ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 --2- 172.21.15.27:0/1734377775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558542f00580 0x558542f13910 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 -- 172.21.15.27:0/1734377775 >> 172.21.15.27:0/1734377775 conn(0x558542efc8a0 msgr2=0x558542f02d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29affff640 0 client.17631 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 -- 172.21.15.27:0/1734377775 shutdown_connections 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 1 -- 172.21.15.27:0/1734377775 wait complete. 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.954+0000 7f29d9be1740 20 client.17631 trim_cache size 0 max 16384 2024-04-25T01:34:40.938 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:34:40.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 -- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f29c00a2d00 msgr2=0x7f29c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f29c00a2d00 0x7f29c00a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f29c8021000 tx=0x7f29c8020000 comp rx=0 tx=0).stop 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 -- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0100fe0 msgr2=0x7f29d0198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0100fe0 0x7f29d0198530 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00d950 tx=0x7f29bc00de10 comp rx=0 tx=0).stop 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 -- 172.21.15.27:0/3752797895 shutdown_connections 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f29d0106630 0x7f29d019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f29c00a2d00 0x7f29c00a51c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d0100fe0 0x7f29d0198530 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 --2- 172.21.15.27:0/3752797895 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29d0100620 0x7f29d0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 -- 172.21.15.27:0/3752797895 >> 172.21.15.27:0/3752797895 conn(0x7f29d00776a0 msgr2=0x7f29d006a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 -- 172.21.15.27:0/3752797895 shutdown_connections 2024-04-25T01:34:40.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:40.955+0000 7f29d6df9640 1 -- 172.21.15.27:0/3752797895 wait complete. 2024-04-25T01:34:40.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:34:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3175404984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4101710 msgr2=0x7f79a4101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3175404984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4101710 0x7f79a4101b90 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f798c009930 tx=0x7f798c02a420 comp rx=0 tx=0).stop 2024-04-25T01:34:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3175404984 shutdown_connections 2024-04-25T01:34:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3175404984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79a41020d0 0x7f79a410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3175404984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4101710 0x7f79a4101b90 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3175404984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79a4107720 0x7f79a4107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3175404984 >> 172.21.15.27:0/3175404984 conn(0x7f79a40fd180 msgr2=0x7f79a40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3175404984 shutdown_connections 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.071+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3175404984 wait complete. 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a8cf5640 1 Processor -- start 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a8cf5640 1 -- start start 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a8cf5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79a4101710 0x7f79a4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a8cf5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79a41020d0 0x7f79a4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a8cf5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4107720 0x7f79a419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a8cf5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f79a4111b30 con 0x7f79a4107720 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a8cf5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41119b0 con 0x7f79a41020d0 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a8cf5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79a4111cb0 con 0x7f79a4101710 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4107720 0x7f79a419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4107720 0x7f79a419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54922/0 (socket says 172.21.15.27:54922) 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a2d9d640 1 -- 172.21.15.27:0/3176087747 learned_addr learned my addr 172.21.15.27:0/3176087747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a259c640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79a4101710 0x7f79a4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f799bfff640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79a41020d0 0x7f79a4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.072+0000 7f79a2d9d640 1 -- 172.21.15.27:0/3176087747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79a4101710 msgr2=0x7f79a4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f79a2d9d640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79a4101710 0x7f79a4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f79a2d9d640 1 -- 172.21.15.27:0/3176087747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79a41020d0 msgr2=0x7f79a4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f79a2d9d640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79a41020d0 0x7f79a4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f79a2d9d640 1 -- 172.21.15.27:0/3176087747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a419cfe0 con 0x7f79a4107720 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f799bfff640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79a41020d0 0x7f79a4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f79a2d9d640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4107720 0x7f79a419c8b0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f799c00b970 tx=0x7f799c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f799b7fe640 1 -- 172.21.15.27:0/3176087747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c019070 con 0x7f79a4107720 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f799b7fe640 1 -- 172.21.15.27:0/3176087747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f799c00cb40 con 0x7f79a4107720 2024-04-25T01:34:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3176087747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a419d2d0 con 0x7f79a4107720 2024-04-25T01:34:41.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f799b7fe640 1 -- 172.21.15.27:0/3176087747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c007650 con 0x7f79a4107720 2024-04-25T01:34:41.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.073+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3176087747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a41a4b60 con 0x7f79a4107720 2024-04-25T01:34:41.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.075+0000 7f799b7fe640 1 -- 172.21.15.27:0/3176087747 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f799c01c070 con 0x7f79a4107720 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79ab9fe740 1 Processor -- start 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79ab9fe740 1 -- start start 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f799b7fe640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f79900a2d80 0x7f79900a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004480 0x563c17004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004da0 0x563c17013440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17015d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563c16f69670 con 0x563c17004da0 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563c16f11970 con 0x563c17004480 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563c16f65630 con 0x563c17013980 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f799b7fe640 1 -- 172.21.15.27:0/3176087747 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f799c0ced50 con 0x7f79a4107720 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f799bfff640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f79900a2d80 0x7f79900a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f799bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004da0 0x563c17013440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f799bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004da0 0x563c17013440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54936/0 (socket says 172.21.15.27:54936) 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f799bfff640 1 -- 172.21.15.27:0/3735375831 learned_addr learned my addr 172.21.15.27:0/3735375831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79a259c640 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004480 0x563c17004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.076+0000 7f79a2d9d640 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17015d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f799bfff640 1 -- 172.21.15.27:0/3735375831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 msgr2=0x563c17015d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f799bfff640 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17015d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f799bfff640 1 -- 172.21.15.27:0/3735375831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004480 msgr2=0x563c17004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f799bfff640 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004480 0x563c17004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f799bfff640 1 -- 172.21.15.27:0/3735375831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c16f35400 con 0x563c17004da0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f799bfff640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f79900a2d80 0x7f79900a5240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f798c033000 tx=0x7f798c009680 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f799bfff640 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004da0 0x563c17013440 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f798c0318f0 tx=0x7f798c044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8ae0d81d34d97f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f7998ff9640 1 -- 172.21.15.27:0/3735375831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c04f070 con 0x563c17004da0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f7998ff9640 1 -- 172.21.15.27:0/3735375831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f798c004400 con 0x563c17004da0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.077+0000 7f7998ff9640 1 -- 172.21.15.27:0/3735375831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c035040 con 0x563c17004da0 2024-04-25T01:34:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- 172.21.15.27:0/3735375831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004da0 msgr2=0x563c17013440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004da0 0x563c17013440 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f798c0318f0 tx=0x7f798c044000 comp rx=0 tx=0).stop 2024-04-25T01:34:41.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- 172.21.15.27:0/3735375831 shutdown_connections 2024-04-25T01:34:41.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17015d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004da0 0x563c17013440 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/3735375831 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004480 0x563c17004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- 172.21.15.27:0/3735375831 >> 172.21.15.27:0/3735375831 conn(0x563c170008a0 msgr2=0x563c17000c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- 172.21.15.27:0/3735375831 shutdown_connections 2024-04-25T01:34:41.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- 172.21.15.27:0/3735375831 wait complete. 2024-04-25T01:34:41.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- start start 2024-04-25T01:34:41.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 Processor -- start 2024-04-25T01:34:41.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c17013270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17004da0 0x563c17010560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17013980 0x563c17010aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563c170166e0 con 0x563c17004480 2024-04-25T01:34:41.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563c16f25050 con 0x563c17013980 2024-04-25T01:34:41.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563c17016860 con 0x563c17004da0 2024-04-25T01:34:41.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.078+0000 7f79ab9fe740 10 client.0 _ll_register_callbacks cb 0x563c16c7cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:41.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c17013270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c17013270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54942/0 (socket says 172.21.15.27:54942) 2024-04-25T01:34:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 -- 172.21.15.27:0/1698840692 learned_addr learned my addr 172.21.15.27:0/1698840692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f799bfff640 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17004da0 0x563c17010560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a2d9d640 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17013980 0x563c17010aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 -- 172.21.15.27:0/1698840692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17004da0 msgr2=0x563c17010560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17004da0 0x563c17010560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 -- 172.21.15.27:0/1698840692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17013980 msgr2=0x563c17010aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17013980 0x563c17010aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 -- 172.21.15.27:0/1698840692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c16ef6dc0 con 0x563c17004480 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79a259c640 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c17013270 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f799400cf10 tx=0x7f79940039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406aa9f1b3ccfcfb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f7989ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f7989ffb640 1 -- 172.21.15.27:0/1698840692 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994011a50 con 0x563c17004480 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79ab9fe740 1 -- 172.21.15.27:0/1698840692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f798c004400 con 0x563c17004480 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f7989ffb640 1 -- 172.21.15.27:0/1698840692 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f7994011bf0 con 0x563c17004480 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f7989ffb640 1 -- 172.21.15.27:0/1698840692 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7994011df0 con 0x563c17004480 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79ab9fe740 10 client.17643 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.079+0000 7f79ab9fe740 1 -- 172.21.15.27:0/1698840692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c16c7d3f0 con 0x563c17004480 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f7989ffb640 1 -- 172.21.15.27:0/1698840692 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f798c004400 con 0x563c17004480 2024-04-25T01:34:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f7989ffb640 1 -- 172.21.15.27:0/1698840692 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7994018070 con 0x563c17004480 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f79ab9fe740 10 client.17643 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f79ab9fe740 10 client.17643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f79ab9fe740 20 client.17643 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c17023920 0x563c17025d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f79ab9fe740 4 client.17643 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f79ab9fe740 1 -- 172.21.15.27:0/1698840692 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c16f76bb0 con 0x563c17023920 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.080+0000 7f799bfff640 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c17023920 0x563c17025d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f799bfff640 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c17023920 0x563c17025d70 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e94f739f9925cc0f server_cookie=bfbac67bff91a5c in_seq=0 out_seq=0 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f7989ffb640 10 client.17643 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f7989ffb640 1 -- 172.21.15.27:0/1698840692 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c16f76bb0 con 0x563c17023920 2024-04-25T01:34:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f7989ffb640 10 client.17643 handle_command_reply: tid=0 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 client.17643 shutdown 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 -- 172.21.15.27:0/1698840692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 msgr2=0x563c17013270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c17013270 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f799400cf10 tx=0x7f79940039c0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 -- 172.21.15.27:0/1698840692 shutdown_connections 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c17023920 0x563c17025d70 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17013980 0x563c17010aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f7989ffb640 0 client.17643 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17004da0 0x563c17010560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/1698840692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c17013270 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.081+0000 7f79ab9fe740 1 -- 172.21.15.27:0/1698840692 >> 172.21.15.27:0/1698840692 conn(0x563c170008a0 msgr2=0x563c16fffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.082+0000 7f79ab9fe740 1 -- 172.21.15.27:0/1698840692 shutdown_connections 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.082+0000 7f79ab9fe740 1 -- 172.21.15.27:0/1698840692 wait complete. 2024-04-25T01:34:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.082+0000 7f79ab9fe740 20 client.17643 trim_cache size 0 max 16384 2024-04-25T01:34:41.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79ab9fe740 1 Processor -- start 2024-04-25T01:34:41.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79ab9fe740 1 -- start start 2024-04-25T01:34:41.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 0x563c1700baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17012da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41af440 con 0x563c17004480 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41acd10 con 0x563c17004da0 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41aab80 con 0x563c17013980 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79a259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79a259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54950/0 (socket says 172.21.15.27:54950) 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79a259c640 1 -- 172.21.15.27:0/2464142898 learned_addr learned my addr 172.21.15.27:0/2464142898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79a2d9d640 1 --2- 172.21.15.27:0/2464142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17012da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79a259c640 1 -- 172.21.15.27:0/2464142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 msgr2=0x563c17012da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79a259c640 1 --2- 172.21.15.27:0/2464142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17012da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79a259c640 1 -- 172.21.15.27:0/2464142898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 msgr2=0x563c1700baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.095+0000 7f79a259c640 1 --2- 172.21.15.27:0/2464142898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 0x563c1700baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79a259c640 1 -- 172.21.15.27:0/2464142898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c16d477d0 con 0x563c17004480 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79a259c640 1 --2- 172.21.15.27:0/2464142898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701f610 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7994004190 tx=0x7f799400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dde980cca3deea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f798bfff640 1 -- 172.21.15.27:0/2464142898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799402d070 con 0x563c17004480 2024-04-25T01:34:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f798bfff640 1 -- 172.21.15.27:0/2464142898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7994019980 con 0x563c17004480 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f798bfff640 1 -- 172.21.15.27:0/2464142898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79940056c0 con 0x563c17004480 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2464142898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 msgr2=0x563c1701f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2464142898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701f610 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7994004190 tx=0x7f799400cdb0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2464142898 shutdown_connections 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2464142898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17012da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2464142898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 0x563c1700baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2464142898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701f610 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2464142898 >> 172.21.15.27:0/2464142898 conn(0x563c170008a0 msgr2=0x563c1700b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2464142898 shutdown_connections 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.096+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2464142898 wait complete. 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 1 -- start start 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 1 Processor -- start 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 0x563c1701bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17016fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41ab1d0 con 0x563c17004480 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41ac640 con 0x563c17004da0 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79a41ab8a0 con 0x563c17013980 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79ab9fe740 10 client.0 _ll_register_callbacks cb 0x7f79940041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54964/0 (socket says 172.21.15.27:54964) 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a259c640 1 -- 172.21.15.27:0/2337870204 learned_addr learned my addr 172.21.15.27:0/2337870204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f799bfff640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 0x563c1701bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a2d9d640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17016fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a259c640 1 -- 172.21.15.27:0/2337870204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 msgr2=0x563c17016fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a259c640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17016fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a259c640 1 -- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 msgr2=0x563c1701bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a259c640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 0x563c1701bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a259c640 1 -- 172.21.15.27:0/2337870204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c170177b0 con 0x563c17004480 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f799bfff640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 0x563c1701bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.097+0000 7f79a2d9d640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17016fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79a259c640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701ba80 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7994019710 tx=0x7f7994011430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da7052789ed62d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f7989ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f7989ffb640 1 -- 172.21.15.27:0/2337870204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994036060 con 0x563c17004480 2024-04-25T01:34:41.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2337870204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7994019980 con 0x563c17004480 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f7989ffb640 1 -- 172.21.15.27:0/2337870204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f79940193e0 con 0x563c17004480 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f7989ffb640 1 -- 172.21.15.27:0/2337870204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f799403d050 con 0x563c17004480 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 10 client.17649 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2337870204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c17089aa0 con 0x563c17004480 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f7989ffb640 1 -- 172.21.15.27:0/2337870204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7994019980 con 0x563c17004480 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f7989ffb640 1 -- 172.21.15.27:0/2337870204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7994030070 con 0x563c17004480 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 10 client.17649 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 20 client.17649 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c1708a0d0 0x563c1708c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 4 client.17649 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.098+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2337870204 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c17010e30 con 0x563c1708a0d0 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f799bfff640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c1708a0d0 0x563c1708c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f7989ffb640 10 client.17649 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f799bfff640 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c1708a0d0 0x563c1708c4f0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78e5811de36b70ed server_cookie=c790f76ede886ea2 in_seq=0 out_seq=0 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f7989ffb640 1 -- 172.21.15.27:0/2337870204 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c17010e30 con 0x563c1708a0d0 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f7989ffb640 10 client.17649 handle_command_reply: tid=0 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 client.17649 shutdown 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2337870204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 msgr2=0x563c1701ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701ba80 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7994019710 tx=0x7f7994011430 comp rx=0 tx=0).stop 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2337870204 shutdown_connections 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563c1708a0d0 0x563c1708c4f0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c17013980 0x563c17016fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563c17004da0 0x563c1701bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 --2- 172.21.15.27:0/2337870204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c17004480 0x563c1701ba80 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f7989ffb640 0 client.17649 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:41.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2337870204 >> 172.21.15.27:0/2337870204 conn(0x563c170008a0 msgr2=0x563c1700f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2337870204 shutdown_connections 2024-04-25T01:34:41.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.099+0000 7f79ab9fe740 1 -- 172.21.15.27:0/2337870204 wait complete. 2024-04-25T01:34:41.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79ab9fe740 20 client.17649 trim_cache size 0 max 16384 2024-04-25T01:34:41.085 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:34:41.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f79900a2d80 msgr2=0x7f79900a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f79900a2d80 0x7f79900a5240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f798c033000 tx=0x7f798c009680 comp rx=0 tx=0).stop 2024-04-25T01:34:41.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4107720 msgr2=0x7f79a419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4107720 0x7f79a419c8b0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f799c00b970 tx=0x7f799c00be30 comp rx=0 tx=0).stop 2024-04-25T01:34:41.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3176087747 shutdown_connections 2024-04-25T01:34:41.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f79900a2d80 0x7f79900a5240 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a4107720 0x7f79a419c8b0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f79a41020d0 0x7f79a4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 --2- 172.21.15.27:0/3176087747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79a4101710 0x7f79a4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3176087747 >> 172.21.15.27:0/3176087747 conn(0x7f79a40fd180 msgr2=0x7f79a40fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3176087747 shutdown_connections 2024-04-25T01:34:41.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:41.100+0000 7f79a8cf5640 1 -- 172.21.15.27:0/3176087747 wait complete. 2024-04-25T01:34:41.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:41 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3088279674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:41.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:41 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3088279674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:41.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:41 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3088279674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:42.088 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:34:42.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:34:42.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3240868015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c107720 msgr2=0x7fd39c107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3240868015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c107720 0x7fd39c107b00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fd394009cd0 tx=0x7fd39402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3240868015 shutdown_connections 2024-04-25T01:34:42.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3240868015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd39c1020d0 0x7fd39c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3240868015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd39c101710 0x7fd39c101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3240868015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c107720 0x7fd39c107b00 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3240868015 >> 172.21.15.27:0/3240868015 conn(0x7fd39c0fd180 msgr2=0x7fd39c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3240868015 shutdown_connections 2024-04-25T01:34:42.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3240868015 wait complete. 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 Processor -- start 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.232+0000 7fd3a39c1640 1 -- start start 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a39c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd39c101710 0x7fd39c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a39c1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c1020d0 0x7fd39c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a39c1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd39c107720 0x7fd39c19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a39c1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c111bb0 con 0x7fd39c1020d0 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a39c1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c111a30 con 0x7fd39c107720 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a39c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c111d30 con 0x7fd39c101710 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c1020d0 0x7fd39c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c1020d0 0x7fd39c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54990/0 (socket says 172.21.15.27:54990) 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd39c107720 0x7fd39c19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 -- 172.21.15.27:0/3480663411 learned_addr learned my addr 172.21.15.27:0/3480663411 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a175d640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd39c101710 0x7fd39c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 -- 172.21.15.27:0/3480663411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd39c101710 msgr2=0x7fd39c198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd39c101710 0x7fd39c198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 -- 172.21.15.27:0/3480663411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd39c107720 msgr2=0x7fd39c19ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd39c107720 0x7fd39c19ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 -- 172.21.15.27:0/3480663411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c19d160 con 0x7fd39c1020d0 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.233+0000 7fd3a0f5c640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c1020d0 0x7fd39c1986b0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00b4a0 tx=0x7fd38c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.234+0000 7fd3927fc640 1 -- 172.21.15.27:0/3480663411 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c014070 con 0x7fd39c1020d0 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.234+0000 7fd3927fc640 1 -- 172.21.15.27:0/3480663411 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd38c010af0 con 0x7fd39c1020d0 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.234+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3480663411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd39c19d450 con 0x7fd39c1020d0 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.234+0000 7fd3927fc640 1 -- 172.21.15.27:0/3480663411 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c017d10 con 0x7fd39c1020d0 2024-04-25T01:34:42.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.234+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3480663411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd39c1a4ce0 con 0x7fd39c1020d0 2024-04-25T01:34:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3927fc640 1 -- 172.21.15.27:0/3480663411 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd38c017070 con 0x7fd39c1020d0 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a67af740 1 Processor -- start 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a67af740 1 -- start start 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97df400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a67af740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aac97362c0 con 0x55aac97d2290 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a67af740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55aac96de910 con 0x55aac97d3980 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a67af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55aac9732280 con 0x55aac97dd010 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a175d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a175d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54996/0 (socket says 172.21.15.27:54996) 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.237+0000 7fd3a175d640 1 -- 172.21.15.27:0/2208879347 learned_addr learned my addr 172.21.15.27:0/2208879347 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a0f5c640 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97df400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3927fc640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3880a2d00 0x7fd3880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a175d640 1 -- 172.21.15.27:0/2208879347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 msgr2=0x55aac97df400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a175d640 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97df400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a175d640 1 -- 172.21.15.27:0/2208879347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 msgr2=0x55aac97d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a175d640 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a175d640 1 -- 172.21.15.27:0/2208879347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aac96effb0 con 0x55aac97d2290 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3880a2d00 0x7fd3880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3927fc640 1 -- 172.21.15.27:0/3480663411 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fd38c0cea50 con 0x7fd39c1020d0 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a175d640 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d2710 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd394009cd0 tx=0x7fd394031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79af920a04b128a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3877fe640 1 -- 172.21.15.27:0/2208879347 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394048070 con 0x55aac97d2290 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3880a2d00 0x7fd3880a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd398009ae0 tx=0x7fd398008040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3877fe640 1 -- 172.21.15.27:0/2208879347 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd394036070 con 0x55aac97d2290 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.238+0000 7fd3877fe640 1 -- 172.21.15.27:0/2208879347 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394004670 con 0x55aac97d2290 2024-04-25T01:34:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 -- 172.21.15.27:0/2208879347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 msgr2=0x55aac97d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d2710 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd394009cd0 tx=0x7fd394031700 comp rx=0 tx=0).stop 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 -- 172.21.15.27:0/2208879347 shutdown_connections 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97df400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d2710 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 --2- 172.21.15.27:0/2208879347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 -- 172.21.15.27:0/2208879347 >> 172.21.15.27:0/2208879347 conn(0x55aac979a950 msgr2=0x55aac97ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 -- 172.21.15.27:0/2208879347 shutdown_connections 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 -- 172.21.15.27:0/2208879347 wait complete. 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 -- start start 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.239+0000 7fd3a67af740 1 Processor -- start 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97da440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97da980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a67af740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aac97dfe40 con 0x55aac97d2290 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a67af740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55aac96f1e90 con 0x55aac97d3980 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a67af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55aac97dffc0 con 0x55aac97dd010 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a67af740 10 client.0 _ll_register_callbacks cb 0x55aac972d970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a175d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a175d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55006/0 (socket says 172.21.15.27:55006) 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a175d640 1 -- 172.21.15.27:0/887068883 learned_addr learned my addr 172.21.15.27:0/887068883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a0f5c640 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97da440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a175d640 1 -- 172.21.15.27:0/887068883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 msgr2=0x55aac97da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97da980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a175d640 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97da980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a175d640 1 -- 172.21.15.27:0/887068883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 msgr2=0x55aac97da440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a175d640 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97da440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a175d640 1 -- 172.21.15.27:0/887068883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aac95127e0 con 0x55aac97d2290 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.240+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97da980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a175d640 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d9f00 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fd394009ca0 tx=0x7fd3940093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89530690ae5e2fac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd384ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd384ff9640 1 -- 172.21.15.27:0/887068883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394052020 con 0x55aac97d2290 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd384ff9640 1 -- 172.21.15.27:0/887068883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fd3940047e0 con 0x55aac97d2290 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a67af740 1 -- 172.21.15.27:0/887068883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd394036070 con 0x55aac97d2290 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd384ff9640 1 -- 172.21.15.27:0/887068883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd394048040 con 0x55aac97d2290 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a67af740 10 client.17661 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a67af740 1 -- 172.21.15.27:0/887068883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55aac9572f70 con 0x55aac97d2290 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd384ff9640 1 -- 172.21.15.27:0/887068883 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd394036070 con 0x55aac97d2290 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd384ff9640 1 -- 172.21.15.27:0/887068883 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd39404b030 con 0x55aac97d2290 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a67af740 10 client.17661 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a67af740 10 client.17661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a67af740 20 client.17661 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a67af740 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55aac97ecef0 0x55aac97ef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.241+0000 7fd3a67af740 4 client.17661 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.242+0000 7fd3a67af740 1 -- 172.21.15.27:0/887068883 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aac970a5a0 con 0x55aac97ecef0 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.242+0000 7fd3a0f5c640 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55aac97ecef0 0x55aac97ef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.242+0000 7fd384ff9640 10 client.17661 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.242+0000 7fd3a0f5c640 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55aac97ecef0 0x55aac97ef2e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3a36a254c9148eb server_cookie=9a7ef3391f2b2524 in_seq=0 out_seq=0 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd384ff9640 1 -- 172.21.15.27:0/887068883 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aac970a5a0 con 0x55aac97ecef0 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd384ff9640 10 client.17661 handle_command_reply: tid=0 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 client.17661 shutdown 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 -- 172.21.15.27:0/887068883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 msgr2=0x55aac97d9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d9f00 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fd394009ca0 tx=0x7fd3940093c0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 -- 172.21.15.27:0/887068883 shutdown_connections 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55aac97ecef0 0x55aac97ef2e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97da980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97da440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 --2- 172.21.15.27:0/887068883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97d9f00 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd3a67af740 1 -- 172.21.15.27:0/887068883 >> 172.21.15.27:0/887068883 conn(0x55aac979a950 msgr2=0x55aac97d45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.243+0000 7fd384ff9640 0 client.17661 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.244+0000 7fd3a67af740 1 -- 172.21.15.27:0/887068883 shutdown_connections 2024-04-25T01:34:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.244+0000 7fd3a67af740 1 -- 172.21.15.27:0/887068883 wait complete. 2024-04-25T01:34:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.244+0000 7fd3a67af740 20 client.17661 trim_cache size 0 max 16384 2024-04-25T01:34:42.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.261+0000 7fd3a67af740 1 Processor -- start 2024-04-25T01:34:42.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.261+0000 7fd3a67af740 1 -- start start 2024-04-25T01:34:42.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a67af740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c1b16b0 con 0x55aac97d2290 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a67af740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c1aef80 con 0x55aac97d3980 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a67af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c1acdf0 con 0x55aac97dd010 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97e8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55022/0 (socket says 172.21.15.27:55022) 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 -- 172.21.15.27:0/989642293 learned_addr learned my addr 172.21.15.27:0/989642293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/989642293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 -- 172.21.15.27:0/989642293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 msgr2=0x55aac97f48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 --2- 172.21.15.27:0/989642293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97f48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 -- 172.21.15.27:0/989642293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 msgr2=0x55aac97dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 --2- 172.21.15.27:0/989642293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 -- 172.21.15.27:0/989642293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aac940a170 con 0x55aac97d2290 2024-04-25T01:34:42.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd3a175d640 1 --2- 172.21.15.27:0/989642293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97e8d40 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd394004380 tx=0x7fd394031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f200afa54e1840b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd386ffd640 1 -- 172.21.15.27:0/989642293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394052070 con 0x55aac97d2290 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd386ffd640 1 -- 172.21.15.27:0/989642293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd394036070 con 0x55aac97d2290 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.262+0000 7fd386ffd640 1 -- 172.21.15.27:0/989642293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394004820 con 0x55aac97d2290 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 -- 172.21.15.27:0/989642293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 msgr2=0x55aac97e8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 --2- 172.21.15.27:0/989642293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97e8d40 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd394004380 tx=0x7fd394031700 comp rx=0 tx=0).stop 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 -- 172.21.15.27:0/989642293 shutdown_connections 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 --2- 172.21.15.27:0/989642293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97dd010 0x55aac97f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 --2- 172.21.15.27:0/989642293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d3980 0x55aac97dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 --2- 172.21.15.27:0/989642293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97d2290 0x55aac97e8d40 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 -- 172.21.15.27:0/989642293 >> 172.21.15.27:0/989642293 conn(0x55aac979a950 msgr2=0x55aac97db730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 -- 172.21.15.27:0/989642293 shutdown_connections 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 -- 172.21.15.27:0/989642293 wait complete. 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 -- start start 2024-04-25T01:34:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.263+0000 7fd3a67af740 1 Processor -- start 2024-04-25T01:34:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d2290 0x55aac97de2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97d3980 0x55aac97de820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a67af740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97dd010 0x55aac97e39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a67af740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c1ad440 con 0x55aac97dd010 2024-04-25T01:34:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a67af740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c1ae8b0 con 0x55aac97d2290 2024-04-25T01:34:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a67af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c1adb10 con 0x55aac97d3980 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a67af740 10 client.0 _ll_register_callbacks cb 0x7fd394046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97dd010 0x55aac97e39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97dd010 0x55aac97e39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55024/0 (socket says 172.21.15.27:55024) 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 -- 172.21.15.27:0/1304413619 learned_addr learned my addr 172.21.15.27:0/1304413619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a0f5c640 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97d3980 0x55aac97de820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a175d640 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d2290 0x55aac97de2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 -- 172.21.15.27:0/1304413619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97d3980 msgr2=0x55aac97de820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97d3980 0x55aac97de820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 -- 172.21.15.27:0/1304413619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d2290 msgr2=0x55aac97de2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d2290 0x55aac97de2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 -- 172.21.15.27:0/1304413619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aac97e3f60 con 0x55aac97dd010 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd3a1f5e640 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97dd010 0x55aac97e39a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd398015710 tx=0x7fd3980090e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8780c1e80e741b99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.264+0000 7fd384ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd384ff9640 1 -- 172.21.15.27:0/1304413619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398026070 con 0x55aac97dd010 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd384ff9640 1 -- 172.21.15.27:0/1304413619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fd3980149f0 con 0x55aac97dd010 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 1 -- 172.21.15.27:0/1304413619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd394036070 con 0x55aac97dd010 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd384ff9640 1 -- 172.21.15.27:0/1304413619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd398032050 con 0x55aac97dd010 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 10 client.17667 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 1 -- 172.21.15.27:0/1304413619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55aac9853a50 con 0x55aac97dd010 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd384ff9640 1 -- 172.21.15.27:0/1304413619 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd394036070 con 0x55aac97dd010 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd384ff9640 1 -- 172.21.15.27:0/1304413619 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd39802a3b0 con 0x55aac97dd010 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 10 client.17667 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 10 client.17667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 20 client.17667 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55aac9854200 0x55aac9856620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 4 client.17667 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:34:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.265+0000 7fd3a67af740 1 -- 172.21.15.27:0/1304413619 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55aac97ef6d0 con 0x55aac9854200 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.266+0000 7fd3a175d640 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55aac9854200 0x55aac9856620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.266+0000 7fd3a175d640 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55aac9854200 0x55aac9856620 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82b49376afaa955e server_cookie=f661133c64eb53d9 in_seq=0 out_seq=0 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.266+0000 7fd384ff9640 10 client.17667 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd384ff9640 1 -- 172.21.15.27:0/1304413619 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55aac97ef6d0 con 0x55aac9854200 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd384ff9640 10 client.17667 handle_command_reply: tid=0 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 client.17667 shutdown 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 -- 172.21.15.27:0/1304413619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97dd010 msgr2=0x55aac97e39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97dd010 0x55aac97e39a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd398015710 tx=0x7fd3980090e0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 -- 172.21.15.27:0/1304413619 shutdown_connections 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55aac9854200 0x55aac9856620 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aac97dd010 0x55aac97e39a0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55aac97d3980 0x55aac97de820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 --2- 172.21.15.27:0/1304413619 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55aac97d2290 0x55aac97de2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd3a67af740 1 -- 172.21.15.27:0/1304413619 >> 172.21.15.27:0/1304413619 conn(0x55aac979a950 msgr2=0x55aac97cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.267+0000 7fd384ff9640 0 client.17667 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a67af740 1 -- 172.21.15.27:0/1304413619 shutdown_connections 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a67af740 1 -- 172.21.15.27:0/1304413619 wait complete. 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a67af740 20 client.17667 trim_cache size 0 max 16384 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "0ba0e335-cd78-42e0-ac13-531ac4118e75", 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:34:42.249 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:34:42.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3880a2d00 msgr2=0x7fd3880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3880a2d00 0x7fd3880a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd398009ae0 tx=0x7fd398008040 comp rx=0 tx=0).stop 2024-04-25T01:34:42.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c1020d0 msgr2=0x7fd39c1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c1020d0 0x7fd39c1986b0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00b4a0 tx=0x7fd38c00b960 comp rx=0 tx=0).stop 2024-04-25T01:34:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3480663411 shutdown_connections 2024-04-25T01:34:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd39c107720 0x7fd39c19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd3880a2d00 0x7fd3880a51c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c1020d0 0x7fd39c1986b0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 --2- 172.21.15.27:0/3480663411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd39c101710 0x7fd39c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3480663411 >> 172.21.15.27:0/3480663411 conn(0x7fd39c0fd180 msgr2=0x7fd39c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3480663411 shutdown_connections 2024-04-25T01:34:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.268+0000 7fd3a39c1640 1 -- 172.21.15.27:0/3480663411 wait complete. 2024-04-25T01:34:42.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:34:42.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:34:42.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 -- 172.21.15.27:0/2468627613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac107730 msgr2=0x7f09ac107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 --2- 172.21.15.27:0/2468627613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac107730 0x7f09ac107b10 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009930 tx=0x7f09a002a420 comp rx=0 tx=0).stop 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 -- 172.21.15.27:0/2468627613 shutdown_connections 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 --2- 172.21.15.27:0/2468627613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac06a780 0x7f09ac10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 --2- 172.21.15.27:0/2468627613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09ac069dc0 0x7f09ac06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 --2- 172.21.15.27:0/2468627613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac107730 0x7f09ac107b10 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 -- 172.21.15.27:0/2468627613 >> 172.21.15.27:0/2468627613 conn(0x7f09ac0776a0 msgr2=0x7f09ac077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 -- 172.21.15.27:0/2468627613 shutdown_connections 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 -- 172.21.15.27:0/2468627613 wait complete. 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.383+0000 7f09b2012640 1 Processor -- start 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09b2012640 1 -- start start 2024-04-25T01:34:42.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09b2012640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac069dc0 0x7f09ac197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09b2012640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac06a780 0x7f09ac1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09b2012640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09ac107730 0x7f09ac19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09b2012640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac111b00 con 0x7f09ac069dc0 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09b2012640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac111980 con 0x7f09ac107730 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09b2012640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac111c80 con 0x7f09ac06a780 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09ab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac069dc0 0x7f09ac197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09ab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac069dc0 0x7f09ac197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55044/0 (socket says 172.21.15.27:55044) 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09ab7fe640 1 -- 172.21.15.27:0/835822451 learned_addr learned my addr 172.21.15.27:0/835822451 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09abfff640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09ac107730 0x7f09ac19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09aaffd640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac06a780 0x7f09ac1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09ab7fe640 1 -- 172.21.15.27:0/835822451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac06a780 msgr2=0x7f09ac1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac06a780 0x7f09ac1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09ab7fe640 1 -- 172.21.15.27:0/835822451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09ac107730 msgr2=0x7f09ac19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09ac107730 0x7f09ac19c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.384+0000 7f09ab7fe640 1 -- 172.21.15.27:0/835822451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac19cf20 con 0x7f09ac069dc0 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.385+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac069dc0 0x7f09ac197f80 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009230 tx=0x7f09a0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.385+0000 7f09a8ff9640 1 -- 172.21.15.27:0/835822451 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a003f030 con 0x7f09ac069dc0 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.385+0000 7f09a8ff9640 1 -- 172.21.15.27:0/835822451 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a0003ad0 con 0x7f09ac069dc0 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.385+0000 7f09b2012640 1 -- 172.21.15.27:0/835822451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ac19d1b0 con 0x7f09ac069dc0 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.385+0000 7f09a8ff9640 1 -- 172.21.15.27:0/835822451 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0035070 con 0x7f09ac069dc0 2024-04-25T01:34:42.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.385+0000 7f09b2012640 1 -- 172.21.15.27:0/835822451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ac1a4aa0 con 0x7f09ac069dc0 2024-04-25T01:34:42.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.387+0000 7f09a8ff9640 1 -- 172.21.15.27:0/835822451 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f09a0042030 con 0x7f09ac069dc0 2024-04-25T01:34:42.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.387+0000 7f09b4dfb740 1 Processor -- start 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.387+0000 7f09b4dfb740 1 -- start start 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a01245a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245f50 0x560a012500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01250620 0x560a01252a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560a011aa6b0 con 0x560a01245f50 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560a01152880 con 0x560a01245630 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560a011a6670 con 0x560a01250620 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245f50 0x560a012500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09a8ff9640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09940a2d00 0x7f09940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245f50 0x560a012500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55050/0 (socket says 172.21.15.27:55050) 2024-04-25T01:34:42.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 -- 172.21.15.27:0/2680803636 learned_addr learned my addr 172.21.15.27:0/2680803636 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09abfff640 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a01245a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09aaffd640 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01250620 0x560a01252a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09abfff640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09940a2d00 0x7f09940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09a8ff9640 1 -- 172.21.15.27:0/835822451 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f09a00efea0 con 0x7f09ac069dc0 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 -- 172.21.15.27:0/2680803636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01250620 msgr2=0x560a01252a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01250620 0x560a01252a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 -- 172.21.15.27:0/2680803636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 msgr2=0x560a01245a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a01245a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 -- 172.21.15.27:0/2680803636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a00f887d0 con 0x560a01245f50 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09abfff640 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a01245a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245f50 0x560a012500e0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f09a00b8980 tx=0x7f09a0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69292798f52a7157 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.388+0000 7f09abfff640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09940a2d00 0x7f09940a51c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f09a4007fd0 tx=0x7f09a4007520 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f0991ffb640 1 -- 172.21.15.27:0/2680803636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0049070 con 0x560a01245f50 2024-04-25T01:34:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f0991ffb640 1 -- 172.21.15.27:0/2680803636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a00bc6b0 con 0x560a01245f50 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f0991ffb640 1 -- 172.21.15.27:0/2680803636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a00b8620 con 0x560a01245f50 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2680803636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245f50 msgr2=0x560a012500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245f50 0x560a012500e0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f09a00b8980 tx=0x7f09a0009260 comp rx=0 tx=0).stop 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2680803636 shutdown_connections 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01250620 0x560a01252a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245f50 0x560a012500e0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2680803636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a01245a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2680803636 >> 172.21.15.27:0/2680803636 conn(0x560a01241950 msgr2=0x560a01241d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2680803636 shutdown_connections 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.389+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2680803636 wait complete. 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 1 -- start start 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 1 Processor -- start 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0124d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a0124dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560a012531d0 con 0x560a01250620 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x560a01165f50 con 0x560a01245630 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560a01253350 con 0x560a01245f50 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09b4dfb740 10 client.0 _ll_register_callbacks cb 0x560a011a1000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55052/0 (socket says 172.21.15.27:55052) 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.390+0000 7f09abfff640 1 -- 172.21.15.27:0/2263732764 learned_addr learned my addr 172.21.15.27:0/2263732764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0124d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09aaffd640 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a0124dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09abfff640 1 -- 172.21.15.27:0/2263732764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 msgr2=0x560a0124dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09abfff640 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a0124dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09abfff640 1 -- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 msgr2=0x560a0124d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09abfff640 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0124d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09abfff640 1 -- 172.21.15.27:0/2263732764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a00ebe3f0 con 0x560a01250620 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0124d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09abfff640 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124e000 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f09a4003f00 tx=0x7f09a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb71a7ad37e6ff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f098affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f098affd640 1 -- 172.21.15.27:0/2263732764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4025070 con 0x560a01250620 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f098affd640 1 -- 172.21.15.27:0/2263732764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f09a4004d20 con 0x560a01250620 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2263732764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a00bc6b0 con 0x560a01250620 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f098affd640 1 -- 172.21.15.27:0/2263732764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f09a4030050 con 0x560a01250620 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09b4dfb740 10 client.17679 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2263732764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560a00e7e0f0 con 0x560a01250620 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f098affd640 1 -- 172.21.15.27:0/2263732764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a00bc6b0 con 0x560a01250620 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.391+0000 7f098affd640 1 -- 172.21.15.27:0/2263732764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f09a4005140 con 0x560a01250620 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f09b4dfb740 10 client.17679 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f09b4dfb740 10 client.17679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f09b4dfb740 20 client.17679 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a01260630 0x560a01262af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f09b4dfb740 4 client.17679 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2263732764 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a011b7bf0 con 0x560a01260630 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a01260630 0x560a01262af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a01260630 0x560a01262af0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad007b01c1181194 server_cookie=f9204e238b560f3 in_seq=0 out_seq=0 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.392+0000 7f098affd640 10 client.17679 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f098affd640 1 -- 172.21.15.27:0/2263732764 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a011b7bf0 con 0x560a01260630 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f098affd640 10 client.17679 handle_command_reply: tid=0 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 client.17679 shutdown 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2263732764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 msgr2=0x560a0124e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124e000 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f09a4003f00 tx=0x7f09a4004a40 comp rx=0 tx=0).stop 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2263732764 shutdown_connections 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a01260630 0x560a01262af0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124e000 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a0124dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f098affd640 0 client.17679 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2263732764 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0124d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2263732764 >> 172.21.15.27:0/2263732764 conn(0x560a01241950 msgr2=0x560a01247610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2263732764 shutdown_connections 2024-04-25T01:34:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.393+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2263732764 wait complete. 2024-04-25T01:34:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.394+0000 7f09b4dfb740 20 client.17679 trim_cache size 0 max 16384 2024-04-25T01:34:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09b4dfb740 1 Processor -- start 2024-04-25T01:34:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09b4dfb740 1 -- start start 2024-04-25T01:34:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0125bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a01241270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac1b1580 con 0x560a01250620 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac1aee50 con 0x560a01245630 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac1accc0 con 0x560a01245f50 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55054/0 (socket says 172.21.15.27:55054) 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.411+0000 7f09abfff640 1 -- 172.21.15.27:0/2612276122 learned_addr learned my addr 172.21.15.27:0/2612276122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2612276122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0125bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09abfff640 1 -- 172.21.15.27:0/2612276122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 msgr2=0x560a01241270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09abfff640 1 --2- 172.21.15.27:0/2612276122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a01241270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09abfff640 1 -- 172.21.15.27:0/2612276122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 msgr2=0x560a0125bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09abfff640 1 --2- 172.21.15.27:0/2612276122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0125bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09abfff640 1 -- 172.21.15.27:0/2612276122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a01233090 con 0x560a01250620 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09abfff640 1 --2- 172.21.15.27:0/2612276122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124dc50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f09a4000f80 tx=0x7f09a4007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76781d42b5b38022 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09917fa640 1 -- 172.21.15.27:0/2612276122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4039070 con 0x560a01250620 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09917fa640 1 -- 172.21.15.27:0/2612276122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a4005750 con 0x560a01250620 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.412+0000 7f09917fa640 1 -- 172.21.15.27:0/2612276122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4025040 con 0x560a01250620 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2612276122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 msgr2=0x560a0124dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2612276122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124dc50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f09a4000f80 tx=0x7f09a4007a30 comp rx=0 tx=0).stop 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2612276122 shutdown_connections 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2612276122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01250620 0x560a0124dc50 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2612276122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a01241270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2612276122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01245630 0x560a0125bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2612276122 >> 172.21.15.27:0/2612276122 conn(0x560a01241950 msgr2=0x560a0124d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2612276122 shutdown_connections 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2612276122 wait complete. 2024-04-25T01:34:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- start start 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 Processor -- start 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245630 0x560a01258940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a01253cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01250620 0x560a01254250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac1ad310 con 0x560a01245630 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac1ae780 con 0x560a01250620 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.413+0000 7f09b4dfb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac1ad9e0 con 0x560a01245f50 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09b4dfb740 10 client.0 _ll_register_callbacks cb 0x7f09a4007c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245630 0x560a01258940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245630 0x560a01258940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55056/0 (socket says 172.21.15.27:55056) 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 -- 172.21.15.27:0/2461755148 learned_addr learned my addr 172.21.15.27:0/2461755148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09abfff640 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01250620 0x560a01254250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09aaffd640 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a01253cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 -- 172.21.15.27:0/2461755148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 msgr2=0x560a01253cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a01253cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 -- 172.21.15.27:0/2461755148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01250620 msgr2=0x560a01254250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01250620 0x560a01254250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 -- 172.21.15.27:0/2461755148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a01254a80 con 0x560a01245630 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09ab7fe640 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245630 0x560a01258940 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f09a00bbf40 tx=0x7f09a00bbe30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452aa3f8d03cdce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f098affd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f098affd640 1 -- 172.21.15.27:0/2461755148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0049040 con 0x560a01245630 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f098affd640 1 -- 172.21.15.27:0/2461755148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f09a0003c80 con 0x560a01245630 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2461755148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a4005750 con 0x560a01245630 2024-04-25T01:34:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f098affd640 1 -- 172.21.15.27:0/2461755148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f09a00bc6b0 con 0x560a01245630 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09b4dfb740 10 client.17685 fetch_fsmap learned FSMap version 27 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2461755148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560a012c6d70 con 0x560a01245630 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.414+0000 7f098affd640 1 -- 172.21.15.27:0/2461755148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a4005750 con 0x560a01245630 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f098affd640 1 -- 172.21.15.27:0/2461755148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f09a00bf040 con 0x560a01245630 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f09b4dfb740 10 client.17685 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f09b4dfb740 10 client.17685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f09b4dfb740 20 client.17685 populate_metadata read hostname 'smithi027' 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a012c7300 0x560a012c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f09b4dfb740 4 client.17685 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2461755148 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560a011b1f40 con 0x560a012c7300 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f09aaffd640 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a012c7300 0x560a012c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f098affd640 10 client.17685 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f09aaffd640 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a012c7300 0x560a012c96f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d439db9ff41fc66e server_cookie=eb4a834ea4bee3ad in_seq=0 out_seq=0 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.415+0000 7f098affd640 1 -- 172.21.15.27:0/2461755148 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560a011b1f40 con 0x560a012c7300 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f098affd640 10 client.17685 handle_command_reply: tid=0 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 client.17685 shutdown 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2461755148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245630 msgr2=0x560a01258940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245630 0x560a01258940 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f09a00bbf40 tx=0x7f09a00bbe30 comp rx=0 tx=0).stop 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2461755148 shutdown_connections 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x560a012c7300 0x560a012c96f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560a01250620 0x560a01254250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f098affd640 0 client.17685 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a01245f50 0x560a01253cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 --2- 172.21.15.27:0/2461755148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560a01245630 0x560a01258940 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2461755148 >> 172.21.15.27:0/2461755148 conn(0x560a01241950 msgr2=0x560a0124ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2461755148 shutdown_connections 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 1 -- 172.21.15.27:0/2461755148 wait complete. 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b4dfb740 20 client.17685 trim_cache size 0 max 16384 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stdout: "0ba0e335-cd78-42e0-ac13-531ac4118e75": { 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:34:42.401 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "0ba0e335-cd78-42e0-ac13-531ac4118e75", 2024-04-25T01:34:42.402 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:34:42.402 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:34:42.402 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:34:42.402 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:34:42.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 -- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09940a2d00 msgr2=0x7f09940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09940a2d00 0x7f09940a51c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f09a4007fd0 tx=0x7f09a4007520 comp rx=0 tx=0).stop 2024-04-25T01:34:42.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 -- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac069dc0 msgr2=0x7f09ac197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac069dc0 0x7f09ac197f80 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009230 tx=0x7f09a0004270 comp rx=0 tx=0).stop 2024-04-25T01:34:42.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 -- 172.21.15.27:0/835822451 shutdown_connections 2024-04-25T01:34:42.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09ac107730 0x7f09ac19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac06a780 0x7f09ac1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f09940a2d00 0x7f09940a51c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 --2- 172.21.15.27:0/835822451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09ac069dc0 0x7f09ac197f80 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.416+0000 7f09b2012640 1 -- 172.21.15.27:0/835822451 >> 172.21.15.27:0/835822451 conn(0x7f09ac0776a0 msgr2=0x7f09ac0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.417+0000 7f09b2012640 1 -- 172.21.15.27:0/835822451 shutdown_connections 2024-04-25T01:34:42.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:34:42.417+0000 7f09b2012640 1 -- 172.21.15.27:0/835822451 wait complete. 2024-04-25T01:34:42.404 INFO:tasks.cephfs.filesystem:scrub status for tag:0ba0e335-cd78-42e0-ac13-531ac4118e75 - {'path': '/', 'tag': '0ba0e335-cd78-42e0-ac13-531ac4118e75', 'options': 'recursive,force'} 2024-04-25T01:34:42.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:42 smithi027 ceph-mon[29197]: pgmap v287: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:42.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:42 smithi153 ceph-mon[32621]: pgmap v287: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:42.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:42 smithi139 ceph-mon[32760]: pgmap v287: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:43.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:43 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:34:43.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:43 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:43.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:43 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:34:43.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:43 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:34:43.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:43 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:43.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:43 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:43.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:43 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:34:43.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:43 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:43.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:43 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:34:43.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:43 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:34:43.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:43 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:43.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:43 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:43 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:34:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:43 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:43 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:34:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:43 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:34:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:43 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:43.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:43 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:44.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:44 smithi153 ceph-mon[32621]: pgmap v288: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:44.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:44 smithi139 ceph-mon[32760]: pgmap v288: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:44.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:44 smithi027 ceph-mon[29197]: pgmap v288: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:46.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:46 smithi153 ceph-mon[32621]: pgmap v289: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:46.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:46 smithi139 ceph-mon[32760]: pgmap v289: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:46.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:46 smithi027 ceph-mon[29197]: pgmap v289: 129 pgs: 129 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2024-04-25T01:34:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:48 smithi153 ceph-mon[32621]: pgmap v290: 129 pgs: 129 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:48.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:48 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:34:48.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:48 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:34:48.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:48 smithi139 ceph-mon[32760]: pgmap v290: 129 pgs: 129 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:48.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:48 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:34:48.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:48 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:34:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:48 smithi027 ceph-mon[29197]: pgmap v290: 129 pgs: 129 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:48.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:48 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:34:48.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:48 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:34:50.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:50 smithi153 ceph-mon[32621]: pgmap v291: 129 pgs: 129 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:34:50.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:50 smithi139 ceph-mon[32760]: pgmap v291: 129 pgs: 129 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:34:50.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:50 smithi027 ceph-mon[29197]: pgmap v291: 129 pgs: 129 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T01:34:52.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:52 smithi153 ceph-mon[32621]: pgmap v292: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:52 smithi139 ceph-mon[32760]: pgmap v292: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:52 smithi027 ceph-mon[29197]: pgmap v292: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:54.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:54 smithi153 ceph-mon[32621]: pgmap v293: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:54.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:54 smithi139 ceph-mon[32760]: pgmap v293: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:54.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:54 smithi027 ceph-mon[29197]: pgmap v293: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:56.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:56 smithi153 ceph-mon[32621]: pgmap v294: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:56.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:56 smithi139 ceph-mon[32760]: pgmap v294: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:56.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:56 smithi027 ceph-mon[29197]: pgmap v294: 129 pgs: 129 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:34:58.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:34:58 smithi153 ceph-mon[32621]: pgmap v295: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:58.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:34:58 smithi139 ceph-mon[32760]: pgmap v295: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:34:58.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:34:58 smithi027 ceph-mon[29197]: pgmap v295: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:35:00.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:00 smithi153 ceph-mon[32621]: pgmap v296: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.3 KiB/s wr, 9 op/s 2024-04-25T01:35:00.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:00 smithi139 ceph-mon[32760]: pgmap v296: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.3 KiB/s wr, 9 op/s 2024-04-25T01:35:00.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:00 smithi027 ceph-mon[29197]: pgmap v296: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.3 KiB/s wr, 9 op/s 2024-04-25T01:35:02.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:02 smithi153 ceph-mon[32621]: pgmap v297: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-04-25T01:35:02.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:02 smithi139 ceph-mon[32760]: pgmap v297: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-04-25T01:35:02.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:02 smithi027 ceph-mon[29197]: pgmap v297: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-04-25T01:35:04.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:04 smithi153 ceph-mon[32621]: pgmap v298: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:35:04.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:04 smithi139 ceph-mon[32760]: pgmap v298: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:35:04.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:04 smithi027 ceph-mon[29197]: pgmap v298: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:35:06.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:06 smithi153 ceph-mon[32621]: pgmap v299: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:35:06.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:06 smithi139 ceph-mon[32760]: pgmap v299: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:35:06.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:06 smithi027 ceph-mon[29197]: pgmap v299: 129 pgs: 129 active+clean; 4.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:35:07.281 INFO:tasks.workunit.client.0.smithi027.stdout:437/600: open fd = 3 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:438/600: open fd = 3 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:439/600: open fd = 3 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.282 INFO:tasks.workunit.client.0.smithi027.stdout:440/600: open fd = 3 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:441/600: open fd = 3 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:442/600: open fd = 3 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:443/600: open fd = 3 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.283 INFO:tasks.workunit.client.0.smithi027.stdout:444/600: open fd = 3 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:445/600: open fd = 3 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:446/600: open fd = 3 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:447/600: open fd = 3 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.284 INFO:tasks.workunit.client.0.smithi027.stdout:448/600: open fd = 3 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:449/600: open fd = 3 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:450/600: open fd = 3 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:451/600: open fd = 3 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:452/600: open fd = 3 2024-04-25T01:35:07.285 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:453/600: open fd = 3 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:454/600: open fd = 3 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:455/600: open fd = 3 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.286 INFO:tasks.workunit.client.0.smithi027.stdout:456/600: open fd = 3 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:457/600: open fd = 3 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:458/600: open fd = 3 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:459/600: open fd = 3 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.287 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:460/600: open fd = 3 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:461/600: open fd = 3 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:462/600: open fd = 3 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:463/600: open fd = 3 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.288 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:464/600: open fd = 3 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:465/600: open fd = 3 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:466/600: open fd = 3 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:467/600: open fd = 3 2024-04-25T01:35:07.289 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:468/600: open fd = 3 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:469/600: open fd = 3 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:470/600: open fd = 3 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.290 INFO:tasks.workunit.client.0.smithi027.stdout:471/600: open fd = 3 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:472/600: open fd = 3 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:473/600: open fd = 3 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:474/600: open fd = 3 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:475/600: open fd = 3 2024-04-25T01:35:07.291 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:476/600: open fd = 3 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:477/600: open fd = 3 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:478/600: open fd = 3 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:479/600: open fd = 3 2024-04-25T01:35:07.292 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:480/600: open fd = 3 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:481/600: open fd = 3 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:482/600: open fd = 3 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.293 INFO:tasks.workunit.client.0.smithi027.stdout:483/600: open fd = 3 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:484/600: open fd = 3 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:485/600: open fd = 3 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:486/600: open fd = 3 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.294 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:487/600: open fd = 3 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:488/600: open fd = 3 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:489/600: open fd = 3 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:490/600: open fd = 3 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.295 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:491/600: open fd = 3 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:492/600: open fd = 3 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:493/600: open fd = 3 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:494/600: open fd = 3 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.296 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:495/600: open fd = 3 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:496/600: open fd = 3 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:497/600: open fd = 3 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:498/600: open fd = 3 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.297 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:35:07.298 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:35:08.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:08 smithi139 ceph-mon[32760]: pgmap v300: 129 pgs: 129 active+clean; 4.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 31 op/s 2024-04-25T01:35:08.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:08 smithi139 ceph-mon[32760]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T01:35:08.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:08 smithi027 ceph-mon[29197]: pgmap v300: 129 pgs: 129 active+clean; 4.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 31 op/s 2024-04-25T01:35:08.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:08 smithi027 ceph-mon[29197]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T01:35:09.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:08 smithi153 ceph-mon[32621]: pgmap v300: 129 pgs: 129 active+clean; 4.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 31 op/s 2024-04-25T01:35:09.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:08 smithi153 ceph-mon[32621]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T01:35:10.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:10 smithi139 ceph-mon[32760]: pgmap v302: 129 pgs: 129 active+clean; 4.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:35:10.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:10 smithi027 ceph-mon[29197]: pgmap v302: 129 pgs: 129 active+clean; 4.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:35:11.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:10 smithi153 ceph-mon[32621]: pgmap v302: 129 pgs: 129 active+clean; 4.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:35:12.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:35:12.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:35:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.545+0000 7fc00b15a640 1 -- 172.21.15.27:0/2764279780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 msgr2=0x7fc004106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.545+0000 7fc00b15a640 1 --2- 172.21.15.27:0/2764279780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 0x7fc004106a10 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009930 tx=0x7fbff802a420 comp rx=0 tx=0).stop 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.545+0000 7fc00b15a640 1 -- 172.21.15.27:0/2764279780 shutdown_connections 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.545+0000 7fc00b15a640 1 --2- 172.21.15.27:0/2764279780 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc004100fe0 0x7fc00410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.545+0000 7fc00b15a640 1 --2- 172.21.15.27:0/2764279780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc004100620 0x7fc004100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.545+0000 7fc00b15a640 1 --2- 172.21.15.27:0/2764279780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 0x7fc004106a10 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.545+0000 7fc00b15a640 1 -- 172.21.15.27:0/2764279780 >> 172.21.15.27:0/2764279780 conn(0x7fc0040776a0 msgr2=0x7fc004077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.545+0000 7fc00b15a640 1 -- 172.21.15.27:0/2764279780 shutdown_connections 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 -- 172.21.15.27:0/2764279780 wait complete. 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 Processor -- start 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 -- start start 2024-04-25T01:35:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc004100620 0x7fc004198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc004100fe0 0x7fc0041993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 0x7fc0041a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc004110a40 con 0x7fc004106630 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041108c0 con 0x7fc004100fe0 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc00b15a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc004110bc0 con 0x7fc004100620 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc0096f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 0x7fc0041a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc0096f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 0x7fc0041a3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39974/0 (socket says 172.21.15.27:39974) 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.546+0000 7fc0096f7640 1 -- 172.21.15.27:0/1678675691 learned_addr learned my addr 172.21.15.27:0/1678675691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc008ef6640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc004100620 0x7fc004198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc003fff640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc004100fe0 0x7fc0041993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc0096f7640 1 -- 172.21.15.27:0/1678675691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc004100620 msgr2=0x7fc004198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc0096f7640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc004100620 0x7fc004198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc0096f7640 1 -- 172.21.15.27:0/1678675691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc004100fe0 msgr2=0x7fc0041993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc0096f7640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc004100fe0 0x7fc0041993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc0096f7640 1 -- 172.21.15.27:0/1678675691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0041a4420 con 0x7fc004106630 2024-04-25T01:35:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc0096f7640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 0x7fc0041a3ee0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00eaa0 tx=0x7fbffc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc001ffb640 1 -- 172.21.15.27:0/1678675691 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00cdb0 con 0x7fc004106630 2024-04-25T01:35:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc001ffb640 1 -- 172.21.15.27:0/1678675691 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbffc005020 con 0x7fc004106630 2024-04-25T01:35:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc00b15a640 1 -- 172.21.15.27:0/1678675691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0041a45f0 con 0x7fc004106630 2024-04-25T01:35:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc001ffb640 1 -- 172.21.15.27:0/1678675691 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc0116b0 con 0x7fc004106630 2024-04-25T01:35:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.547+0000 7fc00b15a640 1 -- 172.21.15.27:0/1678675691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0041a4b40 con 0x7fc004106630 2024-04-25T01:35:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.549+0000 7fc001ffb640 1 -- 172.21.15.27:0/1678675691 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbffc005560 con 0x7fc004106630 2024-04-25T01:35:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc00de67740 1 Processor -- start 2024-04-25T01:35:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc001ffb640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbff40a2d80 0x7fbff40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc00de67740 1 -- start start 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc001ffb640 1 -- 172.21.15.27:0/1678675691 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7fbffc0cfa10 con 0x7fc004106630 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc003fff640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbff40a2d80 0x7fbff40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc00de67740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1630 0x5594d66d1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc00de67740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1f50 0x5594d66dc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc00de67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66dea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc00de67740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5594d66366b0 con 0x5594d66d1f50 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc00de67740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5594d65de880 con 0x5594d66d1630 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc00de67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5594d6632670 con 0x5594d66dc620 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.550+0000 7fc003fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1f50 0x5594d66dc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1f50 0x5594d66dc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39986/0 (socket says 172.21.15.27:39986) 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 -- 172.21.15.27:0/679435141 learned_addr learned my addr 172.21.15.27:0/679435141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc0096f7640 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc008ef6640 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1630 0x5594d66d1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbff40a2d80 0x7fbff40a5240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbff000bfd0 tx=0x7fbff00138e0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 -- 172.21.15.27:0/679435141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 msgr2=0x5594d66dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66dea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 -- 172.21.15.27:0/679435141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1630 msgr2=0x5594d66d1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1630 0x5594d66d1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 -- 172.21.15.27:0/679435141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594d64147d0 con 0x5594d66d1f50 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fc003fff640 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1f50 0x5594d66dc0e0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fbff0013de0 tx=0x7fbff001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5c203a4b219dfb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fbfeeffd640 1 -- 172.21.15.27:0/679435141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff001f7c0 con 0x5594d66d1f50 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fbfeeffd640 1 -- 172.21.15.27:0/679435141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff001fe40 con 0x5594d66d1f50 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.551+0000 7fbfeeffd640 1 -- 172.21.15.27:0/679435141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff001ee20 con 0x5594d66d1f50 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 -- 172.21.15.27:0/679435141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1f50 msgr2=0x5594d66dc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1f50 0x5594d66dc0e0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fbff0013de0 tx=0x7fbff001d000 comp rx=0 tx=0).stop 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 -- 172.21.15.27:0/679435141 shutdown_connections 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1f50 0x5594d66dc0e0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 --2- 172.21.15.27:0/679435141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1630 0x5594d66d1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 -- 172.21.15.27:0/679435141 >> 172.21.15.27:0/679435141 conn(0x5594d66cd950 msgr2=0x5594d66cdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 -- 172.21.15.27:0/679435141 shutdown_connections 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 -- 172.21.15.27:0/679435141 wait complete. 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 -- start start 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.552+0000 7fc00de67740 1 Processor -- start 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc00de67740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66d9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc00de67740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc00de67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc00de67740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5594d66df3b0 con 0x5594d66d1630 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc00de67740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5594d65f1f50 con 0x5594d66d1f50 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc00de67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5594d66df530 con 0x5594d66dc620 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc00de67740 10 client.0 _ll_register_callbacks cb 0x5594d662d000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc008ef6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66d9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc008ef6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66d9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39988/0 (socket says 172.21.15.27:39988) 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc008ef6640 1 -- 172.21.15.27:0/1015794043 learned_addr learned my addr 172.21.15.27:0/1015794043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc003fff640 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc0096f7640 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc008ef6640 1 -- 172.21.15.27:0/1015794043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 msgr2=0x5594d66d9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc008ef6640 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc008ef6640 1 -- 172.21.15.27:0/1015794043 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 msgr2=0x5594d66d9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc008ef6640 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.553+0000 7fc008ef6640 1 -- 172.21.15.27:0/1015794043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594d634a3f0 con 0x5594d66d1630 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fc008ef6640 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66d9530 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fbff8033000 tx=0x7fbff8031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08c35f4a8276ea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fbfe3fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1015794043 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8035040 con 0x5594d66d1630 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1015794043 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7fbff8004460 con 0x5594d66d1630 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fc00de67740 1 -- 172.21.15.27:0/1015794043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5594d6643bf0 con 0x5594d66d1630 2024-04-25T01:35:12.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1015794043 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbff804e050 con 0x5594d66d1630 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fc00de67740 10 client.17697 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fc00de67740 1 -- 172.21.15.27:0/1015794043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5594d630a0f0 con 0x5594d66d1630 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1015794043 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5594d6643bf0 con 0x5594d66d1630 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1015794043 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbff803d850 con 0x5594d66d1630 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fc00de67740 10 client.17697 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.554+0000 7fc00de67740 10 client.17697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.555+0000 7fc00de67740 20 client.17697 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.555+0000 7fc00de67740 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5594d66ec830 0x5594d66eec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.555+0000 7fc00de67740 4 client.17697 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.555+0000 7fc00de67740 1 -- 172.21.15.27:0/1015794043 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594d663df40 con 0x5594d66ec830 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.555+0000 7fc003fff640 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5594d66ec830 0x5594d66eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.555+0000 7fc003fff640 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5594d66ec830 0x5594d66eec60 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=540dc31fd6d52411 server_cookie=153b46f67872fe2f in_seq=0 out_seq=0 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.555+0000 7fbfe3fff640 10 client.17697 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1015794043 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5594d663df40 con 0x5594d66ec830 2024-04-25T01:35:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fbfe3fff640 10 client.17697 handle_command_reply: tid=0 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 client.17697 shutdown 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 -- 172.21.15.27:0/1015794043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 msgr2=0x5594d66d9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66d9530 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fbff8033000 tx=0x7fbff8031920 comp rx=0 tx=0).stop 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 -- 172.21.15.27:0/1015794043 shutdown_connections 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5594d66ec830 0x5594d66eec60 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 --2- 172.21.15.27:0/1015794043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66d9530 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 -- 172.21.15.27:0/1015794043 >> 172.21.15.27:0/1015794043 conn(0x5594d66cd950 msgr2=0x5594d66d3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fbfe3fff640 0 client.17697 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 -- 172.21.15.27:0/1015794043 shutdown_connections 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 1 -- 172.21.15.27:0/1015794043 wait complete. 2024-04-25T01:35:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.556+0000 7fc00de67740 20 client.17697 trim_cache size 0 max 16384 2024-04-25T01:35:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.574+0000 7fc00de67740 1 Processor -- start 2024-04-25T01:35:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.574+0000 7fc00de67740 1 -- start start 2024-04-25T01:35:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.574+0000 7fc00de67740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66e8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.574+0000 7fc00de67740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc00de67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc00de67740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041b1720 con 0x5594d66d1630 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc00de67740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041aeff0 con 0x5594d66d1f50 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc00de67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041ace60 con 0x5594d66dc620 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39998/0 (socket says 172.21.15.27:39998) 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 -- 172.21.15.27:0/3739146687 learned_addr learned my addr 172.21.15.27:0/3739146687 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc0096f7640 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 -- 172.21.15.27:0/3739146687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 msgr2=0x5594d66d8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 -- 172.21.15.27:0/3739146687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 msgr2=0x5594d66d3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc003fff640 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 -- 172.21.15.27:0/3739146687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594d66bf090 con 0x5594d66d1630 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc0096f7640 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc003fff640 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fc008ef6640 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66e8270 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fbff8005940 tx=0x7fbff8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61558aa4e8e4cdf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fbfee7fc640 1 -- 172.21.15.27:0/3739146687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8056070 con 0x5594d66d1630 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fbfee7fc640 1 -- 172.21.15.27:0/3739146687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff803ee50 con 0x5594d66d1630 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.575+0000 7fbfee7fc640 1 -- 172.21.15.27:0/3739146687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8035030 con 0x5594d66d1630 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 -- 172.21.15.27:0/3739146687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 msgr2=0x5594d66e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66e8270 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fbff8005940 tx=0x7fbff8005780 comp rx=0 tx=0).stop 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 -- 172.21.15.27:0/3739146687 shutdown_connections 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66dc620 0x5594d66d8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 --2- 172.21.15.27:0/3739146687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66d1630 0x5594d66e8270 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 -- 172.21.15.27:0/3739146687 >> 172.21.15.27:0/3739146687 conn(0x5594d66cd950 msgr2=0x5594d66d7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 -- 172.21.15.27:0/3739146687 shutdown_connections 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 -- 172.21.15.27:0/3739146687 wait complete. 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 -- start start 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.576+0000 7fc00de67740 1 Processor -- start 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc00de67740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66d1630 0x5594d66e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc00de67740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc00de67740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66dc620 0x5594d66d7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc00de67740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041ad4b0 con 0x5594d66dc620 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc00de67740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041ae920 con 0x5594d66d1f50 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc00de67740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041adb80 con 0x5594d66d1630 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc00de67740 10 client.0 _ll_register_callbacks cb 0x7fbff8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66dc620 0x5594d66d7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66dc620 0x5594d66d7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40000/0 (socket says 172.21.15.27:40000) 2024-04-25T01:35:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 -- 172.21.15.27:0/1307973973 learned_addr learned my addr 172.21.15.27:0/1307973973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 -- 172.21.15.27:0/1307973973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66d1630 msgr2=0x5594d66e4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66d1630 0x5594d66e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 -- 172.21.15.27:0/1307973973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 msgr2=0x5594d66d6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 -- 172.21.15.27:0/1307973973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594d66d5a60 con 0x5594d66dc620 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fc0096f7640 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66dc620 0x5594d66d7200 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0ccd80 tx=0x7fbffc00ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332f0a715df1d11c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.577+0000 7fbfe3fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1307973973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc0b4070 con 0x5594d66dc620 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1307973973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7fbffc0a42d0 con 0x5594d66dc620 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 1 -- 172.21.15.27:0/1307973973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbff803ee50 con 0x5594d66dc620 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1307973973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbffc01f070 con 0x5594d66dc620 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 10 client.17703 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 1 -- 172.21.15.27:0/1307973973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5594d6752270 con 0x5594d66dc620 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1307973973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff803ee50 con 0x5594d66dc620 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1307973973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbffc00f040 con 0x5594d66dc620 2024-04-25T01:35:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 10 client.17703 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 10 client.17703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 20 client.17703 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5594d6752990 0x5594d6754e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 4 client.17703 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.578+0000 7fc00de67740 1 -- 172.21.15.27:0/1307973973 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5594d65bb6f0 con 0x5594d6752990 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc008ef6640 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5594d6752990 0x5594d6754e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fbfe3fff640 10 client.17703 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc008ef6640 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5594d6752990 0x5594d6754e30 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66e5cce1d6464f79 server_cookie=fd8907fbe8b6190b in_seq=0 out_seq=0 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fbfe3fff640 1 -- 172.21.15.27:0/1307973973 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5594d65bb6f0 con 0x5594d6752990 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fbfe3fff640 10 client.17703 handle_command_reply: tid=0 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 client.17703 shutdown 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 -- 172.21.15.27:0/1307973973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66dc620 msgr2=0x5594d66d7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66dc620 0x5594d66d7200 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0ccd80 tx=0x7fbffc00ea20 comp rx=0 tx=0).stop 2024-04-25T01:35:12.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 -- 172.21.15.27:0/1307973973 shutdown_connections 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5594d6752990 0x5594d6754e30 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594d66dc620 0x5594d66d7200 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5594d66d1f50 0x5594d66d6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fbfe3fff640 0 client.17703 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 --2- 172.21.15.27:0/1307973973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5594d66d1630 0x5594d66e4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 -- 172.21.15.27:0/1307973973 >> 172.21.15.27:0/1307973973 conn(0x5594d66cd950 msgr2=0x5594d66ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.579+0000 7fc00de67740 1 -- 172.21.15.27:0/1307973973 shutdown_connections 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00de67740 1 -- 172.21.15.27:0/1307973973 wait complete. 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00de67740 20 client.17703 trim_cache size 0 max 16384 2024-04-25T01:35:12.562 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:35:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:35:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:35:12.563 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:35:12.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 -- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbff40a2d80 msgr2=0x7fbff40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbff40a2d80 0x7fbff40a5240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbff000bfd0 tx=0x7fbff00138e0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 -- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 msgr2=0x7fc0041a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 0x7fc0041a3ee0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00eaa0 tx=0x7fbffc00ef60 comp rx=0 tx=0).stop 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 -- 172.21.15.27:0/1678675691 shutdown_connections 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbff40a2d80 0x7fbff40a5240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc004106630 0x7fc0041a3ee0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc004100fe0 0x7fc0041993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 --2- 172.21.15.27:0/1678675691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc004100620 0x7fc004198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 -- 172.21.15.27:0/1678675691 >> 172.21.15.27:0/1678675691 conn(0x7fc0040776a0 msgr2=0x7fc004103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 -- 172.21.15.27:0/1678675691 shutdown_connections 2024-04-25T01:35:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.580+0000 7fc00b15a640 1 -- 172.21.15.27:0/1678675691 wait complete. 2024-04-25T01:35:12.568 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:35:12.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:35:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 -- 172.21.15.27:0/4100626529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb15410bd60 msgr2=0x7fb15410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 --2- 172.21.15.27:0/4100626529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb15410bd60 0x7fb15410e1d0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009ca0 tx=0x7fb14c02a950 comp rx=0 tx=0).stop 2024-04-25T01:35:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 -- 172.21.15.27:0/4100626529 shutdown_connections 2024-04-25T01:35:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 --2- 172.21.15.27:0/4100626529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb15410bd60 0x7fb15410e1d0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 --2- 172.21.15.27:0/4100626529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb154103340 0x7fb15410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 --2- 172.21.15.27:0/4100626529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb154102a20 0x7fb154102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 -- 172.21.15.27:0/4100626529 >> 172.21.15.27:0/4100626529 conn(0x7fb1540777a0 msgr2=0x7fb154077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 -- 172.21.15.27:0/4100626529 shutdown_connections 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.693+0000 7fb15a676640 1 -- 172.21.15.27:0/4100626529 wait complete. 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb15a676640 1 Processor -- start 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb15a676640 1 -- start start 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb15a676640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb154102a20 0x7fb154197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb15a676640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb154103340 0x7fb154198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb15a676640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb15410bd60 0x7fb15419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb15a676640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb154111b30 con 0x7fb154102a20 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb15a676640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541119b0 con 0x7fb15410bd60 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb15a676640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb154111cb0 con 0x7fb154103340 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb153fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb154102a20 0x7fb154197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb153fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb154102a20 0x7fb154197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40006/0 (socket says 172.21.15.27:40006) 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb153fff640 1 -- 172.21.15.27:0/3391482055 learned_addr learned my addr 172.21.15.27:0/3391482055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb158c13640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb15410bd60 0x7fb15419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb1537fe640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb154103340 0x7fb154198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb153fff640 1 -- 172.21.15.27:0/3391482055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb154103340 msgr2=0x7fb154198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb153fff640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb154103340 0x7fb154198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb153fff640 1 -- 172.21.15.27:0/3391482055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb15410bd60 msgr2=0x7fb15419c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb153fff640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb15410bd60 0x7fb15419c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb153fff640 1 -- 172.21.15.27:0/3391482055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15419cf20 con 0x7fb154102a20 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.694+0000 7fb1537fe640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb154103340 0x7fb154198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.695+0000 7fb153fff640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb154102a20 0x7fb154197f30 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00c920 tx=0x7fb13c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.695+0000 7fb1517fa640 1 -- 172.21.15.27:0/3391482055 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c007c70 con 0x7fb154102a20 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.695+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb15419d210 con 0x7fb154102a20 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.695+0000 7fb1517fa640 1 -- 172.21.15.27:0/3391482055 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb13c007e10 con 0x7fb154102a20 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.695+0000 7fb1517fa640 1 -- 172.21.15.27:0/3391482055 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c018890 con 0x7fb154102a20 2024-04-25T01:35:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.695+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1541a4aa0 con 0x7fb154102a20 2024-04-25T01:35:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.696+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1540689b0 con 0x7fb154102a20 2024-04-25T01:35:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.697+0000 7fb1517fa640 1 -- 172.21.15.27:0/3391482055 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb13c018a50 con 0x7fb154102a20 2024-04-25T01:35:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.698+0000 7fb1517fa640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1400a2d00 0x7fb1400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.698+0000 7fb1537fe640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1400a2d00 0x7fb1400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.698+0000 7fb1517fa640 1 -- 172.21.15.27:0/3391482055 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7fb13c0cfc70 con 0x7fb154102a20 2024-04-25T01:35:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.698+0000 7fb1537fe640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1400a2d00 0x7fb1400a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb144005fd0 tx=0x7fb1440094d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:12.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.701+0000 7fb1517fa640 1 -- 172.21.15.27:0/3391482055 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb13c014030 con 0x7fb154102a20 2024-04-25T01:35:12.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:12 smithi027 ceph-mon[29197]: pgmap v303: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:12.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:12 smithi139 ceph-mon[32760]: pgmap v303: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:12.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.926+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1541a4d30 con 0x7fb154102a20 2024-04-25T01:35:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.927+0000 7fb1517fa640 1 -- 172.21.15.27:0/3391482055 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7fb13c09c4e0 con 0x7fb154102a20 2024-04-25T01:35:12.904 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:35:12.904 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:35:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:35:12.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1400a2d00 msgr2=0x7fb1400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1400a2d00 0x7fb1400a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb144005fd0 tx=0x7fb1440094d0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb154102a20 msgr2=0x7fb154197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:12.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb154102a20 0x7fb154197f30 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00c920 tx=0x7fb13c00cde0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 shutdown_connections 2024-04-25T01:35:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb15410bd60 0x7fb15419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb154103340 0x7fb154198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1400a2d00 0x7fb1400a51c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 --2- 172.21.15.27:0/3391482055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb154102a20 0x7fb154197f30 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 >> 172.21.15.27:0/3391482055 conn(0x7fb1540777a0 msgr2=0x7fb1540fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 shutdown_connections 2024-04-25T01:35:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:12.929+0000 7fb15a676640 1 -- 172.21.15.27:0/3391482055 wait complete. 2024-04-25T01:35:12.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:35:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 -- 172.21.15.27:0/1760652145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d41020d0 msgr2=0x7f53d410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 --2- 172.21.15.27:0/1760652145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d41020d0 0x7f53d410e5e0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009cd0 tx=0x7f53cc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 -- 172.21.15.27:0/1760652145 shutdown_connections 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 --2- 172.21.15.27:0/1760652145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d41020d0 0x7f53d410e5e0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 --2- 172.21.15.27:0/1760652145 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53d4101710 0x7f53d4101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 --2- 172.21.15.27:0/1760652145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d4107720 0x7f53d4107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 -- 172.21.15.27:0/1760652145 >> 172.21.15.27:0/1760652145 conn(0x7f53d40fd180 msgr2=0x7f53d40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 -- 172.21.15.27:0/1760652145 shutdown_connections 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.044+0000 7f53db34f640 1 -- 172.21.15.27:0/1760652145 wait complete. 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53db34f640 1 Processor -- start 2024-04-25T01:35:13.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53db34f640 1 -- start start 2024-04-25T01:35:13.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:12 smithi153 ceph-mon[32621]: pgmap v303: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53db34f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d4101710 0x7f53d419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53db34f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d41020d0 0x7f53d419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53db34f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53d4107720 0x7f53d419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53db34f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53d4111b30 con 0x7f53d4101710 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53db34f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53d41119b0 con 0x7f53d4107720 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53db34f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53d4111cb0 con 0x7f53d41020d0 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53d90eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d4101710 0x7f53d419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53d90eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d4101710 0x7f53d419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40022/0 (socket says 172.21.15.27:40022) 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53d90eb640 1 -- 172.21.15.27:0/14740562 learned_addr learned my addr 172.21.15.27:0/14740562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53d88ea640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d41020d0 0x7f53d419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.045+0000 7f53d98ec640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53d4107720 0x7f53d419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53d90eb640 1 -- 172.21.15.27:0/14740562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d41020d0 msgr2=0x7f53d419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53d90eb640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d41020d0 0x7f53d419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53d90eb640 1 -- 172.21.15.27:0/14740562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53d4107720 msgr2=0x7f53d419eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53d90eb640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53d4107720 0x7f53d419eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53d90eb640 1 -- 172.21.15.27:0/14740562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d419f1f0 con 0x7f53d4101710 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53d90eb640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d4101710 0x7f53d419a200 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f53d000c970 tx=0x7f53d000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53c67fc640 1 -- 172.21.15.27:0/14740562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0007bb0 con 0x7f53d4101710 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53db34f640 1 -- 172.21.15.27:0/14740562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d419f4e0 con 0x7f53d4101710 2024-04-25T01:35:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53c67fc640 1 -- 172.21.15.27:0/14740562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53d0007d50 con 0x7f53d4101710 2024-04-25T01:35:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53c67fc640 1 -- 172.21.15.27:0/14740562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d000f450 con 0x7f53d4101710 2024-04-25T01:35:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.046+0000 7f53db34f640 1 -- 172.21.15.27:0/14740562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d41a6d70 con 0x7f53d4101710 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.048+0000 7f53c67fc640 1 -- 172.21.15.27:0/14740562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f53d0005020 con 0x7f53d4101710 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.048+0000 7f53de059740 1 Processor -- start 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53de059740 1 -- start start 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53de059740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bb9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53de059740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53de059740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53de059740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d354b1e670 con 0x55d354bb9ea0 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53de059740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d354ac6970 con 0x55d354bb9580 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53de059740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d354b1a630 con 0x55d354bc4570 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d90eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d90eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40036/0 (socket says 172.21.15.27:40036) 2024-04-25T01:35:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d90eb640 1 -- 172.21.15.27:0/3096444169 learned_addr learned my addr 172.21.15.27:0/3096444169 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d98ec640 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bb9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d88ea640 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53c67fc640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53c00a2d00 0x7f53c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d98ec640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53c00a2d00 0x7f53c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53c67fc640 1 -- 172.21.15.27:0/14740562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f53d00cf570 con 0x7f53d4101710 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d90eb640 1 -- 172.21.15.27:0/3096444169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 msgr2=0x55d354bc6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d90eb640 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d90eb640 1 -- 172.21.15.27:0/3096444169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 msgr2=0x55d354bb9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d90eb640 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bb9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d90eb640 1 -- 172.21.15.27:0/3096444169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d354aea400 con 0x55d354bb9ea0 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.049+0000 7f53d98ec640 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bb9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.050+0000 7f53d90eb640 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc4030 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f53d00cc7f0 tx=0x7f53d0004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298ae3d714bd8f0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.050+0000 7f53bf7fe640 1 -- 172.21.15.27:0/3096444169 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d001d070 con 0x55d354bb9ea0 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.050+0000 7f53d98ec640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53c00a2d00 0x7f53c00a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f53cc004ab0 tx=0x7f53cc02a8e0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.050+0000 7f53bf7fe640 1 -- 172.21.15.27:0/3096444169 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53d00d2cd0 con 0x55d354bb9ea0 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.050+0000 7f53bf7fe640 1 -- 172.21.15.27:0/3096444169 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d00a42c0 con 0x55d354bb9ea0 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.050+0000 7f53de059740 1 -- 172.21.15.27:0/3096444169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 msgr2=0x55d354bc4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.050+0000 7f53de059740 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc4030 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f53d00cc7f0 tx=0x7f53d0004a20 comp rx=0 tx=0).stop 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 -- 172.21.15.27:0/3096444169 shutdown_connections 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc4030 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 --2- 172.21.15.27:0/3096444169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bb9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 -- 172.21.15.27:0/3096444169 >> 172.21.15.27:0/3096444169 conn(0x55d354bb58a0 msgr2=0x55d354bb5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 -- 172.21.15.27:0/3096444169 shutdown_connections 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 -- 172.21.15.27:0/3096444169 wait complete. 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 -- start start 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 Processor -- start 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bc3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d354bc7300 con 0x55d354bb9ea0 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d354ada050 con 0x55d354bb9580 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d354bc7480 con 0x55d354bc4570 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53de059740 10 client.0 _ll_register_callbacks cb 0x55d354831be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.051+0000 7f53d88ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d88ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40050/0 (socket says 172.21.15.27:40050) 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d88ea640 1 -- 172.21.15.27:0/2048174621 learned_addr learned my addr 172.21.15.27:0/2048174621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d98ec640 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d88ea640 1 -- 172.21.15.27:0/2048174621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 msgr2=0x55d354bc17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d88ea640 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d90eb640 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bc3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d88ea640 1 -- 172.21.15.27:0/2048174621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 msgr2=0x55d354bc3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d88ea640 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bc3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d88ea640 1 -- 172.21.15.27:0/2048174621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d354aabdc0 con 0x55d354bb9ea0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d90eb640 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bc3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53d88ea640 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc1200 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f53c800bad0 tx=0x7f53c800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68896f518bb29588 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53bcff9640 1 -- 172.21.15.27:0/2048174621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8003e20 con 0x55d354bb9ea0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53bcff9640 1 -- 172.21.15.27:0/2048174621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f53c8011440 con 0x55d354bb9ea0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53de059740 1 -- 172.21.15.27:0/2048174621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d354b2bbb0 con 0x55d354bb9ea0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53bcff9640 1 -- 172.21.15.27:0/2048174621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f53c8005020 con 0x55d354bb9ea0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53de059740 10 client.17721 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.052+0000 7f53de059740 1 -- 172.21.15.27:0/2048174621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d3548323f0 con 0x55d354bb9ea0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53bcff9640 1 -- 172.21.15.27:0/2048174621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d354b2bbb0 con 0x55d354bb9ea0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53bcff9640 1 -- 172.21.15.27:0/2048174621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f53c8011640 con 0x55d354bb9ea0 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53de059740 10 client.17721 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53de059740 10 client.17721 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53de059740 20 client.17721 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53de059740 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55d354bd4650 0x55d354bd6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53de059740 4 client.17721 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53de059740 1 -- 172.21.15.27:0/2048174621 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d354b25f00 con 0x55d354bd4650 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.053+0000 7f53d90eb640 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55d354bd4650 0x55d354bd6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.054+0000 7f53d90eb640 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55d354bd4650 0x55d354bd6af0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=469c09a2c20a3f7f server_cookie=1ac9ad2af1e96c8f in_seq=0 out_seq=0 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.054+0000 7f53bcff9640 10 client.17721 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.054+0000 7f53bcff9640 1 -- 172.21.15.27:0/2048174621 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d354b25f00 con 0x55d354bd4650 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.054+0000 7f53bcff9640 10 client.17721 handle_command_reply: tid=0 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.054+0000 7f53de059740 1 client.17721 shutdown 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 -- 172.21.15.27:0/2048174621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 msgr2=0x55d354bc1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc1200 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f53c800bad0 tx=0x7f53c800bf90 comp rx=0 tx=0).stop 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 -- 172.21.15.27:0/2048174621 shutdown_connections 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55d354bd4650 0x55d354bd6af0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9ea0 0x55d354bc1200 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 --2- 172.21.15.27:0/2048174621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9580 0x55d354bc3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53bcff9640 0 client.17721 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:35:13.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 -- 172.21.15.27:0/2048174621 >> 172.21.15.27:0/2048174621 conn(0x55d354bb58a0 msgr2=0x55d354bb4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 -- 172.21.15.27:0/2048174621 shutdown_connections 2024-04-25T01:35:13.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 1 -- 172.21.15.27:0/2048174621 wait complete. 2024-04-25T01:35:13.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.055+0000 7f53de059740 20 client.17721 trim_cache size 0 max 16384 2024-04-25T01:35:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53de059740 1 Processor -- start 2024-04-25T01:35:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53de059740 1 -- start start 2024-04-25T01:35:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53de059740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bb9580 0x55d354bd01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53de059740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53de059740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bc4570 0x55d354bdc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53de059740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53d41b39e0 con 0x55d354bc4570 2024-04-25T01:35:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53de059740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53d41b12b0 con 0x55d354bb9ea0 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53de059740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53d41af120 con 0x55d354bb9580 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.068+0000 7f53d98ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bc4570 0x55d354bdc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d98ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bc4570 0x55d354bdc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40064/0 (socket says 172.21.15.27:40064) 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d98ec640 1 -- 172.21.15.27:0/2944808450 learned_addr learned my addr 172.21.15.27:0/2944808450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d98ec640 1 -- 172.21.15.27:0/2944808450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bb9580 msgr2=0x55d354bd01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d98ec640 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bb9580 0x55d354bd01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d88ea640 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d98ec640 1 -- 172.21.15.27:0/2944808450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 msgr2=0x55d354bc3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d98ec640 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d98ec640 1 -- 172.21.15.27:0/2944808450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3548fc7d0 con 0x55d354bc4570 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d88ea640 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53d98ec640 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bc4570 0x55d354bdc3c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009cd0 tx=0x7f53cc005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44fb3a907f9afed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53beffd640 1 -- 172.21.15.27:0/2944808450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc004690 con 0x55d354bc4570 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53beffd640 1 -- 172.21.15.27:0/2944808450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53cc036050 con 0x55d354bc4570 2024-04-25T01:35:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.069+0000 7f53beffd640 1 -- 172.21.15.27:0/2944808450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc04a7b0 con 0x55d354bc4570 2024-04-25T01:35:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 -- 172.21.15.27:0/2944808450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bc4570 msgr2=0x55d354bdc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bc4570 0x55d354bdc3c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009cd0 tx=0x7f53cc005fa0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 -- 172.21.15.27:0/2944808450 shutdown_connections 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bc4570 0x55d354bdc3c0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 --2- 172.21.15.27:0/2944808450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bb9580 0x55d354bd01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 -- 172.21.15.27:0/2944808450 >> 172.21.15.27:0/2944808450 conn(0x55d354bb58a0 msgr2=0x55d354bbb560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 -- 172.21.15.27:0/2944808450 shutdown_connections 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 -- 172.21.15.27:0/2944808450 wait complete. 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 -- start start 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.070+0000 7f53de059740 1 Processor -- start 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53de059740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9580 0x55d354bc81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53de059740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53de059740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53de059740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53d41af770 con 0x55d354bb9580 2024-04-25T01:35:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53de059740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f53d41b0be0 con 0x55d354bb9ea0 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53de059740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53d41afe40 con 0x55d354bc4570 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53de059740 10 client.0 _ll_register_callbacks cb 0x7f53c8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9580 0x55d354bc81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9580 0x55d354bc81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40068/0 (socket says 172.21.15.27:40068) 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 -- 172.21.15.27:0/1900479215 learned_addr learned my addr 172.21.15.27:0/1900479215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d98ec640 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d88ea640 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 -- 172.21.15.27:0/1900479215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 msgr2=0x55d354bc8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 -- 172.21.15.27:0/1900479215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 msgr2=0x55d354bc8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 -- 172.21.15.27:0/1900479215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d354c39d30 con 0x55d354bb9580 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d88ea640 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53d90eb640 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9580 0x55d354bc81e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f53d00cce80 tx=0x7f53d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3e08d77c78493f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53bcff9640 1 -- 172.21.15.27:0/1900479215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d001d040 con 0x55d354bb9580 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53bcff9640 1 -- 172.21.15.27:0/1900479215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f53d00ccb00 con 0x55d354bb9580 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.071+0000 7f53de059740 1 -- 172.21.15.27:0/1900479215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53cc036050 con 0x55d354bb9580 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53bcff9640 1 -- 172.21.15.27:0/1900479215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f53d009b240 con 0x55d354bb9580 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53de059740 10 client.17727 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53de059740 1 -- 172.21.15.27:0/1900479215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d354c3af80 con 0x55d354bb9580 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53bcff9640 1 -- 172.21.15.27:0/1900479215 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53cc036050 con 0x55d354bb9580 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53bcff9640 1 -- 172.21.15.27:0/1900479215 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f53d00cf160 con 0x55d354bb9580 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53de059740 10 client.17727 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53de059740 10 client.17727 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53de059740 20 client.17727 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53de059740 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55d354c3b280 0x55d354c3d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53de059740 4 client.17727 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53de059740 1 -- 172.21.15.27:0/1900479215 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d354aa3820 con 0x55d354c3b280 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.072+0000 7f53d88ea640 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55d354c3b280 0x55d354c3d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53d88ea640 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55d354c3b280 0x55d354c3d670 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d754e353aee11fa6 server_cookie=e3a3363b51f199b0 in_seq=0 out_seq=0 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53bcff9640 10 client.17727 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53bcff9640 1 -- 172.21.15.27:0/1900479215 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d354aa3820 con 0x55d354c3b280 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53bcff9640 10 client.17727 handle_command_reply: tid=0 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 client.17727 shutdown 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 -- 172.21.15.27:0/1900479215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9580 msgr2=0x55d354bc81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9580 0x55d354bc81e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f53d00cce80 tx=0x7f53d000c8c0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 -- 172.21.15.27:0/1900479215 shutdown_connections 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x55d354c3b280 0x55d354c3d670 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d354bc4570 0x55d354bc8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d354bb9ea0 0x55d354bc8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 --2- 172.21.15.27:0/1900479215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d354bb9580 0x55d354bc81e0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53bcff9640 0 client.17727 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 -- 172.21.15.27:0/1900479215 >> 172.21.15.27:0/1900479215 conn(0x55d354bb58a0 msgr2=0x55d354bbacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 -- 172.21.15.27:0/1900479215 shutdown_connections 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 1 -- 172.21.15.27:0/1900479215 wait complete. 2024-04-25T01:35:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.073+0000 7f53de059740 20 client.17727 trim_cache size 0 max 16384 2024-04-25T01:35:13.056 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:35:13.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 -- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53c00a2d00 msgr2=0x7f53c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53c00a2d00 0x7f53c00a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f53cc004ab0 tx=0x7f53cc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 -- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d4101710 msgr2=0x7f53d419a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d4101710 0x7f53d419a200 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f53d000c970 tx=0x7f53d000ce30 comp rx=0 tx=0).stop 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 -- 172.21.15.27:0/14740562 shutdown_connections 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53d4107720 0x7f53d419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53d41020d0 0x7f53d419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f53c00a2d00 0x7f53c00a51c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 --2- 172.21.15.27:0/14740562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53d4101710 0x7f53d419a200 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 -- 172.21.15.27:0/14740562 >> 172.21.15.27:0/14740562 conn(0x7f53d40fd180 msgr2=0x7f53d410cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 -- 172.21.15.27:0/14740562 shutdown_connections 2024-04-25T01:35:13.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.074+0000 7f53db34f640 1 -- 172.21.15.27:0/14740562 wait complete. 2024-04-25T01:35:13.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:35:13.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 -- 172.21.15.27:0/4175673968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a81066b0 msgr2=0x7ff1a8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 --2- 172.21.15.27:0/4175673968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a81066b0 0x7ff1a8106a90 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff198009930 tx=0x7ff19802a420 comp rx=0 tx=0).stop 2024-04-25T01:35:13.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 -- 172.21.15.27:0/4175673968 shutdown_connections 2024-04-25T01:35:13.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 --2- 172.21.15.27:0/4175673968 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a8100fd0 0x7ff1a810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 --2- 172.21.15.27:0/4175673968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a8100610 0x7ff1a8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 --2- 172.21.15.27:0/4175673968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a81066b0 0x7ff1a8106a90 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 -- 172.21.15.27:0/4175673968 >> 172.21.15.27:0/4175673968 conn(0x7ff1a8077740 msgr2=0x7ff1a8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 -- 172.21.15.27:0/4175673968 shutdown_connections 2024-04-25T01:35:13.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.188+0000 7ff1ad542640 1 -- 172.21.15.27:0/4175673968 wait complete. 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1ad542640 1 Processor -- start 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1ad542640 1 -- start start 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1ad542640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a8100610 0x7ff1a810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1ad542640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a8100fd0 0x7ff1a810e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1ad542640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a81066b0 0x7ff1a810e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1ad542640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a8111b30 con 0x7ff1a8100fd0 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1ad542640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a81119b0 con 0x7ff1a81066b0 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1ad542640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a8111cb0 con 0x7ff1a8100610 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a8100fd0 0x7ff1a810e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.189+0000 7ff1a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a8100fd0 0x7ff1a810e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40080/0 (socket says 172.21.15.27:40080) 2024-04-25T01:35:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a67fc640 1 -- 172.21.15.27:0/2039318292 learned_addr learned my addr 172.21.15.27:0/2039318292 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a8100610 0x7ff1a810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a81066b0 0x7ff1a810e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a67fc640 1 -- 172.21.15.27:0/2039318292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a8100610 msgr2=0x7ff1a810df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a8100610 0x7ff1a810df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a67fc640 1 -- 172.21.15.27:0/2039318292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a81066b0 msgr2=0x7ff1a810e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a81066b0 0x7ff1a810e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a67fc640 1 -- 172.21.15.27:0/2039318292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a81a6470 con 0x7ff1a8100fd0 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a8100fd0 0x7ff1a810e4b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00d950 tx=0x7ff19c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff18ffff640 1 -- 172.21.15.27:0/2039318292 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c017070 con 0x7ff1a8100fd0 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff18ffff640 1 -- 172.21.15.27:0/2039318292 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff19c009c30 con 0x7ff1a8100fd0 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1ad542640 1 -- 172.21.15.27:0/2039318292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a81a6760 con 0x7ff1a8100fd0 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff18ffff640 1 -- 172.21.15.27:0/2039318292 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c00f6b0 con 0x7ff1a8100fd0 2024-04-25T01:35:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.190+0000 7ff1ad542640 1 -- 172.21.15.27:0/2039318292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a81a6cb0 con 0x7ff1a8100fd0 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff18ffff640 1 -- 172.21.15.27:0/2039318292 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff19c0109d0 con 0x7ff1a8100fd0 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1b024f740 1 Processor -- start 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1b024f740 1 -- start start 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e94e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e959030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e95b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1b024f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bc9e8b3670 con 0x55bc9e94e580 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1b024f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bc9e85b970 con 0x55bc9e959570 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1b024f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bc9e8af630 con 0x55bc9e94eea0 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e94e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e94e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40096/0 (socket says 172.21.15.27:40096) 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1a77fe640 1 -- 172.21.15.27:0/2196631677 learned_addr learned my addr 172.21.15.27:0/2196631677 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff18ffff640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1900a2d30 0x7ff1900a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e95b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e959030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.193+0000 7ff1a77fe640 1 -- 172.21.15.27:0/2196631677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 msgr2=0x55bc9e959030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e959030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff1a77fe640 1 -- 172.21.15.27:0/2196631677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 msgr2=0x55bc9e95b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e95b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff1a77fe640 1 -- 172.21.15.27:0/2196631677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc9e87f400 con 0x55bc9e94e580 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e959030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1900a2d30 0x7ff1900a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff18ffff640 1 -- 172.21.15.27:0/2039318292 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7ff19c0cf4d0 con 0x7ff1a8100fd0 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e94e960 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009440 tx=0x7ff1a000e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba233b85d5bb103 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff18d7fa640 1 -- 172.21.15.27:0/2196631677 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0004b60 con 0x55bc9e94e580 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1900a2d30 0x7ff1900a51f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff1a001f000 tx=0x7ff1a0020000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff18d7fa640 1 -- 172.21.15.27:0/2196631677 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a001bab0 con 0x55bc9e94e580 2024-04-25T01:35:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.194+0000 7ff18d7fa640 1 -- 172.21.15.27:0/2196631677 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0011a20 con 0x55bc9e94e580 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- 172.21.15.27:0/2196631677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 msgr2=0x55bc9e94e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e94e960 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009440 tx=0x7ff1a000e780 comp rx=0 tx=0).stop 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- 172.21.15.27:0/2196631677 shutdown_connections 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e95b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e959030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 --2- 172.21.15.27:0/2196631677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e94e960 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- 172.21.15.27:0/2196631677 >> 172.21.15.27:0/2196631677 conn(0x55bc9e94a8a0 msgr2=0x55bc9e94ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- 172.21.15.27:0/2196631677 shutdown_connections 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- 172.21.15.27:0/2196631677 wait complete. 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- start start 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 Processor -- start 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e958f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e956230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e956840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bc9e95c360 con 0x55bc9e94e580 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bc9e86f050 con 0x55bc9e959570 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bc9e95c4e0 con 0x55bc9e94eea0 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1b024f740 10 client.0 _ll_register_callbacks cb 0x55bc9e5c6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e958f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.195+0000 7ff1a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e958f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40100/0 (socket says 172.21.15.27:40100) 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a6ffd640 1 -- 172.21.15.27:0/218648074 learned_addr learned my addr 172.21.15.27:0/218648074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e956840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e956230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a6ffd640 1 -- 172.21.15.27:0/218648074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 msgr2=0x55bc9e956230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e956230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a6ffd640 1 -- 172.21.15.27:0/218648074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 msgr2=0x55bc9e956840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e956840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a6ffd640 1 -- 172.21.15.27:0/218648074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc9e840dc0 con 0x55bc9e94e580 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e956840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e958f40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff198009900 tx=0x7ff198031860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454c1231f54e2622 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff18aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff18aff5640 1 -- 172.21.15.27:0/218648074 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198035040 con 0x55bc9e94e580 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff18aff5640 1 -- 172.21.15.27:0/218648074 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7ff198004240 con 0x55bc9e94e580 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1b024f740 1 -- 172.21.15.27:0/218648074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1a001bab0 con 0x55bc9e94e580 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff18aff5640 1 -- 172.21.15.27:0/218648074 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff198004440 con 0x55bc9e94e580 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1b024f740 10 client.17739 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff1b024f740 1 -- 172.21.15.27:0/218648074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bc9e5c73f0 con 0x55bc9e94e580 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.196+0000 7ff18aff5640 1 -- 172.21.15.27:0/218648074 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a001bab0 con 0x55bc9e94e580 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.197+0000 7ff18aff5640 1 -- 172.21.15.27:0/218648074 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff198004c90 con 0x55bc9e94e580 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.197+0000 7ff1b024f740 10 client.17739 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.197+0000 7ff1b024f740 10 client.17739 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.197+0000 7ff1b024f740 20 client.17739 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.197+0000 7ff1b024f740 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc9e969670 0x55bc9e96baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.197+0000 7ff1b024f740 4 client.17739 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.197+0000 7ff1b024f740 1 -- 172.21.15.27:0/218648074 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc9e8c0bb0 con 0x55bc9e969670 2024-04-25T01:35:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.198+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc9e969670 0x55bc9e96baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.198+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc9e969670 0x55bc9e96baa0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eb27aa4e45a0e7e4 server_cookie=22e3d54f2c03bcec in_seq=0 out_seq=0 2024-04-25T01:35:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.198+0000 7ff18aff5640 10 client.17739 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff18aff5640 1 -- 172.21.15.27:0/218648074 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc9e8c0bb0 con 0x55bc9e969670 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff18aff5640 10 client.17739 handle_command_reply: tid=0 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 client.17739 shutdown 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 -- 172.21.15.27:0/218648074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 msgr2=0x55bc9e958f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e958f40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff198009900 tx=0x7ff198031860 comp rx=0 tx=0).stop 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 -- 172.21.15.27:0/218648074 shutdown_connections 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc9e969670 0x55bc9e96baa0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e956840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff18aff5640 0 client.17739 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94eea0 0x55bc9e956230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 --2- 172.21.15.27:0/218648074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94e580 0x55bc9e958f40 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 -- 172.21.15.27:0/218648074 >> 172.21.15.27:0/218648074 conn(0x55bc9e94a8a0 msgr2=0x55bc9e9506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 -- 172.21.15.27:0/218648074 shutdown_connections 2024-04-25T01:35:13.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 1 -- 172.21.15.27:0/218648074 wait complete. 2024-04-25T01:35:13.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.199+0000 7ff1b024f740 20 client.17739 trim_cache size 0 max 16384 2024-04-25T01:35:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.212+0000 7ff1b024f740 1 Processor -- start 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.212+0000 7ff1b024f740 1 -- start start 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94e580 0x55bc9e965220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e9504f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e958a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1b024f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a81b3980 con 0x55bc9e94eea0 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1b024f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a81b1250 con 0x55bc9e959570 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1b024f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a81af0c0 con 0x55bc9e94e580 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e9504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e9504f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40110/0 (socket says 172.21.15.27:40110) 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 -- 172.21.15.27:0/242356880 learned_addr learned my addr 172.21.15.27:0/242356880 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94e580 0x55bc9e965220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e958a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 -- 172.21.15.27:0/242356880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94e580 msgr2=0x55bc9e965220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94e580 0x55bc9e965220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 -- 172.21.15.27:0/242356880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 msgr2=0x55bc9e958a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e958a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 -- 172.21.15.27:0/242356880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc9e6917d0 con 0x55bc9e94eea0 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a77fe640 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e958a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e9504f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0cc890 tx=0x7ff19c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117a7149355d6dae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff18cff9640 1 -- 172.21.15.27:0/242356880 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c020040 con 0x55bc9e94eea0 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff18cff9640 1 -- 172.21.15.27:0/242356880 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff19c09b5b0 con 0x55bc9e94eea0 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.213+0000 7ff18cff9640 1 -- 172.21.15.27:0/242356880 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0cca60 con 0x55bc9e94eea0 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 -- 172.21.15.27:0/242356880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 msgr2=0x55bc9e9504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e9504f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0cc890 tx=0x7ff19c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 -- 172.21.15.27:0/242356880 shutdown_connections 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e959570 0x55bc9e958a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e9504f0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 --2- 172.21.15.27:0/242356880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e94e580 0x55bc9e965220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 -- 172.21.15.27:0/242356880 >> 172.21.15.27:0/242356880 conn(0x55bc9e94a8a0 msgr2=0x55bc9e94fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 -- 172.21.15.27:0/242356880 shutdown_connections 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 -- 172.21.15.27:0/242356880 wait complete. 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 -- start start 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.214+0000 7ff1b024f740 1 Processor -- start 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e94e580 0x55bc9e961940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e95ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e959570 0x55bc9e95d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a81af710 con 0x55bc9e94eea0 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a81b0b80 con 0x55bc9e94e580 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a81afde0 con 0x55bc9e959570 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 10 client.0 _ll_register_callbacks cb 0x7ff198005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e95ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e95ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40122/0 (socket says 172.21.15.27:40122) 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 -- 172.21.15.27:0/1809318944 learned_addr learned my addr 172.21.15.27:0/1809318944 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e94e580 0x55bc9e961940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 -- 172.21.15.27:0/1809318944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e959570 msgr2=0x55bc9e95d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e959570 0x55bc9e95d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 -- 172.21.15.27:0/1809318944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e94e580 msgr2=0x55bc9e961940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e94e580 0x55bc9e961940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 -- 172.21.15.27:0/1809318944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc9e95d830 con 0x55bc9e94eea0 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e94e580 0x55bc9e961940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1a67fc640 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e95ccb0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7ff19c097f80 tx=0x7ff19c00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91506f0c2a41fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff18aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff18aff5640 1 -- 172.21.15.27:0/1809318944 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0b4060 con 0x55bc9e94eea0 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff18aff5640 1 -- 172.21.15.27:0/1809318944 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7ff19c097c20 con 0x55bc9e94eea0 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 1 -- 172.21.15.27:0/1809318944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff19c09b5b0 con 0x55bc9e94eea0 2024-04-25T01:35:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff18aff5640 1 -- 172.21.15.27:0/1809318944 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff19c020030 con 0x55bc9e94eea0 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 10 client.17745 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.215+0000 7ff1b024f740 1 -- 172.21.15.27:0/1809318944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bc9e9cf9a0 con 0x55bc9e94eea0 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff18aff5640 1 -- 172.21.15.27:0/1809318944 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff19c09b5b0 con 0x55bc9e94eea0 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff18aff5640 1 -- 172.21.15.27:0/1809318944 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff19c097300 con 0x55bc9e94eea0 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff1b024f740 10 client.17745 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff1b024f740 10 client.17745 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff1b024f740 20 client.17745 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff1b024f740 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc9e9d0070 0x55bc9e9d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff1b024f740 4 client.17745 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff1b024f740 1 -- 172.21.15.27:0/1809318944 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc9e8baf00 con 0x55bc9e9d0070 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.216+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc9e9d0070 0x55bc9e9d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1a6ffd640 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc9e9d0070 0x55bc9e9d2490 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=408ec706391ee28d server_cookie=3cfe81a2dde8a3fb in_seq=0 out_seq=0 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff18aff5640 10 client.17745 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff18aff5640 1 -- 172.21.15.27:0/1809318944 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc9e8baf00 con 0x55bc9e9d0070 2024-04-25T01:35:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff18aff5640 10 client.17745 handle_command_reply: tid=0 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 client.17745 shutdown 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 -- 172.21.15.27:0/1809318944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 msgr2=0x55bc9e95ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e95ccb0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7ff19c097f80 tx=0x7ff19c00bf60 comp rx=0 tx=0).stop 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 -- 172.21.15.27:0/1809318944 shutdown_connections 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bc9e9d0070 0x55bc9e9d2490 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bc9e959570 0x55bc9e95d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff18aff5640 0 client.17745 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bc9e94eea0 0x55bc9e95ccb0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 --2- 172.21.15.27:0/1809318944 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bc9e94e580 0x55bc9e961940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 -- 172.21.15.27:0/1809318944 >> 172.21.15.27:0/1809318944 conn(0x55bc9e94a8a0 msgr2=0x55bc9e950600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 -- 172.21.15.27:0/1809318944 shutdown_connections 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 1 -- 172.21.15.27:0/1809318944 wait complete. 2024-04-25T01:35:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.217+0000 7ff1b024f740 20 client.17745 trim_cache size 0 max 16384 2024-04-25T01:35:13.202 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:35:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 -- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1900a2d30 msgr2=0x7ff1900a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1900a2d30 0x7ff1900a51f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff1a001f000 tx=0x7ff1a0020000 comp rx=0 tx=0).stop 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 -- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a8100fd0 msgr2=0x7ff1a810e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a8100fd0 0x7ff1a810e4b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00d950 tx=0x7ff19c00de10 comp rx=0 tx=0).stop 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 -- 172.21.15.27:0/2039318292 shutdown_connections 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a81066b0 0x7ff1a810e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1900a2d30 0x7ff1900a51f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a8100fd0 0x7ff1a810e4b0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 --2- 172.21.15.27:0/2039318292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a8100610 0x7ff1a810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 -- 172.21.15.27:0/2039318292 >> 172.21.15.27:0/2039318292 conn(0x7ff1a8077740 msgr2=0x7ff1a80ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 -- 172.21.15.27:0/2039318292 shutdown_connections 2024-04-25T01:35:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.218+0000 7ff1ad542640 1 -- 172.21.15.27:0/2039318292 wait complete. 2024-04-25T01:35:13.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:35:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 -- 172.21.15.27:0/3490270965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0101710 msgr2=0x7facf0101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 --2- 172.21.15.27:0/3490270965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0101710 0x7facf0101b90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7facdc009930 tx=0x7facdc02a420 comp rx=0 tx=0).stop 2024-04-25T01:35:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 -- 172.21.15.27:0/3490270965 shutdown_connections 2024-04-25T01:35:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 --2- 172.21.15.27:0/3490270965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf01020d0 0x7facf010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 --2- 172.21.15.27:0/3490270965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0101710 0x7facf0101b90 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 --2- 172.21.15.27:0/3490270965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facf0107720 0x7facf0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 -- 172.21.15.27:0/3490270965 >> 172.21.15.27:0/3490270965 conn(0x7facf0078b90 msgr2=0x7facf0078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 -- 172.21.15.27:0/3490270965 shutdown_connections 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.332+0000 7facf6dd5640 1 -- 172.21.15.27:0/3490270965 wait complete. 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf6dd5640 1 Processor -- start 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf6dd5640 1 -- start start 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf6dd5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf0101710 0x7facf0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf6dd5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facf01020d0 0x7facf0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf6dd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0107720 0x7facf019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf6dd5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7facf0111b30 con 0x7facf0107720 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf6dd5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7facf01119b0 con 0x7facf01020d0 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf6dd5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7facf0111cb0 con 0x7facf0101710 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf5372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0107720 0x7facf019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf5372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0107720 0x7facf019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40138/0 (socket says 172.21.15.27:40138) 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf5372640 1 -- 172.21.15.27:0/4272484595 learned_addr learned my addr 172.21.15.27:0/4272484595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf4b71640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf0101710 0x7facf0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf5372640 1 -- 172.21.15.27:0/4272484595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf0101710 msgr2=0x7facf0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf5372640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf0101710 0x7facf0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf5372640 1 -- 172.21.15.27:0/4272484595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facf01020d0 msgr2=0x7facf0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7faceffff640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facf01020d0 0x7facf0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf5372640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facf01020d0 0x7facf0198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.333+0000 7facf5372640 1 -- 172.21.15.27:0/4272484595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf019d000 con 0x7facf0107720 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.334+0000 7facf5372640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0107720 0x7facf019c8d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7face800cad0 tx=0x7face800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.334+0000 7facedffb640 1 -- 172.21.15.27:0/4272484595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face800ae20 con 0x7facf0107720 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.334+0000 7facedffb640 1 -- 172.21.15.27:0/4272484595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face8005020 con 0x7facf0107720 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.334+0000 7facedffb640 1 -- 172.21.15.27:0/4272484595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face800f6f0 con 0x7facf0107720 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.334+0000 7facf6dd5640 1 -- 172.21.15.27:0/4272484595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf019d2f0 con 0x7facf0107720 2024-04-25T01:35:13.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.334+0000 7facf6dd5640 1 -- 172.21.15.27:0/4272484595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf01a4b80 con 0x7facf0107720 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.336+0000 7facedffb640 1 -- 172.21.15.27:0/4272484595 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7face80051d0 con 0x7facf0107720 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.336+0000 7facf9bc3740 1 Processor -- start 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf9bc3740 1 -- start start 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facedffb640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7face00a7190 0x7face00a9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97580 0x555f9ab97960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97ea0 0x555f9aba2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7faceffff640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7face00a7190 0x7face00a9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facedffb640 1 -- 172.21.15.27:0/4272484595 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7face80cfc70 con 0x7facf0107720 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 0x555f9aba4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf9bc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555f9aafc670 con 0x555f9ab97580 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf9bc3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555f9aaa4970 con 0x555f9ab97ea0 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf9bc3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555f9aaf8630 con 0x555f9aba2570 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97580 0x555f9ab97960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97580 0x555f9ab97960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40150/0 (socket says 172.21.15.27:40150) 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 -- 172.21.15.27:0/1728842717 learned_addr learned my addr 172.21.15.27:0/1728842717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7faceffff640 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97ea0 0x555f9aba2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf5372640 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 0x555f9aba4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 -- 172.21.15.27:0/1728842717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 msgr2=0x555f9aba4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 0x555f9aba4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 -- 172.21.15.27:0/1728842717 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97ea0 msgr2=0x555f9aba2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97ea0 0x555f9aba2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 -- 172.21.15.27:0/1728842717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f9aac8400 con 0x555f9ab97580 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7faceffff640 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97ea0 0x555f9aba2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7faceffff640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7face00a7190 0x7face00a9650 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7facdc0096c0 tx=0x7facdc005e80 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.337+0000 7facf4b71640 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97580 0x555f9ab97960 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7face400cbe0 tx=0x7face400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeecfbf4d572fb09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facdaffd640 1 -- 172.21.15.27:0/1728842717 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4005220 con 0x555f9ab97580 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facdaffd640 1 -- 172.21.15.27:0/1728842717 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face4004c80 con 0x555f9ab97580 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facdaffd640 1 -- 172.21.15.27:0/1728842717 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4010770 con 0x555f9ab97580 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 -- 172.21.15.27:0/1728842717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97580 msgr2=0x555f9ab97960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97580 0x555f9ab97960 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7face400cbe0 tx=0x7face400e720 comp rx=0 tx=0).stop 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 -- 172.21.15.27:0/1728842717 shutdown_connections 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 0x555f9aba4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97ea0 0x555f9aba2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 --2- 172.21.15.27:0/1728842717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97580 0x555f9ab97960 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 -- 172.21.15.27:0/1728842717 >> 172.21.15.27:0/1728842717 conn(0x555f9ab938a0 msgr2=0x555f9ab93c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 -- 172.21.15.27:0/1728842717 shutdown_connections 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.338+0000 7facf9bc3740 1 -- 172.21.15.27:0/1728842717 wait complete. 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 1 -- start start 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 1 Processor -- start 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 0x555f9aba1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 0x555f9ab9f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555f9aba5360 con 0x555f9ab97ea0 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x555f9aab8050 con 0x555f9ab97580 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555f9aba54e0 con 0x555f9aba2570 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf9bc3740 10 client.0 _ll_register_callbacks cb 0x555f9a80fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7faceffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7faceffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40162/0 (socket says 172.21.15.27:40162) 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7faceffff640 1 -- 172.21.15.27:0/2997455693 learned_addr learned my addr 172.21.15.27:0/2997455693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.339+0000 7facf5372640 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 0x555f9ab9f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7facf4b71640 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 0x555f9aba1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faceffff640 1 -- 172.21.15.27:0/2997455693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 msgr2=0x555f9ab9f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faceffff640 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 0x555f9ab9f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faceffff640 1 -- 172.21.15.27:0/2997455693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 msgr2=0x555f9aba1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faceffff640 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 0x555f9aba1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faceffff640 1 -- 172.21.15.27:0/2997455693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f9aa89dc0 con 0x555f9ab97ea0 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faceffff640 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9f170 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7facdc033000 tx=0x7facdc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa49f0f90fda535 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faccffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faccffff640 1 -- 172.21.15.27:0/2997455693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc035050 con 0x555f9ab97ea0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7facf9bc3740 1 -- 172.21.15.27:0/2997455693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face4004c80 con 0x555f9ab97ea0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faccffff640 1 -- 172.21.15.27:0/2997455693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7facdc005450 con 0x555f9ab97ea0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faccffff640 1 -- 172.21.15.27:0/2997455693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7facdc056050 con 0x555f9ab97ea0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7facf9bc3740 10 client.17757 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7facf9bc3740 1 -- 172.21.15.27:0/2997455693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f9a8103f0 con 0x555f9ab97ea0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faccffff640 1 -- 172.21.15.27:0/2997455693 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face4004c80 con 0x555f9ab97ea0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.340+0000 7faccffff640 1 -- 172.21.15.27:0/2997455693 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7facdc047070 con 0x555f9ab97ea0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7facf9bc3740 10 client.17757 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7facf9bc3740 10 client.17757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7facf9bc3740 20 client.17757 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7facf9bc3740 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f9abb2530 0x555f9abb4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7facf9bc3740 4 client.17757 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7facf9bc3740 1 -- 172.21.15.27:0/2997455693 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f9ab09bb0 con 0x555f9abb2530 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7facf4b71640 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f9abb2530 0x555f9abb4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7faccffff640 10 client.17757 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:13.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.341+0000 7facf4b71640 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f9abb2530 0x555f9abb4960 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c7fbc737a5e314 server_cookie=832de18b65afa547 in_seq=0 out_seq=0 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7faccffff640 1 -- 172.21.15.27:0/2997455693 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f9ab09bb0 con 0x555f9abb2530 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7faccffff640 10 client.17757 handle_command_reply: tid=0 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 client.17757 shutdown 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 -- 172.21.15.27:0/2997455693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 msgr2=0x555f9ab9f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9f170 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7facdc033000 tx=0x7facdc02a420 comp rx=0 tx=0).stop 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 -- 172.21.15.27:0/2997455693 shutdown_connections 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f9abb2530 0x555f9abb4960 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9aba2570 0x555f9ab9f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9f170 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7faccffff640 0 client.17757 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 --2- 172.21.15.27:0/2997455693 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 0x555f9aba1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.342+0000 7facf9bc3740 1 -- 172.21.15.27:0/2997455693 >> 172.21.15.27:0/2997455693 conn(0x555f9ab938a0 msgr2=0x555f9ab92b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.343+0000 7facf9bc3740 1 -- 172.21.15.27:0/2997455693 shutdown_connections 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.343+0000 7facf9bc3740 1 -- 172.21.15.27:0/2997455693 wait complete. 2024-04-25T01:35:13.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.343+0000 7facf9bc3740 20 client.17757 trim_cache size 0 max 16384 2024-04-25T01:35:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf9bc3740 1 Processor -- start 2024-04-25T01:35:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf9bc3740 1 -- start start 2024-04-25T01:35:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 0x555f9abae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97ea0 0x555f9ab99420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9aba2570 0x555f9aba19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf9bc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7facf01af590 con 0x555f9aba2570 2024-04-25T01:35:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf9bc3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ace60 con 0x555f9ab97580 2024-04-25T01:35:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf9bc3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7facf01aacd0 con 0x555f9ab97ea0 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf5372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9aba2570 0x555f9aba19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf5372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9aba2570 0x555f9aba19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40166/0 (socket says 172.21.15.27:40166) 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf5372640 1 -- 172.21.15.27:0/254580723 learned_addr learned my addr 172.21.15.27:0/254580723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf4b71640 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 0x555f9abae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7faceffff640 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97ea0 0x555f9ab99420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf5372640 1 -- 172.21.15.27:0/254580723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97ea0 msgr2=0x555f9ab99420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf5372640 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97ea0 0x555f9ab99420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf5372640 1 -- 172.21.15.27:0/254580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 msgr2=0x555f9abae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf5372640 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 0x555f9abae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.356+0000 7facf5372640 1 -- 172.21.15.27:0/254580723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f9a8da7d0 con 0x555f9aba2570 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf5372640 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9aba2570 0x555f9aba19b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7face800a6e0 tx=0x7face809bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42d836d3f6b7a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facda7fc640 1 -- 172.21.15.27:0/254580723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face801d040 con 0x555f9aba2570 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facda7fc640 1 -- 172.21.15.27:0/254580723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face8005560 con 0x555f9aba2570 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facda7fc640 1 -- 172.21.15.27:0/254580723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face80a4730 con 0x555f9aba2570 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 -- 172.21.15.27:0/254580723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9aba2570 msgr2=0x555f9aba19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9aba2570 0x555f9aba19b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7face800a6e0 tx=0x7face809bdb0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 -- 172.21.15.27:0/254580723 shutdown_connections 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9aba2570 0x555f9aba19b0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97ea0 0x555f9ab99420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 --2- 172.21.15.27:0/254580723 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9ab97580 0x555f9abae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 -- 172.21.15.27:0/254580723 >> 172.21.15.27:0/254580723 conn(0x555f9ab938a0 msgr2=0x555f9ab98d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 -- 172.21.15.27:0/254580723 shutdown_connections 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.357+0000 7facf9bc3740 1 -- 172.21.15.27:0/254580723 wait complete. 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 1 -- start start 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 1 Processor -- start 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97580 0x555f9abaa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9aba2570 0x555f9ab9bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ab320 con 0x555f9ab97ea0 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ac790 con 0x555f9aba2570 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ab9f0 con 0x555f9ab97580 2024-04-25T01:35:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf9bc3740 10 client.0 _ll_register_callbacks cb 0x7facdc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7faceffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7faceffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40178/0 (socket says 172.21.15.27:40178) 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7facf4b71640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97580 0x555f9abaa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.358+0000 7faceffff640 1 -- 172.21.15.27:0/3015475747 learned_addr learned my addr 172.21.15.27:0/3015475747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7facf5372640 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9aba2570 0x555f9ab9bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faceffff640 1 -- 172.21.15.27:0/3015475747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97580 msgr2=0x555f9abaa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faceffff640 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97580 0x555f9abaa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faceffff640 1 -- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9aba2570 msgr2=0x555f9ab9bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faceffff640 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9aba2570 0x555f9ab9bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faceffff640 1 -- 172.21.15.27:0/3015475747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f9ab9c490 con 0x555f9ab97ea0 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7facf5372640 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9aba2570 0x555f9ab9bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faceffff640 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9b9e0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x555f9aba0740 tx=0x7facdc004490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a636f78cac932bcc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faccffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faccffff640 1 -- 172.21.15.27:0/3015475747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc035040 con 0x555f9ab97ea0 2024-04-25T01:35:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faccffff640 1 -- 172.21.15.27:0/3015475747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7facdc005250 con 0x555f9ab97ea0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7facf9bc3740 1 -- 172.21.15.27:0/3015475747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face8005560 con 0x555f9ab97ea0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faccffff640 1 -- 172.21.15.27:0/3015475747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7facdc068050 con 0x555f9ab97ea0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7facf9bc3740 10 client.17763 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7facf9bc3740 1 -- 172.21.15.27:0/3015475747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f9ac18830 con 0x555f9ab97ea0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.359+0000 7faccffff640 1 -- 172.21.15.27:0/3015475747 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face8005560 con 0x555f9ab97ea0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7faccffff640 1 -- 172.21.15.27:0/3015475747 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7facdc047030 con 0x555f9ab97ea0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7facf9bc3740 10 client.17763 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7facf9bc3740 10 client.17763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7facf9bc3740 20 client.17763 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7facf9bc3740 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f9ac18fd0 0x555f9ac1b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7facf9bc3740 4 client.17763 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7facf9bc3740 1 -- 172.21.15.27:0/3015475747 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f9ab9fa40 con 0x555f9ac18fd0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7facf4b71640 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f9ac18fd0 0x555f9ac1b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7facf4b71640 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f9ac18fd0 0x555f9ac1b3f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=425d97791391fea6 server_cookie=d56c15ac0058b320 in_seq=0 out_seq=0 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.360+0000 7faccffff640 10 client.17763 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7faccffff640 1 -- 172.21.15.27:0/3015475747 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f9ab9fa40 con 0x555f9ac18fd0 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7faccffff640 10 client.17763 handle_command_reply: tid=0 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 client.17763 shutdown 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 -- 172.21.15.27:0/3015475747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 msgr2=0x555f9ab9b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9b9e0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x555f9aba0740 tx=0x7facdc004490 comp rx=0 tx=0).stop 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 -- 172.21.15.27:0/3015475747 shutdown_connections 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x555f9ac18fd0 0x555f9ac1b3f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x555f9aba2570 0x555f9ab9bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555f9ab97ea0 0x555f9ab9b9e0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7faccffff640 0 client.17763 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 --2- 172.21.15.27:0/3015475747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f9ab97580 0x555f9abaa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 -- 172.21.15.27:0/3015475747 >> 172.21.15.27:0/3015475747 conn(0x555f9ab938a0 msgr2=0x555f9ab9a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 -- 172.21.15.27:0/3015475747 shutdown_connections 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 1 -- 172.21.15.27:0/3015475747 wait complete. 2024-04-25T01:35:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf9bc3740 20 client.17763 trim_cache size 0 max 16384 2024-04-25T01:35:13.344 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:35:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf6dd5640 1 -- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7face00a7190 msgr2=0x7face00a9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf6dd5640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7face00a7190 0x7face00a9650 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7facdc0096c0 tx=0x7facdc005e80 comp rx=0 tx=0).stop 2024-04-25T01:35:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.361+0000 7facf6dd5640 1 -- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0107720 msgr2=0x7facf019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0107720 0x7facf019c8d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7face800cad0 tx=0x7face800cf90 comp rx=0 tx=0).stop 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 -- 172.21.15.27:0/4272484595 shutdown_connections 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7face00a7190 0x7face00a9650 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facf0107720 0x7facf019c8d0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7facf01020d0 0x7facf0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 --2- 172.21.15.27:0/4272484595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf0101710 0x7facf0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 -- 172.21.15.27:0/4272484595 >> 172.21.15.27:0/4272484595 conn(0x7facf0078b90 msgr2=0x7facf00ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 -- 172.21.15.27:0/4272484595 shutdown_connections 2024-04-25T01:35:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:13.362+0000 7facf6dd5640 1 -- 172.21.15.27:0/4272484595 wait complete. 2024-04-25T01:35:13.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:13 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3391482055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:14.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:13 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3391482055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:14.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:13 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3391482055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:14.350 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:35:14.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:35:14.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:35:14.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.496+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/2750628234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 msgr2=0x7ff6c406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.496+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/2750628234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 0x7ff6c406a240 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac009930 tx=0x7ff6ac02a420 comp rx=0 tx=0).stop 2024-04-25T01:35:14.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/2750628234 shutdown_connections 2024-04-25T01:35:14.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/2750628234 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6c406a780 0x7ff6c410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/2750628234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 0x7ff6c406a240 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/2750628234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6c4107730 0x7ff6c4107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/2750628234 >> 172.21.15.27:0/2750628234 conn(0x7ff6c40776a0 msgr2=0x7ff6c4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/2750628234 shutdown_connections 2024-04-25T01:35:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/2750628234 wait complete. 2024-04-25T01:35:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 Processor -- start 2024-04-25T01:35:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.497+0000 7ff6ca8a3640 1 -- start start 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6ca8a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 0x7ff6c4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6ca8a3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6c406a780 0x7ff6c4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6ca8a3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6c4107730 0x7ff6c419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6ca8a3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c4111b40 con 0x7ff6c4069dc0 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6ca8a3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c41119c0 con 0x7ff6c406a780 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6ca8a3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c4111cc0 con 0x7ff6c4107730 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 0x7ff6c4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 0x7ff6c4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40196/0 (socket says 172.21.15.27:40196) 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 -- 172.21.15.27:0/980741429 learned_addr learned my addr 172.21.15.27:0/980741429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c37fe640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6c406a780 0x7ff6c4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6c4107730 0x7ff6c419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 -- 172.21.15.27:0/980741429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6c4107730 msgr2=0x7ff6c419c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6c4107730 0x7ff6c419c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 -- 172.21.15.27:0/980741429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6c406a780 msgr2=0x7ff6c4198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6c406a780 0x7ff6c4198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 -- 172.21.15.27:0/980741429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c419cff0 con 0x7ff6c4069dc0 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.498+0000 7ff6c3fff640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 0x7ff6c4198000 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400c970 tx=0x7ff6b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.499+0000 7ff6c17fa640 1 -- 172.21.15.27:0/980741429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4007bb0 con 0x7ff6c4069dc0 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.499+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/980741429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c419d2e0 con 0x7ff6c4069dc0 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.499+0000 7ff6c17fa640 1 -- 172.21.15.27:0/980741429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b4007d50 con 0x7ff6c4069dc0 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.499+0000 7ff6c17fa640 1 -- 172.21.15.27:0/980741429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b400f450 con 0x7ff6c4069dc0 2024-04-25T01:35:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.499+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/980741429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c41a4b70 con 0x7ff6c4069dc0 2024-04-25T01:35:14.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.501+0000 7ff6c17fa640 1 -- 172.21.15.27:0/980741429 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff6b4005020 con 0x7ff6c4069dc0 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.501+0000 7ff6cd5a8740 1 Processor -- start 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.501+0000 7ff6cd5a8740 1 -- start start 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.501+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15ce9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.501+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cea290 0x55ae15cea710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.501+0000 7ff6c17fa640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6b00a2d00 0x7ff6b00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cf5010 0x55ae15cf7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ae15c4e2c0 con 0x55ae15ceb980 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ae15bf6910 con 0x55ae15cf5010 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ae15c4a280 con 0x55ae15cea290 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15ce9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c3fff640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6b00a2d00 0x7ff6b00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15ce9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40208/0 (socket says 172.21.15.27:40208) 2024-04-25T01:35:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 -- 172.21.15.27:0/1213737646 learned_addr learned my addr 172.21.15.27:0/1213737646 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c37fe640 1 --2- 172.21.15.27:0/1213737646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cea290 0x55ae15cea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c17fa640 1 -- 172.21.15.27:0/980741429 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7ff6b40cf620 con 0x7ff6c4069dc0 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 -- 172.21.15.27:0/1213737646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cea290 msgr2=0x55ae15cea710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/1213737646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cea290 0x55ae15cea710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 -- 172.21.15.27:0/1213737646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cf5010 msgr2=0x55ae15cf7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/1213737646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cf5010 0x55ae15cf7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 -- 172.21.15.27:0/1213737646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae15c07fb0 con 0x55ae15ceb980 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c3fff640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6b00a2d00 0x7ff6b00a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40cc9e0 tx=0x7ff6b4097360 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/1213737646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15ce9cc0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc011400 tx=0x7ff6bc011800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9486ab55d8d2855 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6ba7fc640 1 -- 172.21.15.27:0/1213737646 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0126b0 con 0x55ae15ceb980 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6ba7fc640 1 -- 172.21.15.27:0/1213737646 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc012d30 con 0x55ae15ceb980 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.502+0000 7ff6ba7fc640 1 -- 172.21.15.27:0/1213737646 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc01ad00 con 0x55ae15ceb980 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1213737646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 msgr2=0x55ae15ce9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1213737646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15ce9cc0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc011400 tx=0x7ff6bc011800 comp rx=0 tx=0).stop 2024-04-25T01:35:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1213737646 shutdown_connections 2024-04-25T01:35:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1213737646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cf5010 0x55ae15cf7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1213737646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cea290 0x55ae15cea710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1213737646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15ce9cc0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1213737646 >> 172.21.15.27:0/1213737646 conn(0x55ae15cb2950 msgr2=0x55ae15ce67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1213737646 shutdown_connections 2024-04-25T01:35:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1213737646 wait complete. 2024-04-25T01:35:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.503+0000 7ff6cd5a8740 1 -- start start 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6cd5a8740 1 Processor -- start 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15cf1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15ceb980 0x55ae15cf2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cf5010 0x55ae15cf2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ae15cf7ea0 con 0x55ae15cf5010 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55ae15c09e90 con 0x55ae15cea290 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ae15cf8020 con 0x55ae15ceb980 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6cd5a8740 10 client.0 _ll_register_callbacks cb 0x55ae15c45970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6c8e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cf5010 0x55ae15cf2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6c8e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cf5010 0x55ae15cf2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40224/0 (socket says 172.21.15.27:40224) 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6c8e40640 1 -- 172.21.15.27:0/1716387507 learned_addr learned my addr 172.21.15.27:0/1716387507 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6c8e40640 1 -- 172.21.15.27:0/1716387507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15ceb980 msgr2=0x55ae15cf2530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15ceb980 0x55ae15cf2530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6c8e40640 1 -- 172.21.15.27:0/1716387507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 msgr2=0x55ae15cf1ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15cf1ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.504+0000 7ff6c8e40640 1 -- 172.21.15.27:0/1716387507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae15a2a7e0 con 0x55ae15cf5010 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cf5010 0x55ae15cf2a70 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0113d0 tx=0x7ff6bc01a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168acd4f8b2f605 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6a37fe640 1 -- 172.21.15.27:0/1716387507 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc004c50 con 0x55ae15cf5010 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6a37fe640 1 -- 172.21.15.27:0/1716387507 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7ff6bc004df0 con 0x55ae15cf5010 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1716387507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6bc012d30 con 0x55ae15cf5010 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6a37fe640 1 -- 172.21.15.27:0/1716387507 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff6bc02e050 con 0x55ae15cf5010 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 10 client.17775 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1716387507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ae15a8af70 con 0x55ae15cf5010 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6a37fe640 1 -- 172.21.15.27:0/1716387507 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc012d30 con 0x55ae15cf5010 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6a37fe640 1 -- 172.21.15.27:0/1716387507 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff6bc005000 con 0x55ae15cf5010 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 10 client.17775 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 20 client.17775 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ae15d050b0 0x55ae15d074a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 4 client.17775 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.505+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1716387507 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae15c225a0 con 0x55ae15d050b0 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.506+0000 7ff6c37fe640 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ae15d050b0 0x55ae15d074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.506+0000 7ff6a37fe640 10 client.17775 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.506+0000 7ff6c37fe640 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ae15d050b0 0x55ae15d074a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e14cc7bd6a7f6f2 server_cookie=c0f9dbbc23ae0ec7 in_seq=0 out_seq=0 2024-04-25T01:35:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.507+0000 7ff6a37fe640 1 -- 172.21.15.27:0/1716387507 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae15c225a0 con 0x55ae15d050b0 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.507+0000 7ff6a37fe640 10 client.17775 handle_command_reply: tid=0 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.507+0000 7ff6cd5a8740 1 client.17775 shutdown 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.507+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1716387507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cf5010 msgr2=0x55ae15cf2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.507+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cf5010 0x55ae15cf2a70 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0113d0 tx=0x7ff6bc01a630 comp rx=0 tx=0).stop 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1716387507 shutdown_connections 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ae15d050b0 0x55ae15d074a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cf5010 0x55ae15cf2a70 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15ceb980 0x55ae15cf2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/1716387507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15cf1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6a37fe640 0 client.17775 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1716387507 >> 172.21.15.27:0/1716387507 conn(0x55ae15cb2950 msgr2=0x55ae15cec660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1716387507 shutdown_connections 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/1716387507 wait complete. 2024-04-25T01:35:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.508+0000 7ff6cd5a8740 20 client.17775 trim_cache size 0 max 16384 2024-04-25T01:35:14.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6cd5a8740 1 Processor -- start 2024-04-25T01:35:14.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6cd5a8740 1 -- start start 2024-04-25T01:35:14.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cb2950 0x55ae15d00da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15d0caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15d0cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c41b1760 con 0x55ae15ceb980 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c41af030 con 0x55ae15cea290 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c41acea0 con 0x55ae15cb2950 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15d0cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15d0cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40226/0 (socket says 172.21.15.27:40226) 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 -- 172.21.15.27:0/789448897 learned_addr learned my addr 172.21.15.27:0/789448897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 -- 172.21.15.27:0/789448897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cb2950 msgr2=0x55ae15d00da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/789448897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cb2950 0x55ae15d00da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 -- 172.21.15.27:0/789448897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 msgr2=0x55ae15d0caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/789448897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15d0caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 -- 172.21.15.27:0/789448897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae15922170 con 0x55ae15ceb980 2024-04-25T01:35:14.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.526+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/789448897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15d0cfe0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0096c0 tx=0x7ff6bc005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6cd1d0f0d568e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6b9ffb640 1 -- 172.21.15.27:0/789448897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc011d00 con 0x55ae15ceb980 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6b9ffb640 1 -- 172.21.15.27:0/789448897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc02b020 con 0x55ae15ceb980 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6b9ffb640 1 -- 172.21.15.27:0/789448897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0050e0 con 0x55ae15ceb980 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/789448897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 msgr2=0x55ae15d0cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/789448897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15d0cfe0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0096c0 tx=0x7ff6bc005900 comp rx=0 tx=0).stop 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/789448897 shutdown_connections 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/789448897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15ceb980 0x55ae15d0cfe0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/789448897 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15d0caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/789448897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15cb2950 0x55ae15d00da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/789448897 >> 172.21.15.27:0/789448897 conn(0x55ae15ce5cf0 msgr2=0x55ae15cf1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/789448897 shutdown_connections 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/789448897 wait complete. 2024-04-25T01:35:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.527+0000 7ff6cd5a8740 1 -- start start 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6cd5a8740 1 Processor -- start 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cb2950 0x55ae15d64620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15d64b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6cd5a8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15ceb980 0x55ae15cf94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c41ad4f0 con 0x55ae15cb2950 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c41ae960 con 0x55ae15cea290 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6cd5a8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c41adbc0 con 0x55ae15ceb980 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6cd5a8740 10 client.0 _ll_register_callbacks cb 0x7ff6bc009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c8e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cb2950 0x55ae15d64620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c8e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cb2950 0x55ae15d64620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40236/0 (socket says 172.21.15.27:40236) 2024-04-25T01:35:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c8e40640 1 -- 172.21.15.27:0/3474940744 learned_addr learned my addr 172.21.15.27:0/3474940744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c8e40640 1 -- 172.21.15.27:0/3474940744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15ceb980 msgr2=0x55ae15cf94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15ceb980 0x55ae15cf94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c37fe640 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15d64b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c8e40640 1 -- 172.21.15.27:0/3474940744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 msgr2=0x55ae15d64b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15d64b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.528+0000 7ff6c8e40640 1 -- 172.21.15.27:0/3474940744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae15cf9b90 con 0x55ae15cb2950 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6c8e40640 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cb2950 0x55ae15d64620 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc000f80 tx=0x7ff6bc005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882b43297fe1fdb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6a37fe640 1 -- 172.21.15.27:0/3474940744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc003970 con 0x55ae15cb2950 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6a37fe640 1 -- 172.21.15.27:0/3474940744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7ff6bc003b10 con 0x55ae15cb2950 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/3474940744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6bc02b020 con 0x55ae15cb2950 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6a37fe640 1 -- 172.21.15.27:0/3474940744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff6bc003d10 con 0x55ae15cb2950 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 10 client.17781 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/3474940744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ae15d6be00 con 0x55ae15cb2950 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6a37fe640 1 -- 172.21.15.27:0/3474940744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc02b020 con 0x55ae15cb2950 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6a37fe640 1 -- 172.21.15.27:0/3474940744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff6bc0046c0 con 0x55ae15cb2950 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 10 client.17781 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 10 client.17781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 20 client.17781 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ae15cf5010 0x55ae15d6e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 4 client.17781 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.529+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/3474940744 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ae15c5b800 con 0x55ae15cf5010 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.530+0000 7ff6c37fe640 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ae15cf5010 0x55ae15d6e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.530+0000 7ff6a37fe640 10 client.17781 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.530+0000 7ff6c37fe640 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ae15cf5010 0x55ae15d6e3a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae2e98efb1bd8396 server_cookie=bc38413431a384ef in_seq=0 out_seq=0 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6a37fe640 1 -- 172.21.15.27:0/3474940744 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ae15c5b800 con 0x55ae15cf5010 2024-04-25T01:35:14.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6a37fe640 10 client.17781 handle_command_reply: tid=0 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 client.17781 shutdown 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/3474940744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cb2950 msgr2=0x55ae15d64620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cb2950 0x55ae15d64620 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc000f80 tx=0x7ff6bc005c60 comp rx=0 tx=0).stop 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/3474940744 shutdown_connections 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55ae15cf5010 0x55ae15d6e3a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ae15ceb980 0x55ae15cf94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55ae15cea290 0x55ae15d64b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6a37fe640 0 client.17781 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 --2- 172.21.15.27:0/3474940744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae15cb2950 0x55ae15d64620 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/3474940744 >> 172.21.15.27:0/3474940744 conn(0x55ae15ce5cf0 msgr2=0x55ae15cf0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/3474940744 shutdown_connections 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.531+0000 7ff6cd5a8740 1 -- 172.21.15.27:0/3474940744 wait complete. 2024-04-25T01:35:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6cd5a8740 20 client.17781 trim_cache size 0 max 16384 2024-04-25T01:35:14.514 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:35:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:35:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "821a6923-d4f4-4dd2-9b5f-350a15502597", 2024-04-25T01:35:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:35:14.514 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6b00a2d00 msgr2=0x7ff6b00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6b00a2d00 0x7ff6b00a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40cc9e0 tx=0x7ff6b4097360 comp rx=0 tx=0).stop 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 msgr2=0x7ff6c4198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 0x7ff6c4198000 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400c970 tx=0x7ff6b400ce30 comp rx=0 tx=0).stop 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/980741429 shutdown_connections 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6c4107730 0x7ff6c419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6c406a780 0x7ff6c4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff6b00a2d00 0x7ff6b00a51c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 --2- 172.21.15.27:0/980741429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6c4069dc0 0x7ff6c4198000 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/980741429 >> 172.21.15.27:0/980741429 conn(0x7ff6c40776a0 msgr2=0x7ff6c40fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/980741429 shutdown_connections 2024-04-25T01:35:14.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.532+0000 7ff6ca8a3640 1 -- 172.21.15.27:0/980741429 wait complete. 2024-04-25T01:35:14.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:35:14.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 -- 172.21.15.27:0/4148222717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f541066b0 msgr2=0x7f6f54106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 --2- 172.21.15.27:0/4148222717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f541066b0 0x7f6f54106a90 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009cd0 tx=0x7f6f4402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 -- 172.21.15.27:0/4148222717 shutdown_connections 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 --2- 172.21.15.27:0/4148222717 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f54100fd0 0x7f6f5410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 --2- 172.21.15.27:0/4148222717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f54100610 0x7f6f54100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 --2- 172.21.15.27:0/4148222717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f541066b0 0x7f6f54106a90 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 -- 172.21.15.27:0/4148222717 >> 172.21.15.27:0/4148222717 conn(0x7f6f54077740 msgr2=0x7f6f54077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 -- 172.21.15.27:0/4148222717 shutdown_connections 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.652+0000 7f6f597f9640 1 -- 172.21.15.27:0/4148222717 wait complete. 2024-04-25T01:35:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f597f9640 1 Processor -- start 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f597f9640 1 -- start start 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f597f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f54100610 0x7f6f54197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f597f9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f54100fd0 0x7f6f541983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f597f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f541066b0 0x7f6f5419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f597f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6f54110a40 con 0x7f6f54100610 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f597f9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6f541108c0 con 0x7f6f54100fd0 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f597f9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f54110bc0 con 0x7f6f541066b0 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f54100610 0x7f6f54197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f54100610 0x7f6f54197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40258/0 (socket says 172.21.15.27:40258) 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.653+0000 7f6f52ffd640 1 -- 172.21.15.27:0/2957346966 learned_addr learned my addr 172.21.15.27:0/2957346966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f52ffd640 1 -- 172.21.15.27:0/2957346966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f541066b0 msgr2=0x7f6f5419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f541066b0 0x7f6f5419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f52ffd640 1 -- 172.21.15.27:0/2957346966 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f54100fd0 msgr2=0x7f6f541983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f54100fd0 0x7f6f541983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f52ffd640 1 -- 172.21.15.27:0/2957346966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5419cea0 con 0x7f6f54100610 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f54100610 0x7f6f54197eb0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000f80 tx=0x7f6f44003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f3bfff640 1 -- 172.21.15.27:0/2957346966 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44004240 con 0x7f6f54100610 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f597f9640 1 -- 172.21.15.27:0/2957346966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f5419d130 con 0x7f6f54100610 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f3bfff640 1 -- 172.21.15.27:0/2957346966 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f440305a0 con 0x7f6f54100610 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f3bfff640 1 -- 172.21.15.27:0/2957346966 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4403a660 con 0x7f6f54100610 2024-04-25T01:35:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.654+0000 7f6f597f9640 1 -- 172.21.15.27:0/2957346966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f541a4a20 con 0x7f6f54100610 2024-04-25T01:35:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.656+0000 7f6f3bfff640 1 -- 172.21.15.27:0/2957346966 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6f44030750 con 0x7f6f54100610 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.656+0000 7f6f5c5e9740 1 Processor -- start 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.656+0000 7f6f5c5e9740 1 -- start start 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 0x563b769d30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769d5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563b7692d6b0 con 0x563b769c8630 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563b768d5880 con 0x563b769d3620 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563b76929670 con 0x563b769c8f50 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f3bfff640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6f3c0a2d00 0x7f6f3c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40274/0 (socket says 172.21.15.27:40274) 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 -- 172.21.15.27:0/1436302015 learned_addr learned my addr 172.21.15.27:0/1436302015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6f3c0a2d00 0x7f6f3c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f3bfff640 1 -- 172.21.15.27:0/2957346966 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f6f4402e030 con 0x7f6f54100610 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f527fc640 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769d5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 0x563b769d30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 -- 172.21.15.27:0/1436302015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 msgr2=0x563b769d30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 0x563b769d30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 -- 172.21.15.27:0/1436302015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 msgr2=0x563b769d5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769d5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 -- 172.21.15.27:0/1436302015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b7670b7d0 con 0x563b769c8630 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 0x563b769d30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.657+0000 7f6f537fe640 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769c8a10 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0139e0 tx=0x7f6f4c013ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8704bd4b743eb05d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f537fe640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6f3c0a2d00 0x7f6f3c0a51c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c013fd0 tx=0x7f6f4c019000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f397fa640 1 -- 172.21.15.27:0/1436302015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c020070 con 0x563b769c8630 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f397fa640 1 -- 172.21.15.27:0/1436302015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f4c004020 con 0x563b769c8630 2024-04-25T01:35:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f397fa640 1 -- 172.21.15.27:0/1436302015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c005220 con 0x563b769c8630 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/1436302015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 msgr2=0x563b769c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769c8a10 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0139e0 tx=0x7f6f4c013ea0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/1436302015 shutdown_connections 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769d5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 0x563b769d30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/1436302015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769c8a10 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/1436302015 >> 172.21.15.27:0/1436302015 conn(0x563b769c4950 msgr2=0x563b769c4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/1436302015 shutdown_connections 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.658+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/1436302015 wait complete. 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 1 -- start start 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 1 Processor -- start 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 0x563b769d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769c8f50 0x563b769d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769d3620 0x563b769d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563b769d63b0 con 0x563b769d3620 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563b768e8f50 con 0x563b769c8f50 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563b769d6530 con 0x563b769c8630 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f5c5e9740 10 client.0 _ll_register_callbacks cb 0x563b76924000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769d3620 0x563b769d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769d3620 0x563b769d10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40286/0 (socket says 172.21.15.27:40286) 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f537fe640 1 -- 172.21.15.27:0/2223555766 learned_addr learned my addr 172.21.15.27:0/2223555766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f537fe640 1 -- 172.21.15.27:0/2223555766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 msgr2=0x563b769d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f537fe640 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 0x563b769d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f537fe640 1 -- 172.21.15.27:0/2223555766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769c8f50 msgr2=0x563b769d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f537fe640 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769c8f50 0x563b769d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.659+0000 7f6f537fe640 1 -- 172.21.15.27:0/2223555766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b766413f0 con 0x563b769d3620 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f537fe640 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769d3620 0x563b769d10c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c005f30 tx=0x7f6f4c013270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eaf63939ea9f319 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f36ff5640 1 -- 172.21.15.27:0/2223555766 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c020030 con 0x563b769d3620 2024-04-25T01:35:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f36ff5640 1 -- 172.21.15.27:0/2223555766 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f6f4c005530 con 0x563b769d3620 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/2223555766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f4c004020 con 0x563b769d3620 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f36ff5640 1 -- 172.21.15.27:0/2223555766 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f4c0047a0 con 0x563b769d3620 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 10 client.17793 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/2223555766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563b766010f0 con 0x563b769d3620 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f36ff5640 1 -- 172.21.15.27:0/2223555766 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f4c004020 con 0x563b769d3620 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f36ff5640 1 -- 172.21.15.27:0/2223555766 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6f4c00c040 con 0x563b769d3620 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 10 client.17793 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 10 client.17793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 20 client.17793 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563b769e3890 0x563b769e5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 4 client.17793 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.660+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/2223555766 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b7693abf0 con 0x563b769e3890 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.661+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563b769e3890 0x563b769e5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.661+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563b769e3890 0x563b769e5c80 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=996209ce815dea11 server_cookie=751d066434ab06ed in_seq=0 out_seq=0 2024-04-25T01:35:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.661+0000 7f6f36ff5640 10 client.17793 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f36ff5640 1 -- 172.21.15.27:0/2223555766 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b7693abf0 con 0x563b769e3890 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f36ff5640 10 client.17793 handle_command_reply: tid=0 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 client.17793 shutdown 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/2223555766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769d3620 msgr2=0x563b769d10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769d3620 0x563b769d10c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c005f30 tx=0x7f6f4c013270 comp rx=0 tx=0).stop 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/2223555766 shutdown_connections 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563b769e3890 0x563b769e5c80 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769d3620 0x563b769d10c0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769c8f50 0x563b769d0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f36ff5640 0 client.17793 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/2223555766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 0x563b769d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/2223555766 >> 172.21.15.27:0/2223555766 conn(0x563b769c4950 msgr2=0x563b769ca750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/2223555766 shutdown_connections 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/2223555766 wait complete. 2024-04-25T01:35:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.662+0000 7f6f5c5e9740 20 client.17793 trim_cache size 0 max 16384 2024-04-25T01:35:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f5c5e9740 1 Processor -- start 2024-04-25T01:35:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f5c5e9740 1 -- start start 2024-04-25T01:35:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769d2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 0x563b769cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769df270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6f541b1650 con 0x563b769c8630 2024-04-25T01:35:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6f541aef20 con 0x563b769d3620 2024-04-25T01:35:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f541acd90 con 0x563b769c8f50 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769d2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769d2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40290/0 (socket says 172.21.15.27:40290) 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 -- 172.21.15.27:0/925566933 learned_addr learned my addr 172.21.15.27:0/925566933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 -- 172.21.15.27:0/925566933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 msgr2=0x563b769cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/925566933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 0x563b769cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 -- 172.21.15.27:0/925566933 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 msgr2=0x563b769df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/925566933 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769df270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 -- 172.21.15.27:0/925566933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b769b6090 con 0x563b769c8630 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/925566933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769d2d70 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f6f440bc9b0 tx=0x7f6f440b8f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de98744ddb98fd5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.680+0000 7f6f38ff9640 1 -- 172.21.15.27:0/925566933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f440c5920 con 0x563b769c8630 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f38ff9640 1 -- 172.21.15.27:0/925566933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f44036040 con 0x563b769c8630 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f38ff9640 1 -- 172.21.15.27:0/925566933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f440ee050 con 0x563b769c8630 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/925566933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 msgr2=0x563b769d2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/925566933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769d2d70 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f6f440bc9b0 tx=0x7f6f440b8f50 comp rx=0 tx=0).stop 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/925566933 shutdown_connections 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/925566933 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769df270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/925566933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8f50 0x563b769cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/925566933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8630 0x563b769d2d70 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/925566933 >> 172.21.15.27:0/925566933 conn(0x563b769c4950 msgr2=0x563b769c3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/925566933 shutdown_connections 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/925566933 wait complete. 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 -- start start 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.681+0000 7f6f5c5e9740 1 Processor -- start 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 0x563b769d7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8f50 0x563b769d7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f5c5e9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769d7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6f541ad3e0 con 0x563b769c8f50 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6f541ae850 con 0x563b769d3620 2024-04-25T01:35:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f5c5e9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f541adab0 con 0x563b769c8630 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f5c5e9740 10 client.0 _ll_register_callbacks cb 0x7f6f48005f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8f50 0x563b769d7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8f50 0x563b769d7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40306/0 (socket says 172.21.15.27:40306) 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 -- 172.21.15.27:0/3732027862 learned_addr learned my addr 172.21.15.27:0/3732027862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 0x563b769d7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f537fe640 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769d7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 -- 172.21.15.27:0/3732027862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 msgr2=0x563b769d7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 0x563b769d7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 -- 172.21.15.27:0/3732027862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 msgr2=0x563b769d7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769d7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 -- 172.21.15.27:0/3732027862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b76a4d020 con 0x563b769c8f50 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f527fc640 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8f50 0x563b769d7690 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x563b769c5ae0 tx=0x7f6f4800fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf9478e39b00edf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f36ff5640 1 -- 172.21.15.27:0/3732027862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48004220 con 0x563b769c8f50 2024-04-25T01:35:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f36ff5640 1 -- 172.21.15.27:0/3732027862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f6f480043c0 con 0x563b769c8f50 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.682+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/3732027862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f44036040 con 0x563b769c8f50 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f36ff5640 1 -- 172.21.15.27:0/3732027862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f48023050 con 0x563b769c8f50 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f5c5e9740 10 client.17799 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/3732027862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563b76a4e360 con 0x563b769c8f50 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f36ff5640 1 -- 172.21.15.27:0/3732027862 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f44036040 con 0x563b769c8f50 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f36ff5640 1 -- 172.21.15.27:0/3732027862 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6f48003ac0 con 0x563b769c8f50 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f5c5e9740 10 client.17799 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f5c5e9740 10 client.17799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f5c5e9740 20 client.17799 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563b76a4eb60 0x563b76a50f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f5c5e9740 4 client.17799 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/3732027862 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b76934f40 con 0x563b76a4eb60 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.683+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563b76a4eb60 0x563b76a50f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f52ffd640 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563b76a4eb60 0x563b76a50f80 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38d8d3e5e31cb018 server_cookie=d20673bb03198cfa in_seq=0 out_seq=0 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f36ff5640 10 client.17799 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f36ff5640 1 -- 172.21.15.27:0/3732027862 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563b76934f40 con 0x563b76a4eb60 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f36ff5640 10 client.17799 handle_command_reply: tid=0 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 client.17799 shutdown 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/3732027862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8f50 msgr2=0x563b769d7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8f50 0x563b769d7690 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x563b769c5ae0 tx=0x7f6f4800fa10 comp rx=0 tx=0).stop 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/3732027862 shutdown_connections 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563b76a4eb60 0x563b76a50f80 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563b769d3620 0x563b769d7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b769c8f50 0x563b769d7690 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f36ff5640 0 client.17799 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 --2- 172.21.15.27:0/3732027862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563b769c8630 0x563b769d7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/3732027862 >> 172.21.15.27:0/3732027862 conn(0x563b769c4950 msgr2=0x563b769d05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/3732027862 shutdown_connections 2024-04-25T01:35:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 1 -- 172.21.15.27:0/3732027862 wait complete. 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.684+0000 7f6f5c5e9740 20 client.17799 trim_cache size 0 max 16384 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "821a6923-d4f4-4dd2-9b5f-350a15502597": { 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "821a6923-d4f4-4dd2-9b5f-350a15502597", 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:35:14.668 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:35:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 -- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6f3c0a2d00 msgr2=0x7f6f3c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6f3c0a2d00 0x7f6f3c0a51c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c013fd0 tx=0x7f6f4c019000 comp rx=0 tx=0).stop 2024-04-25T01:35:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 -- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f54100610 msgr2=0x7f6f54197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f54100610 0x7f6f54197eb0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000f80 tx=0x7f6f44003c10 comp rx=0 tx=0).stop 2024-04-25T01:35:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 -- 172.21.15.27:0/2957346966 shutdown_connections 2024-04-25T01:35:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f541066b0 0x7f6f5419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f54100fd0 0x7f6f541983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6f3c0a2d00 0x7f6f3c0a51c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 --2- 172.21.15.27:0/2957346966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f54100610 0x7f6f54197eb0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 -- 172.21.15.27:0/2957346966 >> 172.21.15.27:0/2957346966 conn(0x7f6f54077740 msgr2=0x7f6f541045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 -- 172.21.15.27:0/2957346966 shutdown_connections 2024-04-25T01:35:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:14.685+0000 7f6f597f9640 1 -- 172.21.15.27:0/2957346966 wait complete. 2024-04-25T01:35:14.671 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:14 smithi027 ceph-mon[29197]: pgmap v304: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:14.671 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:14 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:35:14.671 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:14 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:14.671 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:14 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:35:14.672 INFO:tasks.cephfs.filesystem:scrub status for tag:821a6923-d4f4-4dd2-9b5f-350a15502597 - {'path': '/', 'tag': '821a6923-d4f4-4dd2-9b5f-350a15502597', 'options': 'recursive,force'} 2024-04-25T01:35:15.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:14 smithi153 ceph-mon[32621]: pgmap v304: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:15.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:14 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:35:15.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:14 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:15.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:14 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:35:15.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:14 smithi139 ceph-mon[32760]: pgmap v304: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:15.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:14 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:35:15.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:14 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:15.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:14 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:35:15.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:15 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:35:15.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:15 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:15.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:15 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:15.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:15 smithi027 ceph-mon[29197]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T01:35:16.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:15 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:35:16.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:15 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:16.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:15 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:16.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:15 smithi153 ceph-mon[32621]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T01:35:16.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:15 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:35:16.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:15 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:16.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:15 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:16.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:15 smithi139 ceph-mon[32760]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T01:35:16.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:16 smithi027 ceph-mon[29197]: pgmap v305: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:17.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:16 smithi153 ceph-mon[32621]: pgmap v305: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:17.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:16 smithi139 ceph-mon[32760]: pgmap v305: 129 pgs: 129 active+clean; 4.7 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:35:18.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:18 smithi027 ceph-mon[29197]: pgmap v307: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:35:19.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:18 smithi153 ceph-mon[32621]: pgmap v307: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:35:19.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:18 smithi139 ceph-mon[32760]: pgmap v307: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:35:20.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:20 smithi027 ceph-mon[29197]: pgmap v308: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:21.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:20 smithi153 ceph-mon[32621]: pgmap v308: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:21.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:20 smithi139 ceph-mon[32760]: pgmap v308: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:22 smithi027 ceph-mon[29197]: pgmap v309: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:22.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:22 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:35:23.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:22 smithi153 ceph-mon[32621]: pgmap v309: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:23.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:22 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:35:23.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:22 smithi139 ceph-mon[32760]: pgmap v309: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:23.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:22 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:35:23.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:23 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:35:24.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:23 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:35:24.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:23 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:35:24.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:24 smithi027 ceph-mon[29197]: pgmap v310: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:25.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:24 smithi153 ceph-mon[32621]: pgmap v310: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:25.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:24 smithi139 ceph-mon[32760]: pgmap v310: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:26.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:26 smithi027 ceph-mon[29197]: pgmap v311: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:27.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:26 smithi153 ceph-mon[32621]: pgmap v311: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:27.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:26 smithi139 ceph-mon[32760]: pgmap v311: 129 pgs: 129 active+clean; 4.8 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-25T01:35:28.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:28 smithi027 ceph-mon[29197]: pgmap v312: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-25T01:35:29.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:28 smithi153 ceph-mon[32621]: pgmap v312: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-25T01:35:29.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:28 smithi139 ceph-mon[32760]: pgmap v312: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-25T01:35:30.840 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:30 smithi139 ceph-mon[32760]: pgmap v313: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:35:30.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:30 smithi027 ceph-mon[29197]: pgmap v313: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:35:31.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:30 smithi153 ceph-mon[32621]: pgmap v313: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:35:32.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:32 smithi027 ceph-mon[29197]: pgmap v314: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:35:33.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:32 smithi153 ceph-mon[32621]: pgmap v314: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:35:33.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:32 smithi139 ceph-mon[32760]: pgmap v314: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:35:34.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:34 smithi027 ceph-mon[29197]: pgmap v315: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:35.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:34 smithi153 ceph-mon[32621]: pgmap v315: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:35.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:34 smithi139 ceph-mon[32760]: pgmap v315: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:36.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:36 smithi027 ceph-mon[29197]: pgmap v316: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:37.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:36 smithi153 ceph-mon[32621]: pgmap v316: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:37.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:36 smithi139 ceph-mon[32760]: pgmap v316: 129 pgs: 129 active+clean; 4.9 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:38.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:38 smithi027 ceph-mon[29197]: pgmap v317: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:35:38.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:38 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:39.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:38 smithi153 ceph-mon[32621]: pgmap v317: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:35:39.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:38 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:39.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:38 smithi139 ceph-mon[32760]: pgmap v317: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-25T01:35:39.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:38 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:40.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: pgmap v318: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:40.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:40.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: pgmap v318: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:40.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:40.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:40.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: pgmap v318: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 9 op/s 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:40.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:35:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:41 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T01:35:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:41 smithi027 ceph-mon[29197]: Unable to set osd_memory_target on smithi139 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-25T01:35:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:41 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T01:35:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:41 smithi027 ceph-mon[29197]: Unable to set osd_memory_target on smithi153 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T01:35:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:41 smithi027 ceph-mon[29197]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T01:35:41.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:41 smithi027 ceph-mon[29197]: Unable to set osd_memory_target on smithi027 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:35:41.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:41 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T01:35:41.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:41 smithi153 ceph-mon[32621]: Unable to set osd_memory_target on smithi139 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-25T01:35:41.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:41 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T01:35:41.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:41 smithi153 ceph-mon[32621]: Unable to set osd_memory_target on smithi153 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T01:35:41.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:41 smithi153 ceph-mon[32621]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T01:35:41.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:41 smithi153 ceph-mon[32621]: Unable to set osd_memory_target on smithi027 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:35:41.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:41 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi139 to 176.2M 2024-04-25T01:35:41.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:41 smithi139 ceph-mon[32760]: Unable to set osd_memory_target on smithi139 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-25T01:35:41.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:41 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-25T01:35:41.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:41 smithi139 ceph-mon[32760]: Unable to set osd_memory_target on smithi153 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T01:35:41.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:41 smithi139 ceph-mon[32760]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T01:35:41.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:41 smithi139 ceph-mon[32760]: Unable to set osd_memory_target on smithi027 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:35:42.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:42 smithi153 ceph-mon[32621]: pgmap v319: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:35:42.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:42 smithi139 ceph-mon[32760]: pgmap v319: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:35:42.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:42 smithi027 ceph-mon[29197]: pgmap v319: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-25T01:35:44.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:35:44.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:35:44.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:44 smithi153 ceph-mon[32621]: pgmap v320: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:35:44.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:44 smithi139 ceph-mon[32760]: pgmap v320: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:35:44.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.816+0000 7f028e163640 1 -- 172.21.15.27:0/2062338878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 msgr2=0x7f0288107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.816+0000 7f028e163640 1 --2- 172.21.15.27:0/2062338878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 0x7f0288107b50 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f0278009930 tx=0x7f027802a420 comp rx=0 tx=0).stop 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 -- 172.21.15.27:0/2062338878 shutdown_connections 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 --2- 172.21.15.27:0/2062338878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0288102120 0x7f028810e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 --2- 172.21.15.27:0/2062338878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0288101760 0x7f0288101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 --2- 172.21.15.27:0/2062338878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 0x7f0288107b50 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 -- 172.21.15.27:0/2062338878 >> 172.21.15.27:0/2062338878 conn(0x7f02880fd250 msgr2=0x7f02880ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 -- 172.21.15.27:0/2062338878 shutdown_connections 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 -- 172.21.15.27:0/2062338878 wait complete. 2024-04-25T01:35:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 Processor -- start 2024-04-25T01:35:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.817+0000 7f028e163640 1 -- start start 2024-04-25T01:35:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f028e163640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0288101760 0x7f02881980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f028e163640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0288102120 0x7f02881985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f028e163640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 0x7f028819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f028e163640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0288111b80 con 0x7f0288107770 2024-04-25T01:35:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f028e163640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0288111a00 con 0x7f0288102120 2024-04-25T01:35:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f028e163640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0288111d00 con 0x7f0288101760 2024-04-25T01:35:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 0x7f028819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 0x7f028819c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43960/0 (socket says 172.21.15.27:43960) 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 -- 172.21.15.27:0/1783442450 learned_addr learned my addr 172.21.15.27:0/1783442450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f02877fe640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0288101760 0x7f02881980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0286ffd640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0288102120 0x7f02881985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 -- 172.21.15.27:0/1783442450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0288101760 msgr2=0x7f02881980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0288101760 0x7f02881980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 -- 172.21.15.27:0/1783442450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0288102120 msgr2=0x7f02881985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0288102120 0x7f02881985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 -- 172.21.15.27:0/1783442450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028819d090 con 0x7f0288107770 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0287fff640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 0x7f028819c960 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f028000eaa0 tx=0x7f028000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.818+0000 7f0284ff9640 1 -- 172.21.15.27:0/1783442450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028000cdb0 con 0x7f0288107770 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.819+0000 7f028e163640 1 -- 172.21.15.27:0/1783442450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028819d380 con 0x7f0288107770 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.819+0000 7f0284ff9640 1 -- 172.21.15.27:0/1783442450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0280005020 con 0x7f0288107770 2024-04-25T01:35:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.819+0000 7f0284ff9640 1 -- 172.21.15.27:0/1783442450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800116d0 con 0x7f0288107770 2024-04-25T01:35:44.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.819+0000 7f028e163640 1 -- 172.21.15.27:0/1783442450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02881a4c10 con 0x7f0288107770 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.821+0000 7f0284ff9640 1 -- 172.21.15.27:0/1783442450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f02800054d0 con 0x7f0288107770 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.821+0000 7f0290e75740 1 Processor -- start 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.821+0000 7f0290e75740 1 -- start start 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.821+0000 7f0284ff9640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f02700a2d00 0x7f02700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0290e75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55a630 0x55b06e55aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0290e75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e5650e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0290e75740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e567a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0290e75740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b06e4bf6b0 con 0x55b06e565620 2024-04-25T01:35:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0290e75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b06e467880 con 0x55b06e55a630 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0290e75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b06e4bb670 con 0x55b06e55af50 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0286ffd640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f02700a2d00 0x7f02700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0284ff9640 1 -- 172.21.15.27:0/1783442450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f02800cf970 con 0x7f0288107770 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e567a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e567a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43970/0 (socket says 172.21.15.27:43970) 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 -- 172.21.15.27:0/1687265271 learned_addr learned my addr 172.21.15.27:0/1687265271 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f02877fe640 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55a630 0x55b06e55aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0286ffd640 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e5650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 -- 172.21.15.27:0/1687265271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 msgr2=0x55b06e5650e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e5650e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 -- 172.21.15.27:0/1687265271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55a630 msgr2=0x55b06e55aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55a630 0x55b06e55aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 -- 172.21.15.27:0/1687265271 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b06e29d7d0 con 0x55b06e565620 2024-04-25T01:35:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0286ffd640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f02700a2d00 0x7f02700a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f027c0058a0 tx=0x7f027c0128d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f0287fff640 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e567a10 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f0280098520 tx=0x7f02800151b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a04d4d69e4ff7e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f026dffb640 1 -- 172.21.15.27:0/1687265271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028009bb50 con 0x55b06e565620 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f026dffb640 1 -- 172.21.15.27:0/1687265271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f028009fe50 con 0x55b06e565620 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.822+0000 7f026dffb640 1 -- 172.21.15.27:0/1687265271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0280007e30 con 0x55b06e565620 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 -- 172.21.15.27:0/1687265271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 msgr2=0x55b06e567a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e567a10 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f0280098520 tx=0x7f02800151b0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 -- 172.21.15.27:0/1687265271 shutdown_connections 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e567a10 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e5650e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 --2- 172.21.15.27:0/1687265271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55a630 0x55b06e55aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 -- 172.21.15.27:0/1687265271 >> 172.21.15.27:0/1687265271 conn(0x55b06e556950 msgr2=0x55b06e556d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 -- 172.21.15.27:0/1687265271 shutdown_connections 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.823+0000 7f0290e75740 1 -- 172.21.15.27:0/1687265271 wait complete. 2024-04-25T01:35:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 1 -- start start 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 1 Processor -- start 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 0x55b06e562620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55af50 0x55b06e562b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e5630a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b06e568410 con 0x55b06e55af50 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b06e47af50 con 0x55b06e565620 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b06e568590 con 0x55b06e55a630 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0290e75740 10 client.0 _ll_register_callbacks cb 0x55b06e4b6000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55af50 0x55b06e562b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55af50 0x55b06e562b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43978/0 (socket says 172.21.15.27:43978) 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0286ffd640 1 -- 172.21.15.27:0/2552956736 learned_addr learned my addr 172.21.15.27:0/2552956736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f02877fe640 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 0x55b06e562620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.824+0000 7f0287fff640 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e5630a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0286ffd640 1 -- 172.21.15.27:0/2552956736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 msgr2=0x55b06e562620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0286ffd640 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 0x55b06e562620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0286ffd640 1 -- 172.21.15.27:0/2552956736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 msgr2=0x55b06e5630a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0286ffd640 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e5630a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0286ffd640 1 -- 172.21.15.27:0/2552956736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b06e1d33f0 con 0x55b06e55af50 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0286ffd640 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55af50 0x55b06e562b60 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f027c00e900 tx=0x7f027c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a65a8f926dae15a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0266ffd640 1 -- 172.21.15.27:0/2552956736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c00fd50 con 0x55b06e55af50 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0266ffd640 1 -- 172.21.15.27:0/2552956736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f027c01f3f0 con 0x55b06e55af50 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0290e75740 1 -- 172.21.15.27:0/2552956736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f028009fe50 con 0x55b06e55af50 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0266ffd640 1 -- 172.21.15.27:0/2552956736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f027c01f5f0 con 0x55b06e55af50 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0290e75740 10 client.17811 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0290e75740 1 -- 172.21.15.27:0/2552956736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b06e1930f0 con 0x55b06e55af50 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.825+0000 7f0266ffd640 1 -- 172.21.15.27:0/2552956736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f028009fe50 con 0x55b06e55af50 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f0266ffd640 1 -- 172.21.15.27:0/2552956736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f027c027760 con 0x55b06e55af50 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f0290e75740 10 client.17811 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f0290e75740 10 client.17811 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f0290e75740 20 client.17811 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f0290e75740 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b06e575960 0x55b06e577d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f0290e75740 4 client.17811 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f0290e75740 1 -- 172.21.15.27:0/2552956736 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b06e4ccbf0 con 0x55b06e575960 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f02877fe640 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b06e575960 0x55b06e577d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f02877fe640 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b06e575960 0x55b06e577d90 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ef3b030259b2784 server_cookie=3a2f55660a62a077 in_seq=0 out_seq=0 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.826+0000 7f0266ffd640 10 client.17811 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0266ffd640 1 -- 172.21.15.27:0/2552956736 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b06e4ccbf0 con 0x55b06e575960 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0266ffd640 10 client.17811 handle_command_reply: tid=0 2024-04-25T01:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 client.17811 shutdown 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 -- 172.21.15.27:0/2552956736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55af50 msgr2=0x55b06e562b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55af50 0x55b06e562b60 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f027c00e900 tx=0x7f027c004730 comp rx=0 tx=0).stop 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 -- 172.21.15.27:0/2552956736 shutdown_connections 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b06e575960 0x55b06e577d90 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e5630a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55af50 0x55b06e562b60 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0266ffd640 0 client.17811 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 --2- 172.21.15.27:0/2552956736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 0x55b06e562620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 -- 172.21.15.27:0/2552956736 >> 172.21.15.27:0/2552956736 conn(0x55b06e556950 msgr2=0x55b06e55c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.827+0000 7f0290e75740 1 -- 172.21.15.27:0/2552956736 shutdown_connections 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.828+0000 7f0290e75740 1 -- 172.21.15.27:0/2552956736 wait complete. 2024-04-25T01:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.828+0000 7f0290e75740 20 client.17811 trim_cache size 0 max 16384 2024-04-25T01:35:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.845+0000 7f0290e75740 1 Processor -- start 2024-04-25T01:35:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.845+0000 7f0290e75740 1 -- start start 2024-04-25T01:35:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.845+0000 7f0290e75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 0x55b06e556270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0290e75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55af50 0x55b06e56dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0290e75740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e55cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0290e75740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f02881b1660 con 0x55b06e565620 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0290e75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f02881aef30 con 0x55b06e55af50 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0290e75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02881acda0 con 0x55b06e55a630 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e55cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e55cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43990/0 (socket says 172.21.15.27:43990) 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 -- 172.21.15.27:0/3019306492 learned_addr learned my addr 172.21.15.27:0/3019306492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f02877fe640 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 0x55b06e556270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0286ffd640 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55af50 0x55b06e56dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 -- 172.21.15.27:0/3019306492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 msgr2=0x55b06e556270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 0x55b06e556270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 -- 172.21.15.27:0/3019306492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55af50 msgr2=0x55b06e56dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55af50 0x55b06e56dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 -- 172.21.15.27:0/3019306492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b06e567d80 con 0x55b06e565620 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f0287fff640 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e55cdd0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f02800a41e0 tx=0x7f0280000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c70d89617740e3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f026d7fa640 1 -- 172.21.15.27:0/3019306492 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800a4710 con 0x55b06e565620 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f026d7fa640 1 -- 172.21.15.27:0/3019306492 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0280007dc0 con 0x55b06e565620 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.846+0000 7f026d7fa640 1 -- 172.21.15.27:0/3019306492 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0280016210 con 0x55b06e565620 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 -- 172.21.15.27:0/3019306492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 msgr2=0x55b06e55cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e55cdd0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f02800a41e0 tx=0x7f0280000f40 comp rx=0 tx=0).stop 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 -- 172.21.15.27:0/3019306492 shutdown_connections 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e565620 0x55b06e55cdd0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e55af50 0x55b06e56dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 --2- 172.21.15.27:0/3019306492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55a630 0x55b06e556270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 -- 172.21.15.27:0/3019306492 >> 172.21.15.27:0/3019306492 conn(0x55b06e556950 msgr2=0x55b06e563d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 -- 172.21.15.27:0/3019306492 shutdown_connections 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 -- 172.21.15.27:0/3019306492 wait complete. 2024-04-25T01:35:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 -- start start 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.847+0000 7f0290e75740 1 Processor -- start 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0290e75740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55a630 0x55b06e55ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0290e75740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e55e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0290e75740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e55edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0290e75740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f02881ad3f0 con 0x55b06e55a630 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0290e75740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f02881ae860 con 0x55b06e565620 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0290e75740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02881adac0 con 0x55b06e55af50 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0290e75740 10 client.0 _ll_register_callbacks cb 0x7f027c00e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55a630 0x55b06e55ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55a630 0x55b06e55ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44002/0 (socket says 172.21.15.27:44002) 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 -- 172.21.15.27:0/1963008678 learned_addr learned my addr 172.21.15.27:0/1963008678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0287fff640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e55edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 -- 172.21.15.27:0/1963008678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 msgr2=0x55b06e55e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0286ffd640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e55e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e55e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 -- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 msgr2=0x55b06e55edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e55edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 -- 172.21.15.27:0/1963008678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b06e55f3b0 con 0x55b06e55a630 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0287fff640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e55edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0286ffd640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e55e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f02877fe640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55a630 0x55b06e55ffa0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f0278009900 tx=0x7f0278031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e660cb5a2013fdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0266ffd640 1 -- 172.21.15.27:0/1963008678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027804c070 con 0x55b06e55a630 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0266ffd640 1 -- 172.21.15.27:0/1963008678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f0278004450 con 0x55b06e55a630 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.848+0000 7f0290e75740 1 -- 172.21.15.27:0/1963008678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0280007dc0 con 0x55b06e55a630 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0266ffd640 1 -- 172.21.15.27:0/1963008678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0278059050 con 0x55b06e55a630 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0290e75740 10 client.17817 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:44.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0290e75740 1 -- 172.21.15.27:0/1963008678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b06e5e0d50 con 0x55b06e55a630 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0266ffd640 1 -- 172.21.15.27:0/1963008678 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0280007dc0 con 0x55b06e55a630 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0266ffd640 1 -- 172.21.15.27:0/1963008678 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f02780047a0 con 0x55b06e55a630 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0290e75740 10 client.17817 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0290e75740 10 client.17817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0290e75740 20 client.17817 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0290e75740 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b06e5e1480 0x55b06e5e38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0290e75740 4 client.17817 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0290e75740 1 -- 172.21.15.27:0/1963008678 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b06e4c6f40 con 0x55b06e5e1480 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.849+0000 7f0286ffd640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b06e5e1480 0x55b06e5e38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0266ffd640 10 client.17817 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0286ffd640 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b06e5e1480 0x55b06e5e38a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5c53a1d17c6129 server_cookie=b81896954c2cd1be in_seq=0 out_seq=0 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0266ffd640 1 -- 172.21.15.27:0/1963008678 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b06e4c6f40 con 0x55b06e5e1480 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0266ffd640 10 client.17817 handle_command_reply: tid=0 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 client.17817 shutdown 2024-04-25T01:35:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 -- 172.21.15.27:0/1963008678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55a630 msgr2=0x55b06e55ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55a630 0x55b06e55ffa0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f0278009900 tx=0x7f0278031a60 comp rx=0 tx=0).stop 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 -- 172.21.15.27:0/1963008678 shutdown_connections 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55b06e5e1480 0x55b06e5e38a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b06e565620 0x55b06e55edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b06e55af50 0x55b06e55e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0266ffd640 0 client.17817 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 --2- 172.21.15.27:0/1963008678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b06e55a630 0x55b06e55ffa0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 -- 172.21.15.27:0/1963008678 >> 172.21.15.27:0/1963008678 conn(0x55b06e556950 msgr2=0x55b06e56d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 -- 172.21.15.27:0/1963008678 shutdown_connections 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 1 -- 172.21.15.27:0/1963008678 wait complete. 2024-04-25T01:35:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.850+0000 7f0290e75740 20 client.17817 trim_cache size 0 max 16384 2024-04-25T01:35:44.835 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:35:44.835 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:35:44.835 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:35:44.835 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:35:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 -- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f02700a2d00 msgr2=0x7f02700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f02700a2d00 0x7f02700a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f027c0058a0 tx=0x7f027c0128d0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 -- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 msgr2=0x7f028819c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 0x7f028819c960 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f028000eaa0 tx=0x7f028000ef60 comp rx=0 tx=0).stop 2024-04-25T01:35:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 -- 172.21.15.27:0/1783442450 shutdown_connections 2024-04-25T01:35:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f02700a2d00 0x7f02700a51c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0288107770 0x7f028819c960 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0288102120 0x7f02881985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 --2- 172.21.15.27:0/1783442450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0288101760 0x7f02881980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 -- 172.21.15.27:0/1783442450 >> 172.21.15.27:0/1783442450 conn(0x7f02880fd250 msgr2=0x7f02880fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:44.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 -- 172.21.15.27:0/1783442450 shutdown_connections 2024-04-25T01:35:44.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.851+0000 7f028e163640 1 -- 172.21.15.27:0/1783442450 wait complete. 2024-04-25T01:35:44.838 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:44 smithi027 ceph-mon[29197]: pgmap v320: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:35:44.839 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:35:44.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 -- 172.21.15.27:0/1024103325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138102a20 msgr2=0x7fb138102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 --2- 172.21.15.27:0/1024103325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138102a20 0x7fb138102e00 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fb120009900 tx=0x7fb12002a480 comp rx=0 tx=0).stop 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 -- 172.21.15.27:0/1024103325 shutdown_connections 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 --2- 172.21.15.27:0/1024103325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb13810bd60 0x7fb13810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 --2- 172.21.15.27:0/1024103325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb138103340 0x7fb13810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 --2- 172.21.15.27:0/1024103325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138102a20 0x7fb138102e00 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 -- 172.21.15.27:0/1024103325 >> 172.21.15.27:0/1024103325 conn(0x7fb1380777a0 msgr2=0x7fb138077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 -- 172.21.15.27:0/1024103325 shutdown_connections 2024-04-25T01:35:44.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 -- 172.21.15.27:0/1024103325 wait complete. 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 Processor -- start 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.965+0000 7fb13f038640 1 -- start start 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13f038640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb138102a20 0x7fb138100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13f038640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138103340 0x7fb138100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13f038640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb13810bd60 0x7fb1381010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13f038640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb138111b30 con 0x7fb138103340 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13f038640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb1381119b0 con 0x7fb13810bd60 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13f038640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb138111cb0 con 0x7fb138102a20 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138103340 0x7fb138100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138103340 0x7fb138100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44016/0 (socket says 172.21.15.27:44016) 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 -- 172.21.15.27:0/53173070 learned_addr learned my addr 172.21.15.27:0/53173070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13cdd4640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb138102a20 0x7fb138100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13d5d5640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb13810bd60 0x7fb1381010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 -- 172.21.15.27:0/53173070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb138102a20 msgr2=0x7fb138100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb138102a20 0x7fb138100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 -- 172.21.15.27:0/53173070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb13810bd60 msgr2=0x7fb1381010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb13810bd60 0x7fb1381010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 -- 172.21.15.27:0/53173070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1381a6450 con 0x7fb138103340 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb13d5d5640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb13810bd60 0x7fb1381010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb137fff640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138103340 0x7fb138100bb0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fb12800c920 tx=0x7fb12800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb135ffb640 1 -- 172.21.15.27:0/53173070 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128007c70 con 0x7fb138103340 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.966+0000 7fb135ffb640 1 -- 172.21.15.27:0/53173070 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb128007e10 con 0x7fb138103340 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.967+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1381a6740 con 0x7fb138103340 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.967+0000 7fb135ffb640 1 -- 172.21.15.27:0/53173070 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128018890 con 0x7fb138103340 2024-04-25T01:35:44.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.967+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1381a6c90 con 0x7fb138103340 2024-04-25T01:35:44.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.968+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1380689b0 con 0x7fb138103340 2024-04-25T01:35:44.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.969+0000 7fb135ffb640 1 -- 172.21.15.27:0/53173070 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb128018a30 con 0x7fb138103340 2024-04-25T01:35:44.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.969+0000 7fb135ffb640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1240a2d30 0x7fb1240a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:44.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.969+0000 7fb13cdd4640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1240a2d30 0x7fb1240a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:44.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.969+0000 7fb135ffb640 1 -- 172.21.15.27:0/53173070 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fb1280cf960 con 0x7fb138103340 2024-04-25T01:35:44.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.970+0000 7fb13cdd4640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1240a2d30 0x7fb1240a51f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb120009230 tx=0x7fb1200058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:44.974+0000 7fb135ffb640 1 -- 172.21.15.27:0/53173070 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb128014030 con 0x7fb138103340 2024-04-25T01:35:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.199+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb138106ee0 con 0x7fb138103340 2024-04-25T01:35:45.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.199+0000 7fb135ffb640 1 -- 172.21.15.27:0/53173070 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7fb12809c190 con 0x7fb138103340 2024-04-25T01:35:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:35:45.177 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:35:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1240a2d30 msgr2=0x7fb1240a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1240a2d30 0x7fb1240a51f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb120009230 tx=0x7fb1200058d0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138103340 msgr2=0x7fb138100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138103340 0x7fb138100bb0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fb12800c920 tx=0x7fb12800cde0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 shutdown_connections 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb13810bd60 0x7fb1381010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb1240a2d30 0x7fb1240a51f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb138103340 0x7fb138100bb0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 --2- 172.21.15.27:0/53173070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb138102a20 0x7fb138100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.201+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 >> 172.21.15.27:0/53173070 conn(0x7fb1380777a0 msgr2=0x7fb1380fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.202+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 shutdown_connections 2024-04-25T01:35:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.202+0000 7fb13f038640 1 -- 172.21.15.27:0/53173070 wait complete. 2024-04-25T01:35:45.189 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:35:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 -- 172.21.15.27:0/2617945538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c010bd00 msgr2=0x7f78c010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2617945538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c010bd00 0x7f78c010e1a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009ca0 tx=0x7f78b002a9e0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 -- 172.21.15.27:0/2617945538 shutdown_connections 2024-04-25T01:35:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2617945538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c010bd00 0x7f78c010e1a0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2617945538 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78c01032e0 0x7f78c010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2617945538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78c01029c0 0x7f78c0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 -- 172.21.15.27:0/2617945538 >> 172.21.15.27:0/2617945538 conn(0x7f78c0077740 msgr2=0x7f78c0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 -- 172.21.15.27:0/2617945538 shutdown_connections 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.316+0000 7f78c51e7640 1 -- 172.21.15.27:0/2617945538 wait complete. 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.317+0000 7f78c51e7640 1 Processor -- start 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.317+0000 7f78c51e7640 1 -- start start 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.317+0000 7f78c51e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c01029c0 0x7f78c0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.317+0000 7f78c51e7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78c01032e0 0x7f78c01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.317+0000 7f78c51e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78c010bd00 0x7f78c019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.317+0000 7f78c51e7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78c0111b40 con 0x7f78c01029c0 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.317+0000 7f78c51e7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01119c0 con 0x7f78c01032e0 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.317+0000 7f78c51e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78c0111cc0 con 0x7f78c010bd00 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c01029c0 0x7f78c0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c01029c0 0x7f78c0198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44042/0 (socket says 172.21.15.27:44042) 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 -- 172.21.15.27:0/2568192178 learned_addr learned my addr 172.21.15.27:0/2568192178 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78be59c640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78c01032e0 0x7f78c01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bf59e640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78c010bd00 0x7f78c019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 -- 172.21.15.27:0/2568192178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78c010bd00 msgr2=0x7f78c019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78c010bd00 0x7f78c019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 -- 172.21.15.27:0/2568192178 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78c01032e0 msgr2=0x7f78c01985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78c01032e0 0x7f78c01985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 -- 172.21.15.27:0/2568192178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c019d020 con 0x7f78c01029c0 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78bed9d640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c01029c0 0x7f78c0198080 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f78b800b4a0 tx=0x7f78b800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78a7fff640 1 -- 172.21.15.27:0/2568192178 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8014070 con 0x7f78c01029c0 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78c51e7640 1 -- 172.21.15.27:0/2568192178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c019d310 con 0x7f78c01029c0 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78a7fff640 1 -- 172.21.15.27:0/2568192178 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b8010af0 con 0x7f78c01029c0 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78a7fff640 1 -- 172.21.15.27:0/2568192178 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8017d10 con 0x7f78c01029c0 2024-04-25T01:35:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.318+0000 7f78c51e7640 1 -- 172.21.15.27:0/2568192178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c01a4ba0 con 0x7f78c01029c0 2024-04-25T01:35:45.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.321+0000 7f78a7fff640 1 -- 172.21.15.27:0/2568192178 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f78b8017070 con 0x7f78c01029c0 2024-04-25T01:35:45.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.321+0000 7f78c7fd4740 1 Processor -- start 2024-04-25T01:35:45.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.321+0000 7f78c7fd4740 1 -- start start 2024-04-25T01:35:45.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.321+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5c0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5cb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56127e525670 con 0x56127e5c0ea0 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56127e4cd970 con 0x56127e5c0580 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56127e521630 con 0x56127e5cb570 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44048/0 (socket says 172.21.15.27:44048) 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 -- 172.21.15.27:0/4101338123 learned_addr learned my addr 172.21.15.27:0/4101338123 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78be59c640 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bf59e640 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5c0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78a7fff640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78a80a2d00 0x7f78a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 -- 172.21.15.27:0/4101338123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 msgr2=0x56127e5cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 -- 172.21.15.27:0/4101338123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 msgr2=0x56127e5c0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5c0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 -- 172.21.15.27:0/4101338123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56127e4f1400 con 0x56127e5c0ea0 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bf59e640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78a80a2d00 0x7f78a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78bed9d640 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5cb030 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f78b80cbf00 tx=0x7f78b80c5920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8549cd274b529689 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78a7fff640 1 -- 172.21.15.27:0/2568192178 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f78b80cebb0 con 0x7f78c01029c0 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.322+0000 7f78a57fa640 1 -- 172.21.15.27:0/4101338123 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b809a6f0 con 0x56127e5c0ea0 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78a57fa640 1 -- 172.21.15.27:0/4101338123 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b809b490 con 0x56127e5c0ea0 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78a57fa640 1 -- 172.21.15.27:0/4101338123 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b80633c0 con 0x56127e5c0ea0 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 -- 172.21.15.27:0/4101338123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 msgr2=0x56127e5cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78bf59e640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78a80a2d00 0x7f78a80a51c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009430 tx=0x7f78b00095c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5cb030 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f78b80cbf00 tx=0x7f78b80c5920 comp rx=0 tx=0).stop 2024-04-25T01:35:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 -- 172.21.15.27:0/4101338123 shutdown_connections 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5cb030 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/4101338123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5c0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 -- 172.21.15.27:0/4101338123 >> 172.21.15.27:0/4101338123 conn(0x56127e5bc8a0 msgr2=0x56127e5bcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 -- 172.21.15.27:0/4101338123 shutdown_connections 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.323+0000 7f78c7fd4740 1 -- 172.21.15.27:0/4101338123 wait complete. 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 1 -- start start 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 1 Processor -- start 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0580 0x56127e5caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0ea0 0x56127e5c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5c8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56127e5ce300 con 0x56127e5c0580 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56127e4e1050 con 0x56127e5c0ea0 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56127e5ce480 con 0x56127e5cb570 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78c7fd4740 10 client.0 _ll_register_callbacks cb 0x56127e238be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0580 0x56127e5caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0580 0x56127e5caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44052/0 (socket says 172.21.15.27:44052) 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 -- 172.21.15.27:0/1129808300 learned_addr learned my addr 172.21.15.27:0/1129808300 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78be59c640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0ea0 0x56127e5c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bf59e640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 -- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 msgr2=0x56127e5c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 -- 172.21.15.27:0/1129808300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0ea0 msgr2=0x56127e5c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0ea0 0x56127e5c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 -- 172.21.15.27:0/1129808300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56127e4b2dc0 con 0x56127e5c0580 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78be59c640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0ea0 0x56127e5c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bf59e640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78bed9d640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0580 0x56127e5caec0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f78b8097760 tx=0x7f78b80d2130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cc25ca825d2c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.324+0000 7f78a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1129808300 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b801e070 con 0x56127e5c0580 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1129808300 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f78b8017360 con 0x56127e5c0580 2024-04-25T01:35:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1129808300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78b809b490 con 0x56127e5c0580 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1129808300 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f78b80a3c50 con 0x56127e5c0580 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 10 client.17835 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1129808300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56127e2393f0 con 0x56127e5c0580 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1129808300 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b809b490 con 0x56127e5c0580 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1129808300 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f78b80cb100 con 0x56127e5c0580 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 10 client.17835 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 10 client.17835 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 20 client.17835 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56127e5db420 0x56127e5dd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 4 client.17835 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.325+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1129808300 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56127e532bb0 con 0x56127e5db420 2024-04-25T01:35:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.326+0000 7f78be59c640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56127e5db420 0x56127e5dd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.326+0000 7f78be59c640 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56127e5db420 0x56127e5dd810 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f380f2c55f850980 server_cookie=1e8e21ab9052ec5 in_seq=0 out_seq=0 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.326+0000 7f78a2ff5640 10 client.17835 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1129808300 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56127e532bb0 con 0x56127e5db420 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78a2ff5640 10 client.17835 handle_command_reply: tid=0 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 client.17835 shutdown 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1129808300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0580 msgr2=0x56127e5caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0580 0x56127e5caec0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f78b8097760 tx=0x7f78b80d2130 comp rx=0 tx=0).stop 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1129808300 shutdown_connections 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56127e5db420 0x56127e5dd810 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78a2ff5640 0 client.17835 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:35:45.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0ea0 0x56127e5c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1129808300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0580 0x56127e5caec0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1129808300 >> 172.21.15.27:0/1129808300 conn(0x56127e5bc8a0 msgr2=0x56127e5bbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1129808300 shutdown_connections 2024-04-25T01:35:45.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1129808300 wait complete. 2024-04-25T01:35:45.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.327+0000 7f78c7fd4740 20 client.17835 trim_cache size 0 max 16384 2024-04-25T01:35:45.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.340+0000 7f78c7fd4740 1 Processor -- start 2024-04-25T01:35:45.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.340+0000 7f78c7fd4740 1 -- start start 2024-04-25T01:35:45.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.340+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.340+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5c0ea0 0x56127e5e0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.340+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5cb570 0x56127e5e0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.340+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01b1730 con 0x56127e5cb570 2024-04-25T01:35:45.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.340+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01af000 con 0x56127e5c0580 2024-04-25T01:35:45.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.340+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01ace70 con 0x56127e5c0ea0 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5cb570 0x56127e5e0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5cb570 0x56127e5e0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44058/0 (socket says 172.21.15.27:44058) 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 -- 172.21.15.27:0/2113273244 learned_addr learned my addr 172.21.15.27:0/2113273244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78be59c640 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5c0ea0 0x56127e5e0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bed9d640 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 -- 172.21.15.27:0/2113273244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5c0ea0 msgr2=0x56127e5e0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5c0ea0 0x56127e5e0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 -- 172.21.15.27:0/2113273244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 msgr2=0x56127e5ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 -- 172.21.15.27:0/2113273244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56127e3037d0 con 0x56127e5cb570 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78bf59e640 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5cb570 0x56127e5e0d70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009ca0 tx=0x7f78b0041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2acdc769b00792 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78a4ff9640 1 -- 172.21.15.27:0/2113273244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0049040 con 0x56127e5cb570 2024-04-25T01:35:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78a4ff9640 1 -- 172.21.15.27:0/2113273244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b0053070 con 0x56127e5cb570 2024-04-25T01:35:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.341+0000 7f78a4ff9640 1 -- 172.21.15.27:0/2113273244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0033040 con 0x56127e5cb570 2024-04-25T01:35:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- 172.21.15.27:0/2113273244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5cb570 msgr2=0x56127e5e0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5cb570 0x56127e5e0d70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009ca0 tx=0x7f78b0041000 comp rx=0 tx=0).stop 2024-04-25T01:35:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- 172.21.15.27:0/2113273244 shutdown_connections 2024-04-25T01:35:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5cb570 0x56127e5e0d70 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5c0ea0 0x56127e5e0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/2113273244 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- 172.21.15.27:0/2113273244 >> 172.21.15.27:0/2113273244 conn(0x56127e5bc8a0 msgr2=0x56127e5c2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- 172.21.15.27:0/2113273244 shutdown_connections 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- 172.21.15.27:0/2113273244 wait complete. 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- start start 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 Processor -- start 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5de2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5de7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5ded20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01ad4c0 con 0x56127e5c0ea0 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01ae930 con 0x56127e5c0580 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01adb90 con 0x56127e5cb570 2024-04-25T01:35:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78c7fd4740 10 client.0 _ll_register_callbacks cb 0x7f78b0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:45.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.342+0000 7f78be59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5de7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78be59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5de7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44070/0 (socket says 172.21.15.27:44070) 2024-04-25T01:35:45.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78be59c640 1 -- 172.21.15.27:0/1946327978 learned_addr learned my addr 172.21.15.27:0/1946327978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78bf59e640 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78bed9d640 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5de2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78be59c640 1 -- 172.21.15.27:0/1946327978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 msgr2=0x56127e5ded20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78be59c640 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5ded20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78be59c640 1 -- 172.21.15.27:0/1946327978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 msgr2=0x56127e5de2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78be59c640 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5de2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78be59c640 1 -- 172.21.15.27:0/1946327978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56127e640630 con 0x56127e5c0ea0 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78bed9d640 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5de2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78be59c640 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5de7e0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f78b400bd60 tx=0x7f78b4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295f633f455c9af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:45.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1946327978 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4023060 con 0x56127e5c0ea0 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1946327978 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f78b4004440 con 0x56127e5c0ea0 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1946327978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78b0053070 con 0x56127e5c0ea0 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1946327978 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f78b402a050 con 0x56127e5c0ea0 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78c7fd4740 10 client.17841 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1946327978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56127e641970 con 0x56127e5c0ea0 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.343+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1946327978 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b0053070 con 0x56127e5c0ea0 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1946327978 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f78b401c3d0 con 0x56127e5c0ea0 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78c7fd4740 10 client.17841 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78c7fd4740 10 client.17841 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78c7fd4740 20 client.17841 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56127e642120 0x56127e644540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78c7fd4740 4 client.17841 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1946327978 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56127e52cf00 con 0x56127e642120 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78bed9d640 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56127e642120 0x56127e644540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78a2ff5640 10 client.17841 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.344+0000 7f78bed9d640 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56127e642120 0x56127e644540 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ece7d5549d31ad11 server_cookie=191033fe094a5cc8 in_seq=0 out_seq=0 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78a2ff5640 1 -- 172.21.15.27:0/1946327978 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56127e52cf00 con 0x56127e642120 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78a2ff5640 10 client.17841 handle_command_reply: tid=0 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 client.17841 shutdown 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1946327978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 msgr2=0x56127e5de7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5de7e0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f78b400bd60 tx=0x7f78b4011810 comp rx=0 tx=0).stop 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1946327978 shutdown_connections 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56127e642120 0x56127e644540 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56127e5cb570 0x56127e5ded20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56127e5c0ea0 0x56127e5de7e0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78a2ff5640 0 client.17841 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:35:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 --2- 172.21.15.27:0/1946327978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56127e5c0580 0x56127e5de2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1946327978 >> 172.21.15.27:0/1946327978 conn(0x56127e5bc8a0 msgr2=0x56127e5d6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1946327978 shutdown_connections 2024-04-25T01:35:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 1 -- 172.21.15.27:0/1946327978 wait complete. 2024-04-25T01:35:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c7fd4740 20 client.17841 trim_cache size 0 max 16384 2024-04-25T01:35:45.333 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:35:45.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 -- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78a80a2d00 msgr2=0x7f78a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78a80a2d00 0x7f78a80a51c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009430 tx=0x7f78b00095c0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 -- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c01029c0 msgr2=0x7f78c0198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c01029c0 0x7f78c0198080 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f78b800b4a0 tx=0x7f78b800b960 comp rx=0 tx=0).stop 2024-04-25T01:35:45.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 -- 172.21.15.27:0/2568192178 shutdown_connections 2024-04-25T01:35:45.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78c010bd00 0x7f78c019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78c01032e0 0x7f78c01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f78a80a2d00 0x7f78a80a51c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 --2- 172.21.15.27:0/2568192178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c01029c0 0x7f78c0198080 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.345+0000 7f78c51e7640 1 -- 172.21.15.27:0/2568192178 >> 172.21.15.27:0/2568192178 conn(0x7f78c0077740 msgr2=0x7f78c00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.346+0000 7f78c51e7640 1 -- 172.21.15.27:0/2568192178 shutdown_connections 2024-04-25T01:35:45.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.346+0000 7f78c51e7640 1 -- 172.21.15.27:0/2568192178 wait complete. 2024-04-25T01:35:45.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 -- 172.21.15.27:0/2864048564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb41066b0 msgr2=0x7f5eb4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 --2- 172.21.15.27:0/2864048564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb41066b0 0x7f5eb4106a90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009930 tx=0x7f5ea802a420 comp rx=0 tx=0).stop 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 -- 172.21.15.27:0/2864048564 shutdown_connections 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 --2- 172.21.15.27:0/2864048564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5eb4100fd0 0x7f5eb410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 --2- 172.21.15.27:0/2864048564 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5eb4100610 0x7f5eb4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 --2- 172.21.15.27:0/2864048564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb41066b0 0x7f5eb4106a90 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 -- 172.21.15.27:0/2864048564 >> 172.21.15.27:0/2864048564 conn(0x7f5eb4077740 msgr2=0x7f5eb4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 -- 172.21.15.27:0/2864048564 shutdown_connections 2024-04-25T01:35:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 -- 172.21.15.27:0/2864048564 wait complete. 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 Processor -- start 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.464+0000 7f5ebb238640 1 -- start start 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5ebb238640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb4100610 0x7f5eb4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5ebb238640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5eb4100fd0 0x7f5eb4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5ebb238640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5eb41066b0 0x7f5eb419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5ebb238640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb4110a30 con 0x7f5eb4100610 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5ebb238640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb41108b0 con 0x7f5eb4100fd0 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5ebb238640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb4110bb0 con 0x7f5eb41066b0 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb4100610 0x7f5eb4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb4100610 0x7f5eb4197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44080/0 (socket says 172.21.15.27:44080) 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/3161593288 learned_addr learned my addr 172.21.15.27:0/3161593288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5eb41066b0 0x7f5eb419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5eb4100fd0 0x7f5eb4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/3161593288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5eb41066b0 msgr2=0x7f5eb419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5eb41066b0 0x7f5eb419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/3161593288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5eb4100fd0 msgr2=0x7f5eb4198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5eb4100fd0 0x7f5eb4198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/3161593288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb419cf60 con 0x7f5eb4100610 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5eb4100fd0 0x7f5eb4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.465+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb4100610 0x7f5eb4197fc0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8005940 tx=0x7f5ea80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.466+0000 7f5eb1ffb640 1 -- 172.21.15.27:0/3161593288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea803f030 con 0x7f5eb4100610 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.466+0000 7f5ebb238640 1 -- 172.21.15.27:0/3161593288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb419d1f0 con 0x7f5eb4100610 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.466+0000 7f5eb1ffb640 1 -- 172.21.15.27:0/3161593288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea8003ce0 con 0x7f5eb4100610 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.466+0000 7f5eb1ffb640 1 -- 172.21.15.27:0/3161593288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8035070 con 0x7f5eb4100610 2024-04-25T01:35:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.466+0000 7f5ebb238640 1 -- 172.21.15.27:0/3161593288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb41a4ae0 con 0x7f5eb4100610 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.468+0000 7f5eb1ffb640 1 -- 172.21.15.27:0/3161593288 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5ea8042030 con 0x7f5eb4100610 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5ebe021740 1 Processor -- start 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5ebe021740 1 -- start start 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x564285952960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x56428595d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x56428595f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5ebe021740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5642858b7670 con 0x564285952ea0 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5ebe021740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56428585f970 con 0x56428595d570 2024-04-25T01:35:45.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5ebe021740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5642858b3630 con 0x564285952580 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5eb1ffb640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5ea40a2d00 0x7f5ea40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5eb8fd4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x56428595d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5eb8fd4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x56428595d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44082/0 (socket says 172.21.15.27:44082) 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/2081740448 learned_addr learned my addr 172.21.15.27:0/2081740448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5ea40a2d00 0x7f5ea40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x56428595f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x564285952960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.469+0000 7f5eb1ffb640 1 -- 172.21.15.27:0/3161593288 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f5ea80efdd0 con 0x7f5eb4100610 2024-04-25T01:35:45.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/2081740448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 msgr2=0x564285952960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x564285952960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/2081740448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 msgr2=0x56428595f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x56428595f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/2081740448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564285883400 con 0x564285952ea0 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5ea40a2d00 0x7f5ea40a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098b0 tx=0x7f5eac009840 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x56428595d030 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80ed130 tx=0x7f5ea8047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22707cf984ac81f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5e9effd640 1 -- 172.21.15.27:0/2081740448 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8049070 con 0x564285952ea0 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5e9effd640 1 -- 172.21.15.27:0/2081740448 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea80c0250 con 0x564285952ea0 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.470+0000 7f5e9effd640 1 -- 172.21.15.27:0/2081740448 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea80b8510 con 0x564285952ea0 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 -- 172.21.15.27:0/2081740448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 msgr2=0x56428595d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x56428595d030 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80ed130 tx=0x7f5ea8047000 comp rx=0 tx=0).stop 2024-04-25T01:35:45.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 -- 172.21.15.27:0/2081740448 shutdown_connections 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x56428595f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x56428595d030 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2081740448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x564285952960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 -- 172.21.15.27:0/2081740448 >> 172.21.15.27:0/2081740448 conn(0x56428594e8a0 msgr2=0x56428594ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 -- 172.21.15.27:0/2081740448 shutdown_connections 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 -- 172.21.15.27:0/2081740448 wait complete. 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 -- start start 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.471+0000 7f5ebe021740 1 Processor -- start 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x56428595cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564285952ea0 0x56428595a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56428595d570 0x56428595a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5ebe021740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5642858b7670 con 0x56428595d570 2024-04-25T01:35:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5ebe021740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564285873050 con 0x564285952ea0 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5ebe021740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564285960120 con 0x564285952580 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5ebe021740 10 client.0 _ll_register_callbacks cb 0x5642855cabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56428595d570 0x56428595a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56428595d570 0x56428595a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44094/0 (socket says 172.21.15.27:44094) 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 -- 172.21.15.27:0/2918426647 learned_addr learned my addr 172.21.15.27:0/2918426647 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x56428595cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564285952ea0 0x56428595a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 -- 172.21.15.27:0/2918426647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 msgr2=0x56428595cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x56428595cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 -- 172.21.15.27:0/2918426647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564285952ea0 msgr2=0x56428595a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564285952ea0 0x56428595a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 -- 172.21.15.27:0/2918426647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564285844dc0 con 0x56428595d570 2024-04-25T01:35:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564285952ea0 0x56428595a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56428595d570 0x56428595a840 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f5eac010b10 tx=0x7f5eac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8dc5df2140685a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5e93fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5e93fff640 1 -- 172.21.15.27:0/2918426647 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eac025070 con 0x56428595d570 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5ebe021740 1 -- 172.21.15.27:0/2918426647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5642858c4bb0 con 0x56428595d570 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.472+0000 7f5e93fff640 1 -- 172.21.15.27:0/2918426647 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f5eac004d20 con 0x56428595d570 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5e93fff640 1 -- 172.21.15.27:0/2918426647 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5eac01eb10 con 0x56428595d570 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5ebe021740 10 client.17853 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5ebe021740 1 -- 172.21.15.27:0/2918426647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5642855cb3f0 con 0x56428595d570 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5e93fff640 1 -- 172.21.15.27:0/2918426647 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5642858c4bb0 con 0x56428595d570 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5e93fff640 1 -- 172.21.15.27:0/2918426647 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5eac00f040 con 0x56428595d570 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5ebe021740 10 client.17853 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5ebe021740 10 client.17853 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5ebe021740 20 client.17853 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56428596d350 0x56428596f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5ebe021740 4 client.17853 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5ebe021740 1 -- 172.21.15.27:0/2918426647 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642858bef00 con 0x56428596d350 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.473+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56428596d350 0x56428596f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.474+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56428596d350 0x56428596f7a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=42a70266ba827018 server_cookie=a8ccf701ada7e545 in_seq=0 out_seq=0 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.474+0000 7f5e93fff640 10 client.17853 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.474+0000 7f5e93fff640 1 -- 172.21.15.27:0/2918426647 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5642858bef00 con 0x56428596d350 2024-04-25T01:35:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.474+0000 7f5e93fff640 10 client.17853 handle_command_reply: tid=0 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.474+0000 7f5ebe021740 1 client.17853 shutdown 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 -- 172.21.15.27:0/2918426647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56428595d570 msgr2=0x56428595a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56428595d570 0x56428595a840 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f5eac010b10 tx=0x7f5eac004a40 comp rx=0 tx=0).stop 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 -- 172.21.15.27:0/2918426647 shutdown_connections 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x56428596d350 0x56428596f7a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56428595d570 0x56428595a840 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564285952ea0 0x56428595a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5e93fff640 0 client.17853 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 --2- 172.21.15.27:0/2918426647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x56428595cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 -- 172.21.15.27:0/2918426647 >> 172.21.15.27:0/2918426647 conn(0x56428594e8a0 msgr2=0x5642859546a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 -- 172.21.15.27:0/2918426647 shutdown_connections 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 1 -- 172.21.15.27:0/2918426647 wait complete. 2024-04-25T01:35:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.475+0000 7f5ebe021740 20 client.17853 trim_cache size 0 max 16384 2024-04-25T01:35:45.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.488+0000 7f5ebe021740 1 Processor -- start 2024-04-25T01:35:45.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.488+0000 7f5ebe021740 1 -- start start 2024-04-25T01:35:45.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x564285968e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x564285959390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x5642859750b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5ebe021740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb41b1620 con 0x564285952ea0 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5ebe021740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb41aeef0 con 0x56428595d570 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5ebe021740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb41acd60 con 0x564285952580 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x564285959390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x564285959390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44096/0 (socket says 172.21.15.27:44096) 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 -- 172.21.15.27:0/1234643477 learned_addr learned my addr 172.21.15.27:0/1234643477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x5642859750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x564285968e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 -- 172.21.15.27:0/1234643477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 msgr2=0x564285968e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x564285968e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 -- 172.21.15.27:0/1234643477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 msgr2=0x5642859750b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x5642859750b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 -- 172.21.15.27:0/1234643477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642856957d0 con 0x564285952ea0 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x564285959390 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000bdb0 tx=0x7f5ea0013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3dcfa0c1213b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5e9e7fc640 1 -- 172.21.15.27:0/1234643477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0004230 con 0x564285952ea0 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5e9e7fc640 1 -- 172.21.15.27:0/1234643477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea00043d0 con 0x564285952ea0 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.489+0000 7f5e9e7fc640 1 -- 172.21.15.27:0/1234643477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0011a40 con 0x564285952ea0 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 -- 172.21.15.27:0/1234643477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 msgr2=0x564285959390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x564285959390 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000bdb0 tx=0x7f5ea0013830 comp rx=0 tx=0).stop 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 -- 172.21.15.27:0/1234643477 shutdown_connections 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x5642859750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952ea0 0x564285959390 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 --2- 172.21.15.27:0/1234643477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952580 0x564285968e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 -- 172.21.15.27:0/1234643477 >> 172.21.15.27:0/1234643477 conn(0x56428594e8a0 msgr2=0x56428594d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 -- 172.21.15.27:0/1234643477 shutdown_connections 2024-04-25T01:35:45.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 -- 172.21.15.27:0/1234643477 wait complete. 2024-04-25T01:35:45.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.490+0000 7f5ebe021740 1 -- start start 2024-04-25T01:35:45.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5ebe021740 1 Processor -- start 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952580 0x564285962630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952ea0 0x564285962b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5ebe021740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x5642859d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5ebe021740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb41ad3b0 con 0x564285952580 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5ebe021740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb41ae820 con 0x56428595d570 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5ebe021740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb41ada80 con 0x564285952ea0 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5ebe021740 10 client.0 _ll_register_callbacks cb 0x7f5eac012e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5eb8fd4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952580 0x564285962630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5eb8fd4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952580 0x564285962630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44104/0 (socket says 172.21.15.27:44104) 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/3652760519 learned_addr learned my addr 172.21.15.27:0/3652760519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952ea0 0x564285962b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5eb97d5640 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x5642859d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.491+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952ea0 msgr2=0x564285962b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.492+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952ea0 0x564285962b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.492+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/3652760519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 msgr2=0x5642859d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.492+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x5642859d1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.492+0000 7f5eb8fd4640 1 -- 172.21.15.27:0/3652760519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642859d1f30 con 0x564285952580 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.492+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952ea0 0x564285962b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.492+0000 7f5eb8fd4640 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952580 0x564285962630 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80bc060 tx=0x7f5ea803b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9f281b578b6f4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.492+0000 7f5e93fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.493+0000 7f5e93fff640 1 -- 172.21.15.27:0/3652760519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8049040 con 0x564285952580 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.493+0000 7f5e93fff640 1 -- 172.21.15.27:0/3652760519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f5ea803b460 con 0x564285952580 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.493+0000 7f5ebe021740 1 -- 172.21.15.27:0/3652760519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ea00043d0 con 0x564285952580 2024-04-25T01:35:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.493+0000 7f5e93fff640 1 -- 172.21.15.27:0/3652760519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5ea8004460 con 0x564285952580 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5ebe021740 10 client.17859 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5ebe021740 1 -- 172.21.15.27:0/3652760519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5642859d3290 con 0x564285952580 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5e93fff640 1 -- 172.21.15.27:0/3652760519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea00043d0 con 0x564285952580 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5e93fff640 1 -- 172.21.15.27:0/3652760519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5ea803bc80 con 0x564285952580 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5ebe021740 10 client.17859 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5ebe021740 10 client.17859 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5ebe021740 20 client.17859 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5ebe021740 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5642859d3690 0x5642859d5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5ebe021740 4 client.17859 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5ebe021740 1 -- 172.21.15.27:0/3652760519 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56428583c820 con 0x5642859d3690 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.494+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5642859d3690 0x5642859d5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5eb3fff640 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5642859d3690 0x5642859d5a80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3607fcd2b4afc8b7 server_cookie=302f87e7c049b5ca in_seq=0 out_seq=0 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5e93fff640 10 client.17859 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:35:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5e93fff640 1 -- 172.21.15.27:0/3652760519 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56428583c820 con 0x5642859d3690 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5e93fff640 10 client.17859 handle_command_reply: tid=0 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 client.17859 shutdown 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 -- 172.21.15.27:0/3652760519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952580 msgr2=0x564285962630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952580 0x564285962630 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80bc060 tx=0x7f5ea803b720 comp rx=0 tx=0).stop 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 -- 172.21.15.27:0/3652760519 shutdown_connections 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x5642859d3690 0x5642859d5a80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56428595d570 0x5642859d1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5e93fff640 0 client.17859 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564285952ea0 0x564285962b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 --2- 172.21.15.27:0/3652760519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564285952580 0x564285962630 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 -- 172.21.15.27:0/3652760519 >> 172.21.15.27:0/3652760519 conn(0x56428594e8a0 msgr2=0x56428594d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 -- 172.21.15.27:0/3652760519 shutdown_connections 2024-04-25T01:35:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 1 -- 172.21.15.27:0/3652760519 wait complete. 2024-04-25T01:35:45.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.495+0000 7f5ebe021740 20 client.17859 trim_cache size 0 max 16384 2024-04-25T01:35:45.478 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 -- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5ea40a2d00 msgr2=0x7f5ea40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5ea40a2d00 0x7f5ea40a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098b0 tx=0x7f5eac009840 comp rx=0 tx=0).stop 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 -- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb4100610 msgr2=0x7f5eb4197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb4100610 0x7f5eb4197fc0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8005940 tx=0x7f5ea80042f0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 -- 172.21.15.27:0/3161593288 shutdown_connections 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5eb41066b0 0x7f5eb419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5eb4100fd0 0x7f5eb4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5ea40a2d00 0x7f5ea40a51c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 --2- 172.21.15.27:0/3161593288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eb4100610 0x7f5eb4197fc0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 -- 172.21.15.27:0/3161593288 >> 172.21.15.27:0/3161593288 conn(0x7f5eb4077740 msgr2=0x7f5eb40ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 -- 172.21.15.27:0/3161593288 shutdown_connections 2024-04-25T01:35:45.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.496+0000 7f5ebb238640 1 -- 172.21.15.27:0/3161593288 wait complete. 2024-04-25T01:35:45.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:35:45.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.611+0000 7f86791dc640 1 -- 172.21.15.27:0/3295994170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 msgr2=0x7f867410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.611+0000 7f86791dc640 1 --2- 172.21.15.27:0/3295994170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 0x7f867410d4f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f866c009cd0 tx=0x7f866c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.611+0000 7f86791dc640 1 -- 172.21.15.27:0/3295994170 shutdown_connections 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.611+0000 7f86791dc640 1 --2- 172.21.15.27:0/3295994170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 0x7f867410d4f0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.611+0000 7f86791dc640 1 --2- 172.21.15.27:0/3295994170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8674100620 0x7f8674100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.611+0000 7f86791dc640 1 --2- 172.21.15.27:0/3295994170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8674106630 0x7f8674106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.611+0000 7f86791dc640 1 -- 172.21.15.27:0/3295994170 >> 172.21.15.27:0/3295994170 conn(0x7f86740776a0 msgr2=0x7f8674077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.612+0000 7f86791dc640 1 -- 172.21.15.27:0/3295994170 shutdown_connections 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.612+0000 7f86791dc640 1 -- 172.21.15.27:0/3295994170 wait complete. 2024-04-25T01:35:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.612+0000 7f86791dc640 1 Processor -- start 2024-04-25T01:35:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.612+0000 7f86791dc640 1 -- start start 2024-04-25T01:35:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f86791dc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8674100620 0x7f867410df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f86791dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 0x7f867410e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f86791dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8674106630 0x7f867410ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f86791dc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8674111b90 con 0x7f8674100fe0 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f86791dc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8674111a10 con 0x7f8674100620 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f86791dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8674111d10 con 0x7f8674106630 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 0x7f867410e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 0x7f867410e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44122/0 (socket says 172.21.15.27:44122) 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 -- 172.21.15.27:0/3192371603 learned_addr learned my addr 172.21.15.27:0/3192371603 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 -- 172.21.15.27:0/3192371603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8674106630 msgr2=0x7f867410ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8674106630 0x7f867410ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 -- 172.21.15.27:0/3192371603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8674100620 msgr2=0x7f867410df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8674100620 0x7f867410df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 -- 172.21.15.27:0/3192371603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86741a6470 con 0x7f8674100fe0 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f867259c640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 0x7f867410e4c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f866400d950 tx=0x7f866400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f866bfff640 1 -- 172.21.15.27:0/3192371603 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8664017070 con 0x7f8674100fe0 2024-04-25T01:35:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f866bfff640 1 -- 172.21.15.27:0/3192371603 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8664009c30 con 0x7f8674100fe0 2024-04-25T01:35:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f86791dc640 1 -- 172.21.15.27:0/3192371603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86741a6760 con 0x7f8674100fe0 2024-04-25T01:35:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.613+0000 7f866bfff640 1 -- 172.21.15.27:0/3192371603 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866400f6b0 con 0x7f8674100fe0 2024-04-25T01:35:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.614+0000 7f86791dc640 1 -- 172.21.15.27:0/3192371603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86741a6d00 con 0x7f8674100fe0 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.616+0000 7f866bfff640 1 -- 172.21.15.27:0/3192371603 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f86640109d0 con 0x7f8674100fe0 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.616+0000 7f867bfc9740 1 Processor -- start 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.616+0000 7f867bfc9740 1 -- start start 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.616+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6580 0x5625ae4e6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.616+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.616+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 0x5625ae4f3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f867bfc9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5625ae44b670 con 0x5625ae4e6ea0 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f867bfc9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5625ae3f3970 con 0x5625ae4e6580 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f867bfc9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5625ae447630 con 0x5625ae4f1570 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44126/0 (socket says 172.21.15.27:44126) 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 -- 172.21.15.27:0/3894557130 learned_addr learned my addr 172.21.15.27:0/3894557130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f867359e640 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6580 0x5625ae4e6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f867259c640 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 0x5625ae4f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 -- 172.21.15.27:0/3894557130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 msgr2=0x5625ae4f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 0x5625ae4f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f866bfff640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2db0 0x7f86600a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 -- 172.21.15.27:0/3894557130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6580 msgr2=0x5625ae4e6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6580 0x5625ae4e6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 -- 172.21.15.27:0/3894557130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625ae417400 con 0x5625ae4e6ea0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f867359e640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2db0 0x7f86600a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f1030 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f865c00a960 tx=0x7f865c00ae20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0206b15e02eacbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f866bfff640 1 -- 172.21.15.27:0/3192371603 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f86640cf480 con 0x7f8674100fe0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f86697fa640 1 -- 172.21.15.27:0/3894557130 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c00cd80 con 0x5625ae4e6ea0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f86697fa640 1 -- 172.21.15.27:0/3894557130 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865c00f930 con 0x5625ae4e6ea0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.617+0000 7f86697fa640 1 -- 172.21.15.27:0/3894557130 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c018960 con 0x5625ae4e6ea0 2024-04-25T01:35:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 -- 172.21.15.27:0/3894557130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 msgr2=0x5625ae4f1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f1030 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f865c00a960 tx=0x7f865c00ae20 comp rx=0 tx=0).stop 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867359e640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2db0 0x7f86600a5270 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f866c004af0 tx=0x7f866c0098c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 -- 172.21.15.27:0/3894557130 shutdown_connections 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 0x5625ae4f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f1030 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3894557130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6580 0x5625ae4e6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 -- 172.21.15.27:0/3894557130 >> 172.21.15.27:0/3894557130 conn(0x5625ae4e28a0 msgr2=0x5625ae4e2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 -- 172.21.15.27:0/3894557130 shutdown_connections 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.618+0000 7f867bfc9740 1 -- 172.21.15.27:0/3894557130 wait complete. 2024-04-25T01:35:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 1 -- start start 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 1 Processor -- start 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 0x5625ae4ee210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4f1570 0x5625ae4ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5625ae4f4360 con 0x5625ae4f1570 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5625ae407050 con 0x5625ae4e6ea0 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5625ae4f44e0 con 0x5625ae4e6580 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867bfc9740 10 client.0 _ll_register_callbacks cb 0x5625ae15ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4f1570 0x5625ae4ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4f1570 0x5625ae4ee7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44138/0 (socket says 172.21.15.27:44138) 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867359e640 1 -- 172.21.15.27:0/2067806513 learned_addr learned my addr 172.21.15.27:0/2067806513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867259c640 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 0x5625ae4ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f8672d9d640 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867359e640 1 -- 172.21.15.27:0/2067806513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 msgr2=0x5625ae4f0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867359e640 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867359e640 1 -- 172.21.15.27:0/2067806513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 msgr2=0x5625ae4ee210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867359e640 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 0x5625ae4ee210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f867359e640 1 -- 172.21.15.27:0/2067806513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625ae3d8dc0 con 0x5625ae4f1570 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.619+0000 7f8672d9d640 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867359e640 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4f1570 0x5625ae4ee7b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f866c035000 tx=0x7f866c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b8ea9cf184f100 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f865a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f865a7fc640 1 -- 172.21.15.27:0/2067806513 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c04c070 con 0x5625ae4f1570 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f865a7fc640 1 -- 172.21.15.27:0/2067806513 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f866c0305a0 con 0x5625ae4f1570 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 1 -- 172.21.15.27:0/2067806513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f865c00f930 con 0x5625ae4f1570 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f865a7fc640 1 -- 172.21.15.27:0/2067806513 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f866c0316c0 con 0x5625ae4f1570 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 10 client.17871 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 1 -- 172.21.15.27:0/2067806513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5625ae15f3f0 con 0x5625ae4f1570 2024-04-25T01:35:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f865a7fc640 1 -- 172.21.15.27:0/2067806513 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865c00f930 con 0x5625ae4f1570 2024-04-25T01:35:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f865a7fc640 1 -- 172.21.15.27:0/2067806513 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f866c044070 con 0x5625ae4f1570 2024-04-25T01:35:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 10 client.17871 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 10 client.17871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 20 client.17871 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5625ae5015a0 0x5625ae503a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 4 client.17871 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.620+0000 7f867bfc9740 1 -- 172.21.15.27:0/2067806513 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625ae458bb0 con 0x5625ae5015a0 2024-04-25T01:35:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.621+0000 7f8672d9d640 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5625ae5015a0 0x5625ae503a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.621+0000 7f8672d9d640 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5625ae5015a0 0x5625ae503a10 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd17db321e13a760 server_cookie=cb9a45c6cc502036 in_seq=0 out_seq=0 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.621+0000 7f865a7fc640 10 client.17871 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f865a7fc640 1 -- 172.21.15.27:0/2067806513 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625ae458bb0 con 0x5625ae5015a0 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f865a7fc640 10 client.17871 handle_command_reply: tid=0 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 client.17871 shutdown 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 -- 172.21.15.27:0/2067806513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4f1570 msgr2=0x5625ae4ee7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4f1570 0x5625ae4ee7b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f866c035000 tx=0x7f866c0095c0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 -- 172.21.15.27:0/2067806513 shutdown_connections 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5625ae5015a0 0x5625ae503a10 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4f1570 0x5625ae4ee7b0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 0x5625ae4ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f865a7fc640 0 client.17871 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 --2- 172.21.15.27:0/2067806513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 -- 172.21.15.27:0/2067806513 >> 172.21.15.27:0/2067806513 conn(0x5625ae4e28a0 msgr2=0x5625ae4e1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 -- 172.21.15.27:0/2067806513 shutdown_connections 2024-04-25T01:35:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 1 -- 172.21.15.27:0/2067806513 wait complete. 2024-04-25T01:35:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.622+0000 7f867bfc9740 20 client.17871 trim_cache size 0 max 16384 2024-04-25T01:35:45.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.635+0000 7f867bfc9740 1 Processor -- start 2024-04-25T01:35:45.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.635+0000 7f867bfc9740 1 -- start start 2024-04-25T01:35:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6580 0x5625ae4fd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 0x5625ae5092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 0x5625ae509820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f867bfc9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f86741b3850 con 0x5625ae4e6580 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f867bfc9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f86741b1120 con 0x5625ae4e6ea0 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f867bfc9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86741aef90 con 0x5625ae4f1570 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6580 0x5625ae4fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6580 0x5625ae4fd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44142/0 (socket says 172.21.15.27:44142) 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 -- 172.21.15.27:0/3585189541 learned_addr learned my addr 172.21.15.27:0/3585189541 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f867259c640 1 --2- 172.21.15.27:0/3585189541 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 0x5625ae5092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 -- 172.21.15.27:0/3585189541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 msgr2=0x5625ae509820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3585189541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 0x5625ae509820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 -- 172.21.15.27:0/3585189541 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 msgr2=0x5625ae5092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3585189541 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 0x5625ae5092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 -- 172.21.15.27:0/3585189541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625ae2297d0 con 0x5625ae4e6580 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3585189541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6580 0x5625ae4fd220 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c6a0 tx=0x7f865c018540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81d99c57e41e159 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8668ff9640 1 -- 172.21.15.27:0/3585189541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c00ce80 con 0x5625ae4e6580 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8668ff9640 1 -- 172.21.15.27:0/3585189541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865c01e040 con 0x5625ae4e6580 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.636+0000 7f8668ff9640 1 -- 172.21.15.27:0/3585189541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c003850 con 0x5625ae4e6580 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 -- 172.21.15.27:0/3585189541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6580 msgr2=0x5625ae4fd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3585189541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6580 0x5625ae4fd220 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c6a0 tx=0x7f865c018540 comp rx=0 tx=0).stop 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 -- 172.21.15.27:0/3585189541 shutdown_connections 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3585189541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4f1570 0x5625ae509820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3585189541 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4e6ea0 0x5625ae5092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3585189541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6580 0x5625ae4fd220 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 -- 172.21.15.27:0/3585189541 >> 172.21.15.27:0/3585189541 conn(0x5625ae4e28a0 msgr2=0x5625ae4e1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 -- 172.21.15.27:0/3585189541 shutdown_connections 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 -- 172.21.15.27:0/3585189541 wait complete. 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.637+0000 7f867bfc9740 1 -- start start 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867bfc9740 1 Processor -- start 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867bfc9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4f1570 0x5625ae4f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867bfc9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f86741af5e0 con 0x5625ae4e6ea0 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867bfc9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f86741b0a50 con 0x5625ae4f1570 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867bfc9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86741afcb0 con 0x5625ae4e6580 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867bfc9740 10 client.0 _ll_register_callbacks cb 0x7f866c031d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44152/0 (socket says 172.21.15.27:44152) 2024-04-25T01:35:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 -- 172.21.15.27:0/3702125142 learned_addr learned my addr 172.21.15.27:0/3702125142 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867359e640 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4f1570 0x5625ae4f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 -- 172.21.15.27:0/3702125142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 msgr2=0x5625ae4f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 -- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4f1570 msgr2=0x5625ae4f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4f1570 0x5625ae4f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 -- 172.21.15.27:0/3702125142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625ae566c60 con 0x5625ae4e6ea0 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867359e640 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4f1570 0x5625ae4f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f865a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f867259c640 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f77a0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f866409b2b0 tx=0x7f8664010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea0d078252b53ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.638+0000 7f865a7fc640 1 -- 172.21.15.27:0/3702125142 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8664020040 con 0x5625ae4e6ea0 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f865a7fc640 1 -- 172.21.15.27:0/3702125142 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f86640cca60 con 0x5625ae4e6ea0 2024-04-25T01:35:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 1 -- 172.21.15.27:0/3702125142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f865c01e040 con 0x5625ae4e6ea0 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f865a7fc640 1 -- 172.21.15.27:0/3702125142 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f86640b4050 con 0x5625ae4e6ea0 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 10 client.17877 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 1 -- 172.21.15.27:0/3702125142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5625ae567fa0 con 0x5625ae4e6ea0 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f865a7fc640 1 -- 172.21.15.27:0/3702125142 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865c01e040 con 0x5625ae4e6ea0 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f865a7fc640 1 -- 172.21.15.27:0/3702125142 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8664097c20 con 0x5625ae4e6ea0 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 10 client.17877 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 10 client.17877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 20 client.17877 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5625ae568750 0x5625ae56ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 4 client.17877 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f867bfc9740 1 -- 172.21.15.27:0/3702125142 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625ae452f00 con 0x5625ae568750 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.639+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5625ae568750 0x5625ae56ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f8672d9d640 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5625ae568750 0x5625ae56ab70 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb624684f4824570 server_cookie=afa41593bc7bba3 in_seq=0 out_seq=0 2024-04-25T01:35:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f865a7fc640 10 client.17877 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f865a7fc640 1 -- 172.21.15.27:0/3702125142 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625ae452f00 con 0x5625ae568750 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f865a7fc640 10 client.17877 handle_command_reply: tid=0 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 client.17877 shutdown 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 -- 172.21.15.27:0/3702125142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 msgr2=0x5625ae4f77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f77a0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f866409b2b0 tx=0x7f8664010d70 comp rx=0 tx=0).stop 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 -- 172.21.15.27:0/3702125142 shutdown_connections 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5625ae568750 0x5625ae56ab70 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5625ae4f1570 0x5625ae4f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f865a7fc640 0 client.17877 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5625ae4e6ea0 0x5625ae4f77a0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 --2- 172.21.15.27:0/3702125142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625ae4e6580 0x5625ae4f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 -- 172.21.15.27:0/3702125142 >> 172.21.15.27:0/3702125142 conn(0x5625ae4e28a0 msgr2=0x5625ae4e1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 -- 172.21.15.27:0/3702125142 shutdown_connections 2024-04-25T01:35:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 1 -- 172.21.15.27:0/3702125142 wait complete. 2024-04-25T01:35:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.640+0000 7f867bfc9740 20 client.17877 trim_cache size 0 max 16384 2024-04-25T01:35:45.624 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 -- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2db0 msgr2=0x7f86600a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2db0 0x7f86600a5270 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f866c004af0 tx=0x7f866c0098c0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 -- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 msgr2=0x7f867410e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 0x7f867410e4c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f866400d950 tx=0x7f866400de10 comp rx=0 tx=0).stop 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 -- 172.21.15.27:0/3192371603 shutdown_connections 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8674106630 0x7f867410ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f86600a2db0 0x7f86600a5270 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8674100fe0 0x7f867410e4c0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 --2- 172.21.15.27:0/3192371603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8674100620 0x7f867410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 -- 172.21.15.27:0/3192371603 >> 172.21.15.27:0/3192371603 conn(0x7f86740776a0 msgr2=0x7f8674103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 -- 172.21.15.27:0/3192371603 shutdown_connections 2024-04-25T01:35:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:45.641+0000 7f86791dc640 1 -- 172.21.15.27:0/3192371603 wait complete. 2024-04-25T01:35:45.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:45 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/53173070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:45.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:45 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/53173070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:45.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:45 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/53173070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:46.628 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:35:46.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:35:46.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:35:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 -- 172.21.15.27:0/61893180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc102120 msgr2=0x7f5dfc10e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 --2- 172.21.15.27:0/61893180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc102120 0x7f5dfc10e630 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009cd0 tx=0x7f5df402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 -- 172.21.15.27:0/61893180 shutdown_connections 2024-04-25T01:35:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 --2- 172.21.15.27:0/61893180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc102120 0x7f5dfc10e630 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 --2- 172.21.15.27:0/61893180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dfc101760 0x7f5dfc101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 --2- 172.21.15.27:0/61893180 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dfc107770 0x7f5dfc107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 -- 172.21.15.27:0/61893180 >> 172.21.15.27:0/61893180 conn(0x7f5dfc0fd250 msgr2=0x7f5dfc0ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 -- 172.21.15.27:0/61893180 shutdown_connections 2024-04-25T01:35:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.770+0000 7f5e010a3640 1 -- 172.21.15.27:0/61893180 wait complete. 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.771+0000 7f5e010a3640 1 Processor -- start 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.771+0000 7f5e010a3640 1 -- start start 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.771+0000 7f5e010a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc101760 0x7f5dfc1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.771+0000 7f5e010a3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dfc102120 0x7f5dfc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5e010a3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dfc107770 0x7f5dfc19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5e010a3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc111b80 con 0x7f5dfc101760 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5e010a3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc111a00 con 0x7f5dfc102120 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5e010a3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc111d00 con 0x7f5dfc107770 2024-04-25T01:35:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc101760 0x7f5dfc1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc101760 0x7f5dfc1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44166/0 (socket says 172.21.15.27:44166) 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 -- 172.21.15.27:0/982945463 learned_addr learned my addr 172.21.15.27:0/982945463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dfc107770 0x7f5dfc19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dfc102120 0x7f5dfc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 -- 172.21.15.27:0/982945463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dfc107770 msgr2=0x7f5dfc19c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dfc107770 0x7f5dfc19c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 -- 172.21.15.27:0/982945463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dfc102120 msgr2=0x7f5dfc198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dfc102120 0x7f5dfc198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 -- 172.21.15.27:0/982945463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc19d060 con 0x7f5dfc101760 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc101760 0x7f5dfc1980c0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00d950 tx=0x7f5dec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5de3fff640 1 -- 172.21.15.27:0/982945463 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec017070 con 0x7f5dfc101760 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5de3fff640 1 -- 172.21.15.27:0/982945463 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dec009c30 con 0x7f5dfc101760 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5e010a3640 1 -- 172.21.15.27:0/982945463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dfc19d350 con 0x7f5dfc101760 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.772+0000 7f5de3fff640 1 -- 172.21.15.27:0/982945463 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec00f6b0 con 0x7f5dfc101760 2024-04-25T01:35:46.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.773+0000 7f5e010a3640 1 -- 172.21.15.27:0/982945463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dfc1a4be0 con 0x7f5dfc101760 2024-04-25T01:35:46.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5de3fff640 1 -- 172.21.15.27:0/982945463 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5dec0109d0 con 0x7f5dfc101760 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5e03da8740 1 Processor -- start 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5e03da8740 1 -- start start 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c20cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c21710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c2e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5e03da8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5566f7b852c0 con 0x5566f7c21290 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5e03da8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5566f7b2d910 con 0x5566f7c2c010 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5e03da8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5566f7b81280 con 0x5566f7c22980 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c21710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c21710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44172/0 (socket says 172.21.15.27:44172) 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.775+0000 7f5dfad9d640 1 -- 172.21.15.27:0/1576511694 learned_addr learned my addr 172.21.15.27:0/1576511694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c2e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5de3fff640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5de40a2d80 0x7f5de40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c20cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfad9d640 1 -- 172.21.15.27:0/1576511694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 msgr2=0x5566f7c20cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c20cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfad9d640 1 -- 172.21.15.27:0/1576511694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 msgr2=0x5566f7c2e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c2e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5de40a2d80 0x7f5de40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfad9d640 1 -- 172.21.15.27:0/1576511694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566f7b3efb0 con 0x5566f7c21290 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5de3fff640 1 -- 172.21.15.27:0/982945463 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f5dec0cf2c0 con 0x7f5dfc101760 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c21710 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0cc590 tx=0x7f5dec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55460c547e902bfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5de17fa640 1 -- 172.21.15.27:0/1576511694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec020070 con 0x5566f7c21290 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5de17fa640 1 -- 172.21.15.27:0/1576511694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dec0046c0 con 0x5566f7c21290 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5de17fa640 1 -- 172.21.15.27:0/1576511694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec009de0 con 0x5566f7c21290 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.776+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5de40a2d80 0x7f5de40a5240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5df4004ab0 tx=0x7f5df402a8e0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 -- 172.21.15.27:0/1576511694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 msgr2=0x5566f7c21710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c21710 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0cc590 tx=0x7f5dec005880 comp rx=0 tx=0).stop 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 -- 172.21.15.27:0/1576511694 shutdown_connections 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c2e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c21710 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 --2- 172.21.15.27:0/1576511694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c20cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 -- 172.21.15.27:0/1576511694 >> 172.21.15.27:0/1576511694 conn(0x5566f7be9950 msgr2=0x5566f7c1d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 -- 172.21.15.27:0/1576511694 shutdown_connections 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 -- 172.21.15.27:0/1576511694 wait complete. 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 -- start start 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.777+0000 7f5e03da8740 1 Processor -- start 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 0x5566f7c28eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c293f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c29930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5e03da8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5566f7c2ee40 con 0x5566f7c22980 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5e03da8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5566f7b40e90 con 0x5566f7c2c010 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5e03da8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5566f7c2efc0 con 0x5566f7c21290 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5e03da8740 10 client.0 _ll_register_callbacks cb 0x5566f7b7c970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c293f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c293f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44182/0 (socket says 172.21.15.27:44182) 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 -- 172.21.15.27:0/381069018 learned_addr learned my addr 172.21.15.27:0/381069018 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 0x5566f7c28eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c29930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 -- 172.21.15.27:0/381069018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 msgr2=0x5566f7c28eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 0x5566f7c28eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 -- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 msgr2=0x5566f7c29930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c29930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 -- 172.21.15.27:0/381069018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566f79617e0 con 0x5566f7c22980 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c29930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:46.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c293f0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5df000bb00 tx=0x7f5df0004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252fe827c3a5fb5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5ddeff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5ddeff5640 1 -- 172.21.15.27:0/381069018 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df0011440 con 0x5566f7c22980 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5ddeff5640 1 -- 172.21.15.27:0/381069018 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f5df00115e0 con 0x5566f7c22980 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.778+0000 7f5e03da8740 1 -- 172.21.15.27:0/381069018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dec0046c0 con 0x5566f7c22980 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5ddeff5640 1 -- 172.21.15.27:0/381069018 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5df0005020 con 0x5566f7c22980 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5e03da8740 10 client.17889 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5e03da8740 1 -- 172.21.15.27:0/381069018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5566f79c1f70 con 0x5566f7c22980 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5ddeff5640 1 -- 172.21.15.27:0/381069018 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dec0046c0 con 0x5566f7c22980 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5ddeff5640 1 -- 172.21.15.27:0/381069018 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5df00047a0 con 0x5566f7c22980 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5e03da8740 10 client.17889 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5e03da8740 10 client.17889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5e03da8740 20 client.17889 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5e03da8740 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566f7c3c100 0x5566f7c3e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5e03da8740 4 client.17889 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5e03da8740 1 -- 172.21.15.27:0/381069018 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566f7b595a0 con 0x5566f7c3c100 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.779+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566f7c3c100 0x5566f7c3e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.780+0000 7f5ddeff5640 10 client.17889 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.780+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566f7c3c100 0x5566f7c3e580 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97ee304b4d92ffb6 server_cookie=14f787c8ba0cc7c in_seq=0 out_seq=0 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.780+0000 7f5ddeff5640 1 -- 172.21.15.27:0/381069018 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566f7b595a0 con 0x5566f7c3c100 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.780+0000 7f5ddeff5640 10 client.17889 handle_command_reply: tid=0 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.780+0000 7f5e03da8740 1 client.17889 shutdown 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 -- 172.21.15.27:0/381069018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 msgr2=0x5566f7c293f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c293f0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5df000bb00 tx=0x7f5df0004580 comp rx=0 tx=0).stop 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 -- 172.21.15.27:0/381069018 shutdown_connections 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566f7c3c100 0x5566f7c3e580 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c29930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c293f0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5ddeff5640 0 client.17889 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:46.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 --2- 172.21.15.27:0/381069018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 0x5566f7c28eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 -- 172.21.15.27:0/381069018 >> 172.21.15.27:0/381069018 conn(0x5566f7be9950 msgr2=0x5566f7c235f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 -- 172.21.15.27:0/381069018 shutdown_connections 2024-04-25T01:35:46.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 1 -- 172.21.15.27:0/381069018 wait complete. 2024-04-25T01:35:46.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.781+0000 7f5e03da8740 20 client.17889 trim_cache size 0 max 16384 2024-04-25T01:35:46.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:46 smithi153 ceph-mon[32621]: pgmap v321: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:35:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.798+0000 7f5e03da8740 1 Processor -- start 2024-04-25T01:35:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.798+0000 7f5e03da8740 1 -- start start 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 0x5566f7c37d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c1c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c43e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5e03da8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1b1740 con 0x5566f7c22980 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5e03da8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1af010 con 0x5566f7c2c010 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5e03da8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1ace80 con 0x5566f7c21290 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c1c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c1c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44198/0 (socket says 172.21.15.27:44198) 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 -- 172.21.15.27:0/743906173 learned_addr learned my addr 172.21.15.27:0/743906173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 0x5566f7c37d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c43e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 -- 172.21.15.27:0/743906173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 msgr2=0x5566f7c37d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 0x5566f7c37d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 -- 172.21.15.27:0/743906173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 msgr2=0x5566f7c43e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c43e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 -- 172.21.15.27:0/743906173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566f7c27ea0 con 0x5566f7c22980 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c43e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c1c7d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f5df0004240 tx=0x7f5df0005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3da6a09418fc6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5de0ff9640 1 -- 172.21.15.27:0/743906173 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df002d070 con 0x5566f7c22980 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5de0ff9640 1 -- 172.21.15.27:0/743906173 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5df00051c0 con 0x5566f7c22980 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.799+0000 7f5de0ff9640 1 -- 172.21.15.27:0/743906173 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df00107c0 con 0x5566f7c22980 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 -- 172.21.15.27:0/743906173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 msgr2=0x5566f7c1c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c1c7d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f5df0004240 tx=0x7f5df0005870 comp rx=0 tx=0).stop 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 -- 172.21.15.27:0/743906173 shutdown_connections 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c43e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c22980 0x5566f7c1c7d0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 --2- 172.21.15.27:0/743906173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c21290 0x5566f7c37d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 -- 172.21.15.27:0/743906173 >> 172.21.15.27:0/743906173 conn(0x5566f7be9950 msgr2=0x5566f7c1c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 -- 172.21.15.27:0/743906173 shutdown_connections 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 -- 172.21.15.27:0/743906173 wait complete. 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 -- start start 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.800+0000 7f5e03da8740 1 Processor -- start 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c34470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c25160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c256a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1ad4d0 con 0x5566f7c21290 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1ae940 con 0x5566f7c2c010 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc1adba0 con 0x5566f7c22980 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 10 client.0 _ll_register_callbacks cb 0x7f5df0004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c34470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c34470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44206/0 (socket says 172.21.15.27:44206) 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 -- 172.21.15.27:0/2193286226 learned_addr learned my addr 172.21.15.27:0/2193286226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c25160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c256a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 -- 172.21.15.27:0/2193286226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 msgr2=0x5566f7c25160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c25160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 -- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 msgr2=0x5566f7c256a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c256a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 -- 172.21.15.27:0/2193286226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566f7859170 con 0x5566f7c21290 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c25160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfb59e640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c256a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5dfad9d640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c34470 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0ccc20 tx=0x7f5dec005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a229977128874d31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5ddeff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5ddeff5640 1 -- 172.21.15.27:0/2193286226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec020040 con 0x5566f7c21290 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5ddeff5640 1 -- 172.21.15.27:0/2193286226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f5dec097b10 con 0x5566f7c21290 2024-04-25T01:35:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 1 -- 172.21.15.27:0/2193286226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5df00051c0 con 0x5566f7c21290 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5ddeff5640 1 -- 172.21.15.27:0/2193286226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5dec009de0 con 0x5566f7c21290 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.801+0000 7f5e03da8740 10 client.17895 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5e03da8740 1 -- 172.21.15.27:0/2193286226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5566f7c26d70 con 0x5566f7c21290 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5ddeff5640 1 -- 172.21.15.27:0/2193286226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5df00051c0 con 0x5566f7c21290 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5ddeff5640 1 -- 172.21.15.27:0/2193286226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5dec0971f0 con 0x5566f7c21290 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5e03da8740 10 client.17895 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5e03da8740 10 client.17895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5e03da8740 20 client.17895 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5e03da8740 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566f7ca3720 0x5566f7ca5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5e03da8740 4 client.17895 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5e03da8740 1 -- 172.21.15.27:0/2193286226 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5566f7b92800 con 0x5566f7ca3720 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566f7ca3720 0x5566f7ca5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.802+0000 7f5dfa59c640 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566f7ca3720 0x5566f7ca5b10 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f2305988bfd5b3b server_cookie=3a999b5d8ae945f in_seq=0 out_seq=0 2024-04-25T01:35:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.803+0000 7f5ddeff5640 10 client.17895 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.803+0000 7f5ddeff5640 1 -- 172.21.15.27:0/2193286226 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5566f7b92800 con 0x5566f7ca3720 2024-04-25T01:35:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.803+0000 7f5ddeff5640 10 client.17895 handle_command_reply: tid=0 2024-04-25T01:35:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 client.17895 shutdown 2024-04-25T01:35:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 -- 172.21.15.27:0/2193286226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 msgr2=0x5566f7c34470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c34470 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0ccc20 tx=0x7f5dec005bd0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 -- 172.21.15.27:0/2193286226 shutdown_connections 2024-04-25T01:35:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5566f7ca3720 0x5566f7ca5b10 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5ddeff5640 0 client.17895 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5566f7c2c010 0x5566f7c256a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5566f7c22980 0x5566f7c25160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 --2- 172.21.15.27:0/2193286226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5566f7c21290 0x5566f7c34470 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 -- 172.21.15.27:0/2193286226 >> 172.21.15.27:0/2193286226 conn(0x5566f7be9950 msgr2=0x5566f7c1c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 -- 172.21.15.27:0/2193286226 shutdown_connections 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 1 -- 172.21.15.27:0/2193286226 wait complete. 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e03da8740 20 client.17895 trim_cache size 0 max 16384 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:35:46.787 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "84652ec1-1d28-45fc-9cbc-789154f780d3", 2024-04-25T01:35:46.788 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:35:46.788 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:35:46.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e010a3640 1 -- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5de40a2d80 msgr2=0x7f5de40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e010a3640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5de40a2d80 0x7f5de40a5240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5df4004ab0 tx=0x7f5df402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e010a3640 1 -- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc101760 msgr2=0x7f5dfc1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.804+0000 7f5e010a3640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc101760 0x7f5dfc1980c0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00d950 tx=0x7f5dec00de10 comp rx=0 tx=0).stop 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.805+0000 7f5e010a3640 1 -- 172.21.15.27:0/982945463 shutdown_connections 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.805+0000 7f5e010a3640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dfc107770 0x7f5dfc19c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.805+0000 7f5e010a3640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5dfc102120 0x7f5dfc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.805+0000 7f5e010a3640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5de40a2d80 0x7f5de40a5240 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.805+0000 7f5e010a3640 1 --2- 172.21.15.27:0/982945463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dfc101760 0x7f5dfc1980c0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.805+0000 7f5e010a3640 1 -- 172.21.15.27:0/982945463 >> 172.21.15.27:0/982945463 conn(0x7f5dfc0fd250 msgr2=0x7f5dfc0fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.805+0000 7f5e010a3640 1 -- 172.21.15.27:0/982945463 shutdown_connections 2024-04-25T01:35:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.805+0000 7f5e010a3640 1 -- 172.21.15.27:0/982945463 wait complete. 2024-04-25T01:35:46.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:46 smithi139 ceph-mon[32760]: pgmap v321: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:35:46.791 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:46 smithi027 ceph-mon[29197]: pgmap v321: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:35:46.792 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:35:46.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.919+0000 7fb625bc1640 1 -- 172.21.15.27:0/2270659490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb62006a780 msgr2=0x7fb62010e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 --2- 172.21.15.27:0/2270659490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb62006a780 0x7fb62010e5f0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb618009cd0 tx=0x7fb61802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 -- 172.21.15.27:0/2270659490 shutdown_connections 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 --2- 172.21.15.27:0/2270659490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb62006a780 0x7fb62010e5f0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 --2- 172.21.15.27:0/2270659490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb620069dc0 0x7fb62006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 --2- 172.21.15.27:0/2270659490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb620107730 0x7fb620107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 -- 172.21.15.27:0/2270659490 >> 172.21.15.27:0/2270659490 conn(0x7fb6200776a0 msgr2=0x7fb620077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 -- 172.21.15.27:0/2270659490 shutdown_connections 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 -- 172.21.15.27:0/2270659490 wait complete. 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 Processor -- start 2024-04-25T01:35:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.920+0000 7fb625bc1640 1 -- start start 2024-04-25T01:35:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb625bc1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb620069dc0 0x7fb620197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb625bc1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62006a780 0x7fb620198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb625bc1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb620107730 0x7fb62019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb625bc1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb620111b90 con 0x7fb620107730 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb625bc1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb620111a10 con 0x7fb620069dc0 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb625bc1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb620111d10 con 0x7fb62006a780 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb620107730 0x7fb62019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb620107730 0x7fb62019c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44228/0 (socket says 172.21.15.27:44228) 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 -- 172.21.15.27:0/6502364 learned_addr learned my addr 172.21.15.27:0/6502364 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61effd640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62006a780 0x7fb620198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb620069dc0 0x7fb620197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 -- 172.21.15.27:0/6502364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62006a780 msgr2=0x7fb620198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62006a780 0x7fb620198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 -- 172.21.15.27:0/6502364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb620069dc0 msgr2=0x7fb620197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb620069dc0 0x7fb620197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 -- 172.21.15.27:0/6502364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62019cee0 con 0x7fb620107730 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61effd640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62006a780 0x7fb620198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61ffff640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb620107730 0x7fb62019c7b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fb618009ca0 tx=0x7fb618003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61cff9640 1 -- 172.21.15.27:0/6502364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618004240 con 0x7fb620107730 2024-04-25T01:35:46.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61cff9640 1 -- 172.21.15.27:0/6502364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6180305a0 con 0x7fb620107730 2024-04-25T01:35:46.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb625bc1640 1 -- 172.21.15.27:0/6502364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62019d170 con 0x7fb620107730 2024-04-25T01:35:46.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb61cff9640 1 -- 172.21.15.27:0/6502364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61803a660 con 0x7fb620107730 2024-04-25T01:35:46.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.921+0000 7fb625bc1640 1 -- 172.21.15.27:0/6502364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6201a4a60 con 0x7fb620107730 2024-04-25T01:35:46.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb61cff9640 1 -- 172.21.15.27:0/6502364 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb618030750 con 0x7fb620107730 2024-04-25T01:35:46.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb6289aa740 1 Processor -- start 2024-04-25T01:35:46.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb6289aa740 1 -- start start 2024-04-25T01:35:46.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb7ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb890e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 0x556a8bb8ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb6289aa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556a8bae36b0 con 0x556a8bb7e630 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb6289aa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556a8ba8b880 con 0x556a8bb89620 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb6289aa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556a8badf670 con 0x556a8bb7ef50 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb61effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb7ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb61effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb7ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44242/0 (socket says 172.21.15.27:44242) 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb61effd640 1 -- 172.21.15.27:0/3727834339 learned_addr learned my addr 172.21.15.27:0/3727834339 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb61cff9640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb6080a2d00 0x7fb6080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb890e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb61effd640 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 0x556a8bb8ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.924+0000 7fb61ffff640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb6080a2d00 0x7fb6080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb61effd640 1 -- 172.21.15.27:0/3727834339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 msgr2=0x556a8bb890e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb61effd640 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb890e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb61effd640 1 -- 172.21.15.27:0/3727834339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 msgr2=0x556a8bb8ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb61effd640 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 0x556a8bb8ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb61effd640 1 -- 172.21.15.27:0/3727834339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a8b8c17d0 con 0x556a8bb7e630 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb61cff9640 1 -- 172.21.15.27:0/6502364 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fb61802e030 con 0x7fb620107730 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb61effd640 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb7ea10 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb614008fc0 tx=0x7fb614008640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eea04c8901c253 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb605ffb640 1 -- 172.21.15.27:0/3727834339 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614018030 con 0x556a8bb7e630 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb61ffff640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb6080a2d00 0x7fb6080a51c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb6180edf20 tx=0x7fb618087c60 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb605ffb640 1 -- 172.21.15.27:0/3727834339 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb614010b20 con 0x556a8bb7e630 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.925+0000 7fb605ffb640 1 -- 172.21.15.27:0/3727834339 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61401ca90 con 0x556a8bb7e630 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- 172.21.15.27:0/3727834339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 msgr2=0x556a8bb7ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb7ea10 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb614008fc0 tx=0x7fb614008640 comp rx=0 tx=0).stop 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- 172.21.15.27:0/3727834339 shutdown_connections 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 0x556a8bb8ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb890e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3727834339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb7ea10 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- 172.21.15.27:0/3727834339 >> 172.21.15.27:0/3727834339 conn(0x556a8bb7a950 msgr2=0x556a8bb7ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- 172.21.15.27:0/3727834339 shutdown_connections 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- 172.21.15.27:0/3727834339 wait complete. 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- start start 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 Processor -- start 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7e630 0x556a8bb86570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7ef50 0x556a8bb86ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb86ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556a8bb8c410 con 0x556a8bb89620 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x556a8ba9ef50 con 0x556a8bb7ef50 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.926+0000 7fb6289aa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556a8bb8c590 con 0x556a8bb7e630 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb6289aa740 10 client.0 _ll_register_callbacks cb 0x556a8bada000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb86ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb86ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44256/0 (socket says 172.21.15.27:44256) 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 -- 172.21.15.27:0/3217242267 learned_addr learned my addr 172.21.15.27:0/3217242267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7e630 0x556a8bb86570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7ef50 0x556a8bb86ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 -- 172.21.15.27:0/3217242267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7e630 msgr2=0x556a8bb86570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7e630 0x556a8bb86570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 -- 172.21.15.27:0/3217242267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7ef50 msgr2=0x556a8bb86ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7ef50 0x556a8bb86ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 -- 172.21.15.27:0/3217242267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a8b7f73f0 con 0x556a8bb89620 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb61effd640 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb86ff0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fb614008610 tx=0x7fb614008c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7379612e7dfe39e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb5feffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb5feffd640 1 -- 172.21.15.27:0/3217242267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614021020 con 0x556a8bb89620 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb5feffd640 1 -- 172.21.15.27:0/3217242267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fb61401cc60 con 0x556a8bb89620 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.927+0000 7fb6289aa740 1 -- 172.21.15.27:0/3217242267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb614010b20 con 0x556a8bb89620 2024-04-25T01:35:46.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb5feffd640 1 -- 172.21.15.27:0/3217242267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb614031050 con 0x556a8bb89620 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb6289aa740 10 client.17907 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb6289aa740 1 -- 172.21.15.27:0/3217242267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556a8b7b70f0 con 0x556a8bb89620 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb5feffd640 1 -- 172.21.15.27:0/3217242267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb614010b20 con 0x556a8bb89620 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb5feffd640 1 -- 172.21.15.27:0/3217242267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb614004a60 con 0x556a8bb89620 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb6289aa740 10 client.17907 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb6289aa740 10 client.17907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb6289aa740 20 client.17907 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556a8bb99890 0x556a8bb9bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb6289aa740 4 client.17907 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb6289aa740 1 -- 172.21.15.27:0/3217242267 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a8baf0bf0 con 0x556a8bb99890 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.928+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556a8bb99890 0x556a8bb9bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.929+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556a8bb99890 0x556a8bb9bc80 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e89b59dabc075c7 server_cookie=2f16c179c195cd38 in_seq=0 out_seq=0 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.929+0000 7fb5feffd640 10 client.17907 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.929+0000 7fb5feffd640 1 -- 172.21.15.27:0/3217242267 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a8baf0bf0 con 0x556a8bb99890 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.929+0000 7fb5feffd640 10 client.17907 handle_command_reply: tid=0 2024-04-25T01:35:46.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.929+0000 7fb6289aa740 1 client.17907 shutdown 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 -- 172.21.15.27:0/3217242267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 msgr2=0x556a8bb86ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb86ff0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fb614008610 tx=0x7fb614008c10 comp rx=0 tx=0).stop 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 -- 172.21.15.27:0/3217242267 shutdown_connections 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556a8bb99890 0x556a8bb9bc80 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb86ff0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7ef50 0x556a8bb86ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb5feffd640 0 client.17907 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3217242267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7e630 0x556a8bb86570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 -- 172.21.15.27:0/3217242267 >> 172.21.15.27:0/3217242267 conn(0x556a8bb7a950 msgr2=0x556a8bb80650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 -- 172.21.15.27:0/3217242267 shutdown_connections 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 1 -- 172.21.15.27:0/3217242267 wait complete. 2024-04-25T01:35:46.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.930+0000 7fb6289aa740 20 client.17907 trim_cache size 0 max 16384 2024-04-25T01:35:46.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.947+0000 7fb6289aa740 1 Processor -- start 2024-04-25T01:35:46.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.947+0000 7fb6289aa740 1 -- start start 2024-04-25T01:35:46.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.947+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7e630 0x556a8bb88ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb95300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb81cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb6289aa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201b1480 con 0x556a8bb89620 2024-04-25T01:35:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb6289aa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201aed50 con 0x556a8bb7e630 2024-04-25T01:35:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb6289aa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201acbc0 con 0x556a8bb7ef50 2024-04-25T01:35:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb81cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb81cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44260/0 (socket says 172.21.15.27:44260) 2024-04-25T01:35:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 -- 172.21.15.27:0/2630245440 learned_addr learned my addr 172.21.15.27:0/2630245440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7e630 0x556a8bb88ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb95300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 -- 172.21.15.27:0/2630245440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 msgr2=0x556a8bb95300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb95300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 -- 172.21.15.27:0/2630245440 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7e630 msgr2=0x556a8bb88ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7e630 0x556a8bb88ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 -- 172.21.15.27:0/2630245440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a8bb6c090 con 0x556a8bb89620 2024-04-25T01:35:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb61effd640 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb81cc0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fb61401c820 tx=0x7fb614014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f661ea62d342387b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb6057fa640 1 -- 172.21.15.27:0/2630245440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614021040 con 0x556a8bb89620 2024-04-25T01:35:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb6057fa640 1 -- 172.21.15.27:0/2630245440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb614019040 con 0x556a8bb89620 2024-04-25T01:35:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.948+0000 7fb6057fa640 1 -- 172.21.15.27:0/2630245440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614018070 con 0x556a8bb89620 2024-04-25T01:35:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 -- 172.21.15.27:0/2630245440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 msgr2=0x556a8bb81cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb81cc0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fb61401c820 tx=0x7fb614014040 comp rx=0 tx=0).stop 2024-04-25T01:35:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 -- 172.21.15.27:0/2630245440 shutdown_connections 2024-04-25T01:35:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb89620 0x556a8bb81cc0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb95300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 --2- 172.21.15.27:0/2630245440 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb7e630 0x556a8bb88ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 -- 172.21.15.27:0/2630245440 >> 172.21.15.27:0/2630245440 conn(0x556a8bb7a950 msgr2=0x556a8bb7fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 -- 172.21.15.27:0/2630245440 shutdown_connections 2024-04-25T01:35:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 -- 172.21.15.27:0/2630245440 wait complete. 2024-04-25T01:35:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 -- start start 2024-04-25T01:35:46.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.949+0000 7fb6289aa740 1 Processor -- start 2024-04-25T01:35:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb91830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb91d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb6289aa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 0x556a8bb8cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb6289aa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201ad210 con 0x556a8bb7e630 2024-04-25T01:35:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb6289aa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201ae680 con 0x556a8bb89620 2024-04-25T01:35:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb6289aa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201ad8e0 con 0x556a8bb7ef50 2024-04-25T01:35:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb6289aa740 10 client.0 _ll_register_callbacks cb 0x7fb6140123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb91830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb91830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44274/0 (socket says 172.21.15.27:44274) 2024-04-25T01:35:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 -- 172.21.15.27:0/3470500160 learned_addr learned my addr 172.21.15.27:0/3470500160 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:35:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 -- 172.21.15.27:0/3470500160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 msgr2=0x556a8bb91d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb91d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 -- 172.21.15.27:0/3470500160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 msgr2=0x556a8bb8cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 0x556a8bb8cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 -- 172.21.15.27:0/3470500160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a8bb8d570 con 0x556a8bb7e630 2024-04-25T01:35:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb61effd640 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb91830 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fb614001eb0 tx=0x7fb6140122d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1821681711ec360d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb5feffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:35:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb5feffd640 1 -- 172.21.15.27:0/3470500160 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614043060 con 0x556a8bb7e630 2024-04-25T01:35:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb5feffd640 1 -- 172.21.15.27:0/3470500160 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fb614008e10 con 0x556a8bb7e630 2024-04-25T01:35:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.950+0000 7fb6289aa740 1 -- 172.21.15.27:0/3470500160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb614019040 con 0x556a8bb7e630 2024-04-25T01:35:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb5feffd640 1 -- 172.21.15.27:0/3470500160 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb614018070 con 0x556a8bb7e630 2024-04-25T01:35:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb6289aa740 10 client.17913 fetch_fsmap learned FSMap version 27 2024-04-25T01:35:46.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb6289aa740 1 -- 172.21.15.27:0/3470500160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556a8bbff8f0 con 0x556a8bb7e630 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb5feffd640 1 -- 172.21.15.27:0/3470500160 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb614019040 con 0x556a8bb7e630 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb5feffd640 1 -- 172.21.15.27:0/3470500160 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb6140035f0 con 0x556a8bb7e630 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb6289aa740 10 client.17913 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb6289aa740 10 client.17913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb6289aa740 20 client.17913 populate_metadata read hostname 'smithi027' 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556a8bbfff90 0x556a8bc023b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb6289aa740 4 client.17913 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb6289aa740 1 -- 172.21.15.27:0/3470500160 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556a8baeaf40 con 0x556a8bbfff90 2024-04-25T01:35:46.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.951+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556a8bbfff90 0x556a8bc023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb5feffd640 10 client.17913 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556a8bbfff90 0x556a8bc023b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba8df3099068e9cb server_cookie=d66e720b004c049f in_seq=0 out_seq=0 2024-04-25T01:35:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb5feffd640 1 -- 172.21.15.27:0/3470500160 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556a8baeaf40 con 0x556a8bbfff90 2024-04-25T01:35:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb5feffd640 10 client.17913 handle_command_reply: tid=0 2024-04-25T01:35:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 client.17913 shutdown 2024-04-25T01:35:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 -- 172.21.15.27:0/3470500160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 msgr2=0x556a8bb91830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb91830 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fb614001eb0 tx=0x7fb6140122d0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 -- 172.21.15.27:0/3470500160 shutdown_connections 2024-04-25T01:35:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x556a8bbfff90 0x556a8bc023b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x556a8bb89620 0x556a8bb8cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb5feffd640 0 client.17913 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:35:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556a8bb7ef50 0x556a8bb91d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 --2- 172.21.15.27:0/3470500160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a8bb7e630 0x556a8bb91830 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 -- 172.21.15.27:0/3470500160 >> 172.21.15.27:0/3470500160 conn(0x556a8bb7a950 msgr2=0x556a8bb85350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 -- 172.21.15.27:0/3470500160 shutdown_connections 2024-04-25T01:35:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 1 -- 172.21.15.27:0/3470500160 wait complete. 2024-04-25T01:35:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.952+0000 7fb6289aa740 20 client.17913 trim_cache size 0 max 16384 2024-04-25T01:35:46.948 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:35:46.948 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:35:46.948 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:35:46.948 INFO:teuthology.orchestra.run.smithi027.stdout: "84652ec1-1d28-45fc-9cbc-789154f780d3": { 2024-04-25T01:35:46.948 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:35:46.948 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "84652ec1-1d28-45fc-9cbc-789154f780d3", 2024-04-25T01:35:46.949 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:35:46.949 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:35:46.949 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:35:46.949 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:35:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 -- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb6080a2d00 msgr2=0x7fb6080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb6080a2d00 0x7fb6080a51c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb6180edf20 tx=0x7fb618087c60 comp rx=0 tx=0).stop 2024-04-25T01:35:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 -- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb620107730 msgr2=0x7fb62019c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb620107730 0x7fb62019c7b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fb618009ca0 tx=0x7fb618003ba0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 -- 172.21.15.27:0/6502364 shutdown_connections 2024-04-25T01:35:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fb6080a2d00 0x7fb6080a51c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb620107730 0x7fb62019c7b0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62006a780 0x7fb620198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 --2- 172.21.15.27:0/6502364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb620069dc0 0x7fb620197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 -- 172.21.15.27:0/6502364 >> 172.21.15.27:0/6502364 conn(0x7fb6200776a0 msgr2=0x7fb6200fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 -- 172.21.15.27:0/6502364 shutdown_connections 2024-04-25T01:35:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:35:46.953+0000 7fb625bc1640 1 -- 172.21.15.27:0/6502364 wait complete. 2024-04-25T01:35:46.951 INFO:tasks.cephfs.filesystem:scrub status for tag:84652ec1-1d28-45fc-9cbc-789154f780d3 - {'path': '/', 'tag': '84652ec1-1d28-45fc-9cbc-789154f780d3', 'options': 'recursive,force'} 2024-04-25T01:35:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:48 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:35:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:48 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:48 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:35:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:48 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:35:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:48 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:48 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:48 smithi153 ceph-mon[32621]: pgmap v322: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:35:48.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:48 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:35:48.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:48 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:48 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:35:48.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:48 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:35:48.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:48 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:48 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:48 smithi139 ceph-mon[32760]: pgmap v322: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:35:48.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:48 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:35:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:48 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:48 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:35:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:48 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:35:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:48 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:48 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:48.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:48 smithi027 ceph-mon[29197]: pgmap v322: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:35:50.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:50 smithi153 ceph-mon[32621]: pgmap v323: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:35:50.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:50 smithi139 ceph-mon[32760]: pgmap v323: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:35:50.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:50 smithi027 ceph-mon[29197]: pgmap v323: 129 pgs: 129 active+clean; 5.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:35:52.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:52 smithi027 ceph-mon[29197]: pgmap v324: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:35:52.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:52 smithi153 ceph-mon[32621]: pgmap v324: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:35:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:52 smithi139 ceph-mon[32760]: pgmap v324: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:35:53.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:53 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:35:53.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:53 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:35:53.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:53 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:35:53.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:53 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:35:53.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:53 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:35:53.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:53 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:35:54.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:54 smithi153 ceph-mon[32621]: pgmap v325: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T01:35:54.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:54 smithi139 ceph-mon[32760]: pgmap v325: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T01:35:54.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:54 smithi027 ceph-mon[29197]: pgmap v325: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T01:35:56.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:56 smithi153 ceph-mon[32621]: pgmap v326: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T01:35:56.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:56 smithi139 ceph-mon[32760]: pgmap v326: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T01:35:56.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:56 smithi027 ceph-mon[29197]: pgmap v326: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T01:35:58.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:35:58 smithi153 ceph-mon[32621]: pgmap v327: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-25T01:35:58.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:35:58 smithi139 ceph-mon[32760]: pgmap v327: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-25T01:35:58.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:35:58 smithi027 ceph-mon[29197]: pgmap v327: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-25T01:36:00.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:00 smithi153 ceph-mon[32621]: pgmap v328: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-25T01:36:00.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:00 smithi139 ceph-mon[32760]: pgmap v328: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-25T01:36:00.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:00 smithi027 ceph-mon[29197]: pgmap v328: 129 pgs: 129 active+clean; 5.1 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-25T01:36:02.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:02 smithi153 ceph-mon[32621]: pgmap v329: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 31 op/s 2024-04-25T01:36:02.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:02 smithi139 ceph-mon[32760]: pgmap v329: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 31 op/s 2024-04-25T01:36:02.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:02 smithi027 ceph-mon[29197]: pgmap v329: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 KiB/s wr, 31 op/s 2024-04-25T01:36:04.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:04 smithi153 ceph-mon[32621]: pgmap v330: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:36:04.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:04 smithi139 ceph-mon[32760]: pgmap v330: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:36:04.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:04 smithi027 ceph-mon[29197]: pgmap v330: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:36:06.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:06 smithi153 ceph-mon[32621]: pgmap v331: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:36:06.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:06 smithi139 ceph-mon[32760]: pgmap v331: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:36:06.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:06 smithi027 ceph-mon[29197]: pgmap v331: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-25T01:36:08.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:08 smithi153 ceph-mon[32621]: pgmap v332: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-25T01:36:08.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:08 smithi153 ceph-mon[32621]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T01:36:08.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:08 smithi139 ceph-mon[32760]: pgmap v332: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-25T01:36:08.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:08 smithi139 ceph-mon[32760]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T01:36:08.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:08 smithi027 ceph-mon[29197]: pgmap v332: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-25T01:36:08.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:08 smithi027 ceph-mon[29197]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T01:36:09.743 INFO:tasks.workunit.client.0.smithi027.stdout:499/600: open fd = 3 2024-04-25T01:36:09.743 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.743 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.743 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.743 INFO:tasks.workunit.client.0.smithi027.stdout:500/600: open fd = 3 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:501/600: open fd = 3 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:502/600: open fd = 3 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:503/600: open fd = 3 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.744 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:504/600: open fd = 3 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:505/600: open fd = 3 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:506/600: open fd = 3 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:507/600: open fd = 3 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.745 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:508/600: open fd = 3 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:509/600: open fd = 3 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:510/600: open fd = 3 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:511/600: open fd = 3 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.746 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:512/600: open fd = 3 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:513/600: open fd = 3 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:514/600: open fd = 3 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:515/600: open fd = 3 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.747 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:516/600: open fd = 3 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:517/600: open fd = 3 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:518/600: open fd = 3 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:519/600: open fd = 3 2024-04-25T01:36:09.748 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:520/600: open fd = 3 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:521/600: open fd = 3 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:522/600: open fd = 3 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:523/600: open fd = 3 2024-04-25T01:36:09.749 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:524/600: open fd = 3 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:525/600: open fd = 3 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:526/600: open fd = 3 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.750 INFO:tasks.workunit.client.0.smithi027.stdout:527/600: open fd = 3 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:528/600: open fd = 3 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:529/600: open fd = 3 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:530/600: open fd = 3 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.751 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:531/600: open fd = 3 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:532/600: open fd = 3 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:533/600: open fd = 3 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:534/600: open fd = 3 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.752 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:535/600: open fd = 3 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:536/600: open fd = 3 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:537/600: open fd = 3 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:538/600: open fd = 3 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.753 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:539/600: open fd = 3 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:540/600: open fd = 3 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:541/600: open fd = 3 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.754 INFO:tasks.workunit.client.0.smithi027.stdout:542/600: open fd = 3 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:543/600: open fd = 3 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:544/600: open fd = 3 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:545/600: open fd = 3 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.755 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:546/600: open fd = 3 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:547/600: open fd = 3 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:548/600: open fd = 3 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:549/600: open fd = 3 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.756 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:550/600: open fd = 3 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:551/600: open fd = 3 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:552/600: open fd = 3 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:553/600: open fd = 3 2024-04-25T01:36:09.757 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:554/600: open fd = 3 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:555/600: open fd = 3 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:556/600: open fd = 3 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.758 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:557/600: open fd = 3 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:558/600: open fd = 3 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:559/600: open fd = 3 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:09.759 INFO:tasks.workunit.client.0.smithi027.stdout:560/600: open fd = 3 2024-04-25T01:36:09.760 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.760 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:09.760 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:10.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:10 smithi153 ceph-mon[32621]: pgmap v334: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:36:10.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:10 smithi139 ceph-mon[32760]: pgmap v334: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:36:10.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:10 smithi027 ceph-mon[29197]: pgmap v334: 129 pgs: 129 active+clean; 5.2 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 64 KiB/s wr, 32 op/s 2024-04-25T01:36:12.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:12 smithi153 ceph-mon[32621]: pgmap v335: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:12.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:12 smithi139 ceph-mon[32760]: pgmap v335: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:12.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:12 smithi027 ceph-mon[29197]: pgmap v335: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:14.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:14 smithi139 ceph-mon[32760]: pgmap v336: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:14.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:14 smithi027 ceph-mon[29197]: pgmap v336: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:15.021 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:14 smithi153 ceph-mon[32621]: pgmap v336: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:16.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:16 smithi153 ceph-mon[32621]: pgmap v337: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:16.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:16 smithi153 ceph-mon[32621]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T01:36:16.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:16 smithi139 ceph-mon[32760]: pgmap v337: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:16.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:16 smithi139 ceph-mon[32760]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T01:36:16.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:16 smithi027 ceph-mon[29197]: pgmap v337: 129 pgs: 129 active+clean; 5.2 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:16.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:16 smithi027 ceph-mon[29197]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T01:36:16.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:36:16.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:36:17.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 -- 172.21.15.27:0/1852899826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec073ae0 msgr2=0x7ff8ec073ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1852899826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec073ae0 0x7ff8ec073ec0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009900 tx=0x7ff8e802a480 comp rx=0 tx=0).stop 2024-04-25T01:36:17.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 -- 172.21.15.27:0/1852899826 shutdown_connections 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1852899826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8ec112770 0x7ff8ec114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1852899826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8ec074400 0x7ff8ec112230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1852899826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec073ae0 0x7ff8ec073ec0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 -- 172.21.15.27:0/1852899826 >> 172.21.15.27:0/1852899826 conn(0x7ff8ec0fd250 msgr2=0x7ff8ec0ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 -- 172.21.15.27:0/1852899826 shutdown_connections 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.092+0000 7ff8f375c640 1 -- 172.21.15.27:0/1852899826 wait complete. 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f375c640 1 Processor -- start 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f375c640 1 -- start start 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f375c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8ec073ae0 0x7ff8ec19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f375c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec074400 0x7ff8ec19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f375c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8ec112770 0x7ff8ec1a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f375c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec118140 con 0x7ff8ec074400 2024-04-25T01:36:17.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f375c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec117fc0 con 0x7ff8ec112770 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f375c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1182c0 con 0x7ff8ec073ae0 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f0cf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec074400 0x7ff8ec19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f0cf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec074400 0x7ff8ec19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53998/0 (socket says 172.21.15.27:53998) 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/1966644961 learned_addr learned my addr 172.21.15.27:0/1966644961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8ec073ae0 0x7ff8ec19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.093+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8ec112770 0x7ff8ec1a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/1966644961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8ec073ae0 msgr2=0x7ff8ec19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8ec073ae0 0x7ff8ec19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/1966644961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8ec112770 msgr2=0x7ff8ec1a2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8ec112770 0x7ff8ec1a2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/1966644961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ec1a3650 con 0x7ff8ec074400 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8ec112770 0x7ff8ec1a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec074400 0x7ff8ec19eba0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00c890 tx=0x7ff8dc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8e27fc640 1 -- 172.21.15.27:0/1966644961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc004220 con 0x7ff8ec074400 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8e27fc640 1 -- 172.21.15.27:0/1966644961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8dc0043c0 con 0x7ff8ec074400 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f375c640 1 -- 172.21.15.27:0/1966644961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ec1a3940 con 0x7ff8ec074400 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8e27fc640 1 -- 172.21.15.27:0/1966644961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0049e0 con 0x7ff8ec074400 2024-04-25T01:36:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.094+0000 7ff8f375c640 1 -- 172.21.15.27:0/1966644961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ec1ab1d0 con 0x7ff8ec074400 2024-04-25T01:36:17.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.096+0000 7ff8e27fc640 1 -- 172.21.15.27:0/1966644961 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff8dc0079b0 con 0x7ff8ec074400 2024-04-25T01:36:17.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f645f740 1 Processor -- start 2024-04-25T01:36:17.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f645f740 1 -- start start 2024-04-25T01:36:17.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc110a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc11b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc11da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f645f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a6cc0756b0 con 0x55a6cc11b620 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f645f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a6cc01d880 con 0x55a6cc110f50 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f645f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a6cc071670 con 0x55a6cc110630 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f0cf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc11da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8e27fc640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8d80a2d00 0x7ff8d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f0cf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc11da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54012/0 (socket says 172.21.15.27:54012) 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.097+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/294457163 learned_addr learned my addr 172.21.15.27:0/294457163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8d80a2d00 0x7ff8d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc11b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc110a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8e27fc640 1 -- 172.21.15.27:0/1966644961 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7ff8dc0cf8a0 con 0x7ff8ec074400 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/294457163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 msgr2=0x55a6cc110a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc110a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/294457163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 msgr2=0x55a6cc11b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc11b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/294457163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6cbe537d0 con 0x55a6cc11b620 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc11da10 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc084f10 tx=0x7ff8dc005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee403052374bb439 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8d80a2d00 0x7ff8d80a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4007310 tx=0x7ff8e4007d10 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8d77fe640 1 -- 172.21.15.27:0/294457163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc01d050 con 0x55a6cc11b620 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8d77fe640 1 -- 172.21.15.27:0/294457163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8dc09fd80 con 0x55a6cc11b620 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.098+0000 7ff8d77fe640 1 -- 172.21.15.27:0/294457163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc007ca0 con 0x55a6cc11b620 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 -- 172.21.15.27:0/294457163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 msgr2=0x55a6cc11da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc11da10 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc084f10 tx=0x7ff8dc005300 comp rx=0 tx=0).stop 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 -- 172.21.15.27:0/294457163 shutdown_connections 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc11da10 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc11b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 --2- 172.21.15.27:0/294457163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc110a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 -- 172.21.15.27:0/294457163 >> 172.21.15.27:0/294457163 conn(0x55a6cc10c950 msgr2=0x55a6cc10cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 -- 172.21.15.27:0/294457163 shutdown_connections 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 -- 172.21.15.27:0/294457163 wait complete. 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 -- start start 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.099+0000 7ff8f645f740 1 Processor -- start 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110630 0x55a6cc1184c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc118a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc118f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f645f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a6cc11e3b0 con 0x55a6cc110630 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f645f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55a6cc030f50 con 0x55a6cc110f50 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f645f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a6cc11e530 con 0x55a6cc11b620 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f645f740 10 client.0 _ll_register_callbacks cb 0x55a6cc06c000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f14f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110630 0x55a6cc1184c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f14f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110630 0x55a6cc1184c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54026/0 (socket says 172.21.15.27:54026) 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f14f8640 1 -- 172.21.15.27:0/546393835 learned_addr learned my addr 172.21.15.27:0/546393835 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc118a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc118f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f14f8640 1 -- 172.21.15.27:0/546393835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 msgr2=0x55a6cc118f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc118f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f14f8640 1 -- 172.21.15.27:0/546393835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 msgr2=0x55a6cc118a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc118a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.100+0000 7ff8f14f8640 1 -- 172.21.15.27:0/546393835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6cbd893f0 con 0x55a6cc110630 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110630 0x55a6cc1184c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009900 tx=0x7ff8e8031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f17f5dd7fc4148 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/546393835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8035040 con 0x55a6cc110630 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/546393835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7ff8e8004380 con 0x55a6cc110630 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 1 -- 172.21.15.27:0/546393835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a6cc082bf0 con 0x55a6cc110630 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/546393835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff8e804f050 con 0x55a6cc110630 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 10 client.17925 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 1 -- 172.21.15.27:0/546393835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6cbd490f0 con 0x55a6cc110630 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/546393835 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a6cc082bf0 con 0x55a6cc110630 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/546393835 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff8e80467e0 con 0x55a6cc110630 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 10 client.17925 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 10 client.17925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 20 client.17925 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6cc12b7e0 0x55a6cc12dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 4 client.17925 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.101+0000 7ff8f645f740 1 -- 172.21.15.27:0/546393835 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6cc07cf40 con 0x55a6cc12b7e0 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.102+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6cc12b7e0 0x55a6cc12dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.102+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6cc12b7e0 0x55a6cc12dc70 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b388142607a80a54 server_cookie=63b7f890f761bb72 in_seq=0 out_seq=0 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.102+0000 7ff8d4ff9640 10 client.17925 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/546393835 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6cc07cf40 con 0x55a6cc12b7e0 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8d4ff9640 10 client.17925 handle_command_reply: tid=0 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 client.17925 shutdown 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 -- 172.21.15.27:0/546393835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110630 msgr2=0x55a6cc1184c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110630 0x55a6cc1184c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009900 tx=0x7ff8e8031a60 comp rx=0 tx=0).stop 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 -- 172.21.15.27:0/546393835 shutdown_connections 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6cc12b7e0 0x55a6cc12dc70 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc118f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc118a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8d4ff9640 0 client.17925 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 --2- 172.21.15.27:0/546393835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110630 0x55a6cc1184c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 -- 172.21.15.27:0/546393835 >> 172.21.15.27:0/546393835 conn(0x55a6cc10c950 msgr2=0x55a6cc10baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 -- 172.21.15.27:0/546393835 shutdown_connections 2024-04-25T01:36:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 1 -- 172.21.15.27:0/546393835 wait complete. 2024-04-25T01:36:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.103+0000 7ff8f645f740 20 client.17925 trim_cache size 0 max 16384 2024-04-25T01:36:17.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f645f740 1 Processor -- start 2024-04-25T01:36:17.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f645f740 1 -- start start 2024-04-25T01:36:17.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc11abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc1272a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc117660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f645f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1b7ba0 con 0x55a6cc11b620 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f645f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1b5470 con 0x55a6cc110f50 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f645f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1b32e0 con 0x55a6cc110630 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f1cf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc117660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f1cf9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc117660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54030/0 (socket says 172.21.15.27:54030) 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f1cf9640 1 -- 172.21.15.27:0/3335577470 learned_addr learned my addr 172.21.15.27:0/3335577470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc11abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc1272a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f1cf9640 1 -- 172.21.15.27:0/3335577470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 msgr2=0x55a6cc11abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc11abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f1cf9640 1 -- 172.21.15.27:0/3335577470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 msgr2=0x55a6cc1272a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc1272a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f1cf9640 1 -- 172.21.15.27:0/3335577470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6cc0fe090 con 0x55a6cc11b620 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.121+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc11abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc117660 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4007850 tx=0x7ff8e401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=243dedd72a9c6d9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8d6ffd640 1 -- 172.21.15.27:0/3335577470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4011940 con 0x55a6cc11b620 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8d6ffd640 1 -- 172.21.15.27:0/3335577470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e401f3f0 con 0x55a6cc11b620 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8d6ffd640 1 -- 172.21.15.27:0/3335577470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4027630 con 0x55a6cc11b620 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8f645f740 1 -- 172.21.15.27:0/3335577470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 msgr2=0x55a6cc117660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc117660 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4007850 tx=0x7ff8e401b000 comp rx=0 tx=0).stop 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8f645f740 1 -- 172.21.15.27:0/3335577470 shutdown_connections 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc11b620 0x55a6cc117660 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110f50 0x55a6cc1272a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3335577470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc110630 0x55a6cc11abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.122+0000 7ff8f645f740 1 -- 172.21.15.27:0/3335577470 >> 172.21.15.27:0/3335577470 conn(0x55a6cc10c950 msgr2=0x55a6cc10b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 -- 172.21.15.27:0/3335577470 shutdown_connections 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 -- 172.21.15.27:0/3335577470 wait complete. 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 -- start start 2024-04-25T01:36:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 Processor -- start 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110630 0x55a6cc1239a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110f50 0x55a6cc114d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc115270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1b3930 con 0x55a6cc110f50 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1b4da0 con 0x55a6cc110630 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1b4000 con 0x55a6cc11b620 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f645f740 10 client.0 _ll_register_callbacks cb 0x7ff8e8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f0cf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110f50 0x55a6cc114d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f0cf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110f50 0x55a6cc114d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54034/0 (socket says 172.21.15.27:54034) 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/3418287055 learned_addr learned my addr 172.21.15.27:0/3418287055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.123+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110630 0x55a6cc1239a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/3418287055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 msgr2=0x55a6cc115270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc115270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/3418287055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110630 msgr2=0x55a6cc1239a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110630 0x55a6cc1239a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f0cf7640 1 -- 172.21.15.27:0/3418287055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6cc1157b0 con 0x55a6cc110f50 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f1cf9640 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f0cf7640 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110f50 0x55a6cc114d30 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc09b6d0 tx=0x7ff8dc00a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc6e673cf6d5e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/3418287055 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0b9060 con 0x55a6cc110f50 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/3418287055 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7ff8dc098040 con 0x55a6cc110f50 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f645f740 1 -- 172.21.15.27:0/3418287055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8e401f3f0 con 0x55a6cc110f50 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/3418287055 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff8dc01d040 con 0x55a6cc110f50 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f645f740 10 client.17931 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8f645f740 1 -- 172.21.15.27:0/3418287055 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6cc191af0 con 0x55a6cc110f50 2024-04-25T01:36:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.124+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/3418287055 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e401f3f0 con 0x55a6cc110f50 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/3418287055 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff8dc007ca0 con 0x55a6cc110f50 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8f645f740 10 client.17931 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8f645f740 10 client.17931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8f645f740 20 client.17931 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6cc191dd0 0x55a6cc1941c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8f645f740 4 client.17931 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8f645f740 1 -- 172.21.15.27:0/3418287055 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a6cbffa6f0 con 0x55a6cc191dd0 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6cc191dd0 0x55a6cc1941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8f14f8640 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6cc191dd0 0x55a6cc1941c0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11b6e018b389a467 server_cookie=aebe611eeebdb065 in_seq=0 out_seq=0 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.125+0000 7ff8d4ff9640 10 client.17931 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8d4ff9640 1 -- 172.21.15.27:0/3418287055 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a6cbffa6f0 con 0x55a6cc191dd0 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8d4ff9640 10 client.17931 handle_command_reply: tid=0 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 client.17931 shutdown 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 -- 172.21.15.27:0/3418287055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110f50 msgr2=0x55a6cc114d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110f50 0x55a6cc114d30 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc09b6d0 tx=0x7ff8dc00a3f0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 -- 172.21.15.27:0/3418287055 shutdown_connections 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55a6cc191dd0 0x55a6cc1941c0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a6cc11b620 0x55a6cc115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6cc110f50 0x55a6cc114d30 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8d4ff9640 0 client.17931 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 --2- 172.21.15.27:0/3418287055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55a6cc110630 0x55a6cc1239a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 -- 172.21.15.27:0/3418287055 >> 172.21.15.27:0/3418287055 conn(0x55a6cc10c950 msgr2=0x55a6cc10baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 -- 172.21.15.27:0/3418287055 shutdown_connections 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 1 -- 172.21.15.27:0/3418287055 wait complete. 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f645f740 20 client.17931 trim_cache size 0 max 16384 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:36:17.109 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:36:17.110 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:36:17.110 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:36:17.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f375c640 1 -- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8d80a2d00 msgr2=0x7ff8d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8d80a2d00 0x7ff8d80a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4007310 tx=0x7ff8e4007d10 comp rx=0 tx=0).stop 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f375c640 1 -- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec074400 msgr2=0x7ff8ec19eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.126+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec074400 0x7ff8ec19eba0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00c890 tx=0x7ff8dc00cd50 comp rx=0 tx=0).stop 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.127+0000 7ff8f375c640 1 -- 172.21.15.27:0/1966644961 shutdown_connections 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.127+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8ec112770 0x7ff8ec1a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.127+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff8d80a2d00 0x7ff8d80a51c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.127+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ec074400 0x7ff8ec19eba0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.127+0000 7ff8f375c640 1 --2- 172.21.15.27:0/1966644961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8ec073ae0 0x7ff8ec19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.127+0000 7ff8f375c640 1 -- 172.21.15.27:0/1966644961 >> 172.21.15.27:0/1966644961 conn(0x7ff8ec0fd250 msgr2=0x7ff8ec0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.127+0000 7ff8f375c640 1 -- 172.21.15.27:0/1966644961 shutdown_connections 2024-04-25T01:36:17.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.127+0000 7ff8f375c640 1 -- 172.21.15.27:0/1966644961 wait complete. 2024-04-25T01:36:17.114 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:36:17.114 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:36:17.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.239+0000 7f148a2a5640 1 -- 172.21.15.27:0/4019861350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841032f0 msgr2=0x7f148410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.239+0000 7f148a2a5640 1 --2- 172.21.15.27:0/4019861350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841032f0 0x7f148410b7d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f1478009900 tx=0x7f147802a510 comp rx=0 tx=0).stop 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 -- 172.21.15.27:0/4019861350 shutdown_connections 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 --2- 172.21.15.27:0/4019861350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f148410bd10 0x7f148410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 --2- 172.21.15.27:0/4019861350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841032f0 0x7f148410b7d0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 --2- 172.21.15.27:0/4019861350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14841029d0 0x7f1484102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 -- 172.21.15.27:0/4019861350 >> 172.21.15.27:0/4019861350 conn(0x7f1484077740 msgr2=0x7f1484077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 -- 172.21.15.27:0/4019861350 shutdown_connections 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 -- 172.21.15.27:0/4019861350 wait complete. 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 Processor -- start 2024-04-25T01:36:17.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 -- start start 2024-04-25T01:36:17.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.240+0000 7f148a2a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841029d0 0x7f1484197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f148a2a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841032f0 0x7f14841984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f148a2a5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f148410bd10 0x7f148419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f148a2a5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1484111b10 con 0x7f14841029d0 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f148a2a5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1484111990 con 0x7f148410bd10 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f148a2a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1484111c90 con 0x7f14841032f0 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841029d0 0x7f1484197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841029d0 0x7f1484197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54058/0 (socket says 172.21.15.27:54058) 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 -- 172.21.15.27:0/2602817615 learned_addr learned my addr 172.21.15.27:0/2602817615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1488842640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f148410bd10 0x7f148419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f14837fe640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841032f0 0x7f14841984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 -- 172.21.15.27:0/2602817615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841032f0 msgr2=0x7f14841984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841032f0 0x7f14841984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 -- 172.21.15.27:0/2602817615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f148410bd10 msgr2=0x7f148419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f148410bd10 0x7f148419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 -- 172.21.15.27:0/2602817615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f148419cf60 con 0x7f14841029d0 2024-04-25T01:36:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f1483fff640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841029d0 0x7f1484197f70 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f147400c920 tx=0x7f147400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f14817fa640 1 -- 172.21.15.27:0/2602817615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1474007c70 con 0x7f14841029d0 2024-04-25T01:36:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f14817fa640 1 -- 172.21.15.27:0/2602817615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1474007e10 con 0x7f14841029d0 2024-04-25T01:36:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f14817fa640 1 -- 172.21.15.27:0/2602817615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1474018890 con 0x7f14841029d0 2024-04-25T01:36:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f148419d250 con 0x7f14841029d0 2024-04-25T01:36:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.241+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14841a4ae0 con 0x7f14841029d0 2024-04-25T01:36:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.242+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1484068560 con 0x7f14841029d0 2024-04-25T01:36:17.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.243+0000 7f14817fa640 1 -- 172.21.15.27:0/2602817615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1474018a70 con 0x7f14841029d0 2024-04-25T01:36:17.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.244+0000 7f14817fa640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f146c0a2d00 0x7f146c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.244+0000 7f14837fe640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f146c0a2d00 0x7f146c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.244+0000 7f14817fa640 1 -- 172.21.15.27:0/2602817615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f14740cf940 con 0x7f14841029d0 2024-04-25T01:36:17.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.245+0000 7f14837fe640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f146c0a2d00 0x7f146c0a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1478009230 tx=0x7f14780058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.249+0000 7f14817fa640 1 -- 172.21.15.27:0/2602817615 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1474014030 con 0x7f14841029d0 2024-04-25T01:36:17.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.473+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14841992c0 con 0x7f14841029d0 2024-04-25T01:36:17.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.474+0000 7f14817fa640 1 -- 172.21.15.27:0/2602817615 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7f147409c110 con 0x7f14841029d0 2024-04-25T01:36:17.451 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:36:17.451 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:36:17.451 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:36:17.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f146c0a2d00 msgr2=0x7f146c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f146c0a2d00 0x7f146c0a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1478009230 tx=0x7f14780058d0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841029d0 msgr2=0x7f1484197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841029d0 0x7f1484197f70 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f147400c920 tx=0x7f147400cde0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 shutdown_connections 2024-04-25T01:36:17.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f148410bd10 0x7f148419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841032f0 0x7f14841984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f146c0a2d00 0x7f146c0a51c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 --2- 172.21.15.27:0/2602817615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14841029d0 0x7f1484197f70 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.476+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 >> 172.21.15.27:0/2602817615 conn(0x7f1484077740 msgr2=0x7f14840fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.477+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 shutdown_connections 2024-04-25T01:36:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.477+0000 7f148a2a5640 1 -- 172.21.15.27:0/2602817615 wait complete. 2024-04-25T01:36:17.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 -- 172.21.15.27:0/3258669294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c101710 msgr2=0x7feb7c101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 --2- 172.21.15.27:0/3258669294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c101710 0x7feb7c101b90 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7feb64009930 tx=0x7feb6402a420 comp rx=0 tx=0).stop 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 -- 172.21.15.27:0/3258669294 shutdown_connections 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 --2- 172.21.15.27:0/3258669294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb7c1020d0 0x7feb7c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 --2- 172.21.15.27:0/3258669294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c101710 0x7feb7c101b90 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 --2- 172.21.15.27:0/3258669294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb7c107720 0x7feb7c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 -- 172.21.15.27:0/3258669294 >> 172.21.15.27:0/3258669294 conn(0x7feb7c0fd180 msgr2=0x7feb7c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 -- 172.21.15.27:0/3258669294 shutdown_connections 2024-04-25T01:36:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.592+0000 7feb80daf640 1 -- 172.21.15.27:0/3258669294 wait complete. 2024-04-25T01:36:17.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb80daf640 1 Processor -- start 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb80daf640 1 -- start start 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb80daf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb7c101710 0x7feb7c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb80daf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb7c1020d0 0x7feb7c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb80daf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c107720 0x7feb7c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb80daf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c111b30 con 0x7feb7c107720 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb80daf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1119b0 con 0x7feb7c101710 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb80daf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c111cb0 con 0x7feb7c1020d0 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c107720 0x7feb7c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c107720 0x7feb7c0744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54076/0 (socket says 172.21.15.27:54076) 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7ad9d640 1 -- 172.21.15.27:0/2666008493 learned_addr learned my addr 172.21.15.27:0/2666008493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb7c1020d0 0x7feb7c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7a59c640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb7c101710 0x7feb7c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7ad9d640 1 -- 172.21.15.27:0/2666008493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb7c1020d0 msgr2=0x7feb7c073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7ad9d640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb7c1020d0 0x7feb7c073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7ad9d640 1 -- 172.21.15.27:0/2666008493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb7c101710 msgr2=0x7feb7c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7ad9d640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb7c101710 0x7feb7c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7ad9d640 1 -- 172.21.15.27:0/2666008493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c1a65f0 con 0x7feb7c107720 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb7c1020d0 0x7feb7c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.593+0000 7feb7a59c640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb7c101710 0x7feb7c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.594+0000 7feb7ad9d640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c107720 0x7feb7c0744c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7feb7400b970 tx=0x7feb7400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.594+0000 7feb737fe640 1 -- 172.21.15.27:0/2666008493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74019070 con 0x7feb7c107720 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.594+0000 7feb737fe640 1 -- 172.21.15.27:0/2666008493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb7400cb40 con 0x7feb7c107720 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.594+0000 7feb80daf640 1 -- 172.21.15.27:0/2666008493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7c1a68e0 con 0x7feb7c107720 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.594+0000 7feb737fe640 1 -- 172.21.15.27:0/2666008493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74007650 con 0x7feb7c107720 2024-04-25T01:36:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.594+0000 7feb80daf640 1 -- 172.21.15.27:0/2666008493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7c1a6e00 con 0x7feb7c107720 2024-04-25T01:36:17.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.596+0000 7feb737fe640 1 -- 172.21.15.27:0/2666008493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7feb7401c070 con 0x7feb7c107720 2024-04-25T01:36:17.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.596+0000 7feb83b95740 1 Processor -- start 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb83b95740 1 -- start start 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb83b95740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824580 0x56027f824960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb83b95740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824ea0 0x56027f82f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb83b95740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f82f570 0x56027f831960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb83b95740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56027f789670 con 0x56027f824580 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb83b95740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56027f731970 con 0x56027f82f570 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb83b95740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56027f785630 con 0x56027f824ea0 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824580 0x56027f824960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824580 0x56027f824960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54088/0 (socket says 172.21.15.27:54088) 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 -- 172.21.15.27:0/2045771597 learned_addr learned my addr 172.21.15.27:0/2045771597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb737fe640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb680a2d00 0x7feb680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f82f570 0x56027f831960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb7ad9d640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb680a2d00 0x7feb680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb7a59c640 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824ea0 0x56027f82f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 -- 172.21.15.27:0/2045771597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824ea0 msgr2=0x56027f82f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824ea0 0x56027f82f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 -- 172.21.15.27:0/2045771597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f82f570 msgr2=0x56027f831960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f82f570 0x56027f831960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb79d9b640 1 -- 172.21.15.27:0/2045771597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56027f755400 con 0x56027f824580 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb7a59c640 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824ea0 0x56027f82f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.597+0000 7feb737fe640 1 -- 172.21.15.27:0/2666008493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7feb740ced70 con 0x7feb7c107720 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824580 0x56027f824960 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7feb64005940 tx=0x7feb64031480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae08641099d7261 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb7ad9d640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb680a2d00 0x7feb680a51c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb7409acb0 tx=0x7feb740bb040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb70ff9640 1 -- 172.21.15.27:0/2045771597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64035050 con 0x56027f824580 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb70ff9640 1 -- 172.21.15.27:0/2045771597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb64031890 con 0x56027f824580 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb70ff9640 1 -- 172.21.15.27:0/2045771597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640046c0 con 0x56027f824580 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb83b95740 1 -- 172.21.15.27:0/2045771597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824580 msgr2=0x56027f824960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb83b95740 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824580 0x56027f824960 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7feb64005940 tx=0x7feb64031480 comp rx=0 tx=0).stop 2024-04-25T01:36:17.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb83b95740 1 -- 172.21.15.27:0/2045771597 shutdown_connections 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.598+0000 7feb83b95740 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f82f570 0x56027f831960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824ea0 0x56027f82f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 --2- 172.21.15.27:0/2045771597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824580 0x56027f824960 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 -- 172.21.15.27:0/2045771597 >> 172.21.15.27:0/2045771597 conn(0x56027f8208a0 msgr2=0x56027f820c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 -- 172.21.15.27:0/2045771597 shutdown_connections 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 -- 172.21.15.27:0/2045771597 wait complete. 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 -- start start 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 Processor -- start 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f82eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f824ea0 0x56027f82c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f82f570 0x56027f82c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56027f832300 con 0x56027f82f570 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56027f745050 con 0x56027f824ea0 2024-04-25T01:36:17.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56027f832480 con 0x56027f824580 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb83b95740 10 client.0 _ll_register_callbacks cb 0x56027f49cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb79d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f82f570 0x56027f82c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb79d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f82f570 0x56027f82c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54094/0 (socket says 172.21.15.27:54094) 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.599+0000 7feb79d9b640 1 -- 172.21.15.27:0/2957029422 learned_addr learned my addr 172.21.15.27:0/2957029422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb7a59c640 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f824ea0 0x56027f82c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f82eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb79d9b640 1 -- 172.21.15.27:0/2957029422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 msgr2=0x56027f82eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f82eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb79d9b640 1 -- 172.21.15.27:0/2957029422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f824ea0 msgr2=0x56027f82c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f824ea0 0x56027f82c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb79d9b640 1 -- 172.21.15.27:0/2957029422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56027f716dc0 con 0x56027f82f570 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb7a59c640 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f824ea0 0x56027f82c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb79d9b640 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f82f570 0x56027f82c710 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7feb64031b60 tx=0x7feb640054c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d58f31c002569b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb61ffb640 1 -- 172.21.15.27:0/2957029422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64035070 con 0x56027f82f570 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb61ffb640 1 -- 172.21.15.27:0/2957029422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7feb64004400 con 0x56027f82f570 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb83b95740 1 -- 172.21.15.27:0/2957029422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb64031890 con 0x56027f82f570 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb61ffb640 1 -- 172.21.15.27:0/2957029422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb64058050 con 0x56027f82f570 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb83b95740 10 client.17949 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb83b95740 1 -- 172.21.15.27:0/2957029422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56027f49d3f0 con 0x56027f82f570 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.600+0000 7feb61ffb640 1 -- 172.21.15.27:0/2957029422 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb64031890 con 0x56027f82f570 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.601+0000 7feb61ffb640 1 -- 172.21.15.27:0/2957029422 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7feb64003bf0 con 0x56027f82f570 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.601+0000 7feb83b95740 10 client.17949 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.601+0000 7feb83b95740 10 client.17949 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.601+0000 7feb83b95740 20 client.17949 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.601+0000 7feb83b95740 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027f83f3c0 0x56027f8417b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.601+0000 7feb83b95740 4 client.17949 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.601+0000 7feb83b95740 1 -- 172.21.15.27:0/2957029422 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56027f796bb0 con 0x56027f83f3c0 2024-04-25T01:36:17.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.602+0000 7feb7a59c640 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027f83f3c0 0x56027f8417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.603+0000 7feb7a59c640 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027f83f3c0 0x56027f8417b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3fd4be1650c53ac7 server_cookie=8f88aeb76eecc651 in_seq=0 out_seq=0 2024-04-25T01:36:17.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.603+0000 7feb61ffb640 10 client.17949 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb61ffb640 1 -- 172.21.15.27:0/2957029422 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56027f796bb0 con 0x56027f83f3c0 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb61ffb640 10 client.17949 handle_command_reply: tid=0 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 client.17949 shutdown 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 -- 172.21.15.27:0/2957029422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f82f570 msgr2=0x56027f82c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f82f570 0x56027f82c710 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7feb64031b60 tx=0x7feb640054c0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 -- 172.21.15.27:0/2957029422 shutdown_connections 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027f83f3c0 0x56027f8417b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f82f570 0x56027f82c710 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f824ea0 0x56027f82c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb61ffb640 0 client.17949 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:36:17.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 --2- 172.21.15.27:0/2957029422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f82eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 -- 172.21.15.27:0/2957029422 >> 172.21.15.27:0/2957029422 conn(0x56027f8208a0 msgr2=0x56027f81fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 -- 172.21.15.27:0/2957029422 shutdown_connections 2024-04-25T01:36:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 1 -- 172.21.15.27:0/2957029422 wait complete. 2024-04-25T01:36:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.604+0000 7feb83b95740 20 client.17949 trim_cache size 0 max 16384 2024-04-25T01:36:17.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb83b95740 1 Processor -- start 2024-04-25T01:36:17.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb83b95740 1 -- start start 2024-04-25T01:36:17.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb83b95740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f82e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb83b95740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f829410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb83b95740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 0x56027f829950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb83b95740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1b37d0 con 0x56027f824ea0 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb83b95740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1b10a0 con 0x56027f827aa0 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb83b95740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1aef10 con 0x56027f824580 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.617+0000 7feb7a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f829410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb7a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f829410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54104/0 (socket says 172.21.15.27:54104) 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb7a59c640 1 -- 172.21.15.27:0/4045186843 learned_addr learned my addr 172.21.15.27:0/4045186843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb79d9b640 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f82e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb7a59c640 1 -- 172.21.15.27:0/4045186843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 msgr2=0x56027f82e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb7a59c640 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f82e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb7a59c640 1 -- 172.21.15.27:0/4045186843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 msgr2=0x56027f829950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb79d9b640 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 0x56027f829950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb7a59c640 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 0x56027f829950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb7a59c640 1 -- 172.21.15.27:0/4045186843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56027f5677d0 con 0x56027f824ea0 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb79d9b640 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 0x56027f829950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:17.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb7a59c640 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f829410 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7feb6c005d00 tx=0x7feb6c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9b7a0354616e82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb63fff640 1 -- 172.21.15.27:0/4045186843 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c004c60 con 0x56027f824ea0 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb63fff640 1 -- 172.21.15.27:0/4045186843 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6c004e00 con 0x56027f824ea0 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.618+0000 7feb63fff640 1 -- 172.21.15.27:0/4045186843 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c003850 con 0x56027f824ea0 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- 172.21.15.27:0/4045186843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 msgr2=0x56027f829410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f829410 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7feb6c005d00 tx=0x7feb6c00b790 comp rx=0 tx=0).stop 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- 172.21.15.27:0/4045186843 shutdown_connections 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 0x56027f829950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f829410 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 --2- 172.21.15.27:0/4045186843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f82e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- 172.21.15.27:0/4045186843 >> 172.21.15.27:0/4045186843 conn(0x56027f8208a0 msgr2=0x56027f83afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- 172.21.15.27:0/4045186843 shutdown_connections 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- 172.21.15.27:0/4045186843 wait complete. 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- start start 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 Processor -- start 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f844590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f844ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 0x56027f842b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1af560 con 0x56027f824ea0 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1b09d0 con 0x56027f827aa0 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1afc30 con 0x56027f824580 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb83b95740 10 client.0 _ll_register_callbacks cb 0x7feb64031640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb7a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f844ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb7a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f844ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54106/0 (socket says 172.21.15.27:54106) 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.619+0000 7feb7a59c640 1 -- 172.21.15.27:0/1143461498 learned_addr learned my addr 172.21.15.27:0/1143461498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb79d9b640 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f844590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb7a59c640 1 -- 172.21.15.27:0/1143461498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 msgr2=0x56027f844590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb7a59c640 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f844590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb7a59c640 1 -- 172.21.15.27:0/1143461498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 msgr2=0x56027f842b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb7a59c640 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 0x56027f842b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb7a59c640 1 -- 172.21.15.27:0/1143461498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56027f843200 con 0x56027f824ea0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb7a59c640 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f844ad0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7feb6c000c00 tx=0x7feb6c01a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ba45fb71aa4e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb61ffb640 1 -- 172.21.15.27:0/1143461498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c01aa00 con 0x56027f824ea0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb61ffb640 1 -- 172.21.15.27:0/1143461498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7feb6c01a330 con 0x56027f824ea0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb83b95740 1 -- 172.21.15.27:0/1143461498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb6c004e00 con 0x56027f824ea0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb61ffb640 1 -- 172.21.15.27:0/1143461498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb6c035050 con 0x56027f824ea0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb83b95740 10 client.17955 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb83b95740 1 -- 172.21.15.27:0/1143461498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56027f8a60b0 con 0x56027f824ea0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.620+0000 7feb61ffb640 1 -- 172.21.15.27:0/1143461498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6c004e00 con 0x56027f824ea0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.621+0000 7feb61ffb640 1 -- 172.21.15.27:0/1143461498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7feb6c003aa0 con 0x56027f824ea0 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.621+0000 7feb83b95740 10 client.17955 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.621+0000 7feb83b95740 10 client.17955 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.621+0000 7feb83b95740 20 client.17955 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.621+0000 7feb83b95740 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027f8a67d0 0x56027f8a8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.621+0000 7feb83b95740 4 client.17955 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.621+0000 7feb83b95740 1 -- 172.21.15.27:0/1143461498 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56027f790f00 con 0x56027f8a67d0 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.621+0000 7feb79d9b640 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027f8a67d0 0x56027f8a8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb79d9b640 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027f8a67d0 0x56027f8a8bf0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=42db0d67eb2f2197 server_cookie=f4633ac56bc5da68 in_seq=0 out_seq=0 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb61ffb640 10 client.17955 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb61ffb640 1 -- 172.21.15.27:0/1143461498 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56027f790f00 con 0x56027f8a67d0 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb61ffb640 10 client.17955 handle_command_reply: tid=0 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 client.17955 shutdown 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 -- 172.21.15.27:0/1143461498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 msgr2=0x56027f844ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f844ad0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7feb6c000c00 tx=0x7feb6c01a840 comp rx=0 tx=0).stop 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 -- 172.21.15.27:0/1143461498 shutdown_connections 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x56027f8a67d0 0x56027f8a8bf0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56027f827aa0 0x56027f842b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb61ffb640 0 client.17955 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:36:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56027f824ea0 0x56027f844ad0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 --2- 172.21.15.27:0/1143461498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56027f824580 0x56027f844590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 -- 172.21.15.27:0/1143461498 >> 172.21.15.27:0/1143461498 conn(0x56027f8208a0 msgr2=0x56027f825ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 -- 172.21.15.27:0/1143461498 shutdown_connections 2024-04-25T01:36:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 1 -- 172.21.15.27:0/1143461498 wait complete. 2024-04-25T01:36:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.622+0000 7feb83b95740 20 client.17955 trim_cache size 0 max 16384 2024-04-25T01:36:17.605 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:36:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 -- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb680a2d00 msgr2=0x7feb680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb680a2d00 0x7feb680a51c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb7409acb0 tx=0x7feb740bb040 comp rx=0 tx=0).stop 2024-04-25T01:36:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 -- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c107720 msgr2=0x7feb7c0744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c107720 0x7feb7c0744c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7feb7400b970 tx=0x7feb7400be30 comp rx=0 tx=0).stop 2024-04-25T01:36:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 -- 172.21.15.27:0/2666008493 shutdown_connections 2024-04-25T01:36:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb680a2d00 0x7feb680a51c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c107720 0x7feb7c0744c0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb7c1020d0 0x7feb7c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 --2- 172.21.15.27:0/2666008493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb7c101710 0x7feb7c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 -- 172.21.15.27:0/2666008493 >> 172.21.15.27:0/2666008493 conn(0x7feb7c0fd180 msgr2=0x7feb7c0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 -- 172.21.15.27:0/2666008493 shutdown_connections 2024-04-25T01:36:17.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.623+0000 7feb80daf640 1 -- 172.21.15.27:0/2666008493 wait complete. 2024-04-25T01:36:17.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:36:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 -- 172.21.15.27:0/2330730964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648102120 msgr2=0x7fc64810e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/2330730964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648102120 0x7fc64810e630 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fc640009cd0 tx=0x7fc64002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 -- 172.21.15.27:0/2330730964 shutdown_connections 2024-04-25T01:36:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/2330730964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648102120 0x7fc64810e630 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/2330730964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc648101760 0x7fc648101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/2330730964 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc648107770 0x7fc648107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 -- 172.21.15.27:0/2330730964 >> 172.21.15.27:0/2330730964 conn(0x7fc6480fd250 msgr2=0x7fc6480ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 -- 172.21.15.27:0/2330730964 shutdown_connections 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.736+0000 7fc64ca9e640 1 -- 172.21.15.27:0/2330730964 wait complete. 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64ca9e640 1 Processor -- start 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64ca9e640 1 -- start start 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64ca9e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648101760 0x7fc648073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64ca9e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc648102120 0x7fc648073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64ca9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc648107770 0x7fc6480744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64ca9e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc648111b80 con 0x7fc648101760 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64ca9e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc648111a00 con 0x7fc648102120 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64ca9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc648111d00 con 0x7fc648107770 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.737+0000 7fc64659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648101760 0x7fc648073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648101760 0x7fc648073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51100/0 (socket says 172.21.15.27:51100) 2024-04-25T01:36:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64659c640 1 -- 172.21.15.27:0/4157306134 learned_addr learned my addr 172.21.15.27:0/4157306134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc645d9b640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc648102120 0x7fc648073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc646d9d640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc648107770 0x7fc6480744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64659c640 1 -- 172.21.15.27:0/4157306134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc648107770 msgr2=0x7fc6480744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64659c640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc648107770 0x7fc6480744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64659c640 1 -- 172.21.15.27:0/4157306134 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc648102120 msgr2=0x7fc648073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64659c640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc648102120 0x7fc648073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64659c640 1 -- 172.21.15.27:0/4157306134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6481a6520 con 0x7fc648101760 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64659c640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648101760 0x7fc648073a40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc63800c920 tx=0x7fc63800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc6377fe640 1 -- 172.21.15.27:0/4157306134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc638007c70 con 0x7fc648101760 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc6377fe640 1 -- 172.21.15.27:0/4157306134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc638007e10 con 0x7fc648101760 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64ca9e640 1 -- 172.21.15.27:0/4157306134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6481a6810 con 0x7fc648101760 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc6377fe640 1 -- 172.21.15.27:0/4157306134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc638018890 con 0x7fc648101760 2024-04-25T01:36:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.738+0000 7fc64ca9e640 1 -- 172.21.15.27:0/4157306134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6481a6d60 con 0x7fc648101760 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.740+0000 7fc6377fe640 1 -- 172.21.15.27:0/4157306134 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc638018a70 con 0x7fc648101760 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.740+0000 7fc64f7a9740 1 Processor -- start 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64f7a9740 1 -- start start 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 0x561fadb6e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb79030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadb7b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561fadad3670 con 0x561fadb6eea0 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561fada7b970 con 0x561fadb6e580 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561fadacf630 con 0x561fadb79570 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb79030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb79030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51108/0 (socket says 172.21.15.27:51108) 2024-04-25T01:36:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64659c640 1 -- 172.21.15.27:0/825377381 learned_addr learned my addr 172.21.15.27:0/825377381 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc646d9d640 1 --2- 172.21.15.27:0/825377381 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 0x561fadb6e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc6377fe640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6300a2d00 0x7fc6300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64659c640 1 -- 172.21.15.27:0/825377381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 msgr2=0x561fadb7b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64659c640 1 --2- 172.21.15.27:0/825377381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadb7b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64659c640 1 -- 172.21.15.27:0/825377381 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 msgr2=0x561fadb6e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64659c640 1 --2- 172.21.15.27:0/825377381 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 0x561fadb6e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc64659c640 1 -- 172.21.15.27:0/825377381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fada9f400 con 0x561fadb6eea0 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.741+0000 7fc646d9d640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6300a2d00 0x7fc6300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.742+0000 7fc6377fe640 1 -- 172.21.15.27:0/4157306134 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fc6380cf890 con 0x7fc648101760 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.742+0000 7fc64659c640 1 --2- 172.21.15.27:0/825377381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb79030 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fc6380ccb60 tx=0x7fc63806ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c64b320bcf0dca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.742+0000 7fc634ff9640 1 -- 172.21.15.27:0/825377381 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63806ce60 con 0x561fadb6eea0 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.742+0000 7fc634ff9640 1 -- 172.21.15.27:0/825377381 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc63809c0c0 con 0x561fadb6eea0 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.742+0000 7fc634ff9640 1 -- 172.21.15.27:0/825377381 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63806ef30 con 0x561fadb6eea0 2024-04-25T01:36:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- 172.21.15.27:0/825377381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 msgr2=0x561fadb79030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/825377381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb79030 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fc6380ccb60 tx=0x7fc63806ea30 comp rx=0 tx=0).stop 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc646d9d640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6300a2d00 0x7fc6300a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc640009a60 tx=0x7fc64003e040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- 172.21.15.27:0/825377381 shutdown_connections 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/825377381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadb7b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/825377381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb79030 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/825377381 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 0x561fadb6e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- 172.21.15.27:0/825377381 >> 172.21.15.27:0/825377381 conn(0x561fadb6a8a0 msgr2=0x561fadb6ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- 172.21.15.27:0/825377381 shutdown_connections 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- 172.21.15.27:0/825377381 wait complete. 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- start start 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 Processor -- start 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 0x561fadb78ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadb76760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561fadb7c2a0 con 0x561fadb6eea0 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561fada8f050 con 0x561fadb6e580 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561fadb7c420 con 0x561fadb79570 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc64f7a9740 10 client.0 _ll_register_callbacks cb 0x561fad7e6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc645d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc645d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb761e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51122/0 (socket says 172.21.15.27:51122) 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.743+0000 7fc645d9b640 1 -- 172.21.15.27:0/2844391896 learned_addr learned my addr 172.21.15.27:0/2844391896 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc64659c640 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 0x561fadb78ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc646d9d640 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadb76760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc645d9b640 1 -- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 msgr2=0x561fadb76760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc645d9b640 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadb76760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc645d9b640 1 -- 172.21.15.27:0/2844391896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 msgr2=0x561fadb78ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc645d9b640 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 0x561fadb78ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc645d9b640 1 -- 172.21.15.27:0/2844391896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fada60dc0 con 0x561fadb6eea0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc646d9d640 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadb76760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc645d9b640 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb761e0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fc63c00b8f0 tx=0x7fc63c00bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5953bdb22c488f53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc62dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc62dffb640 1 -- 172.21.15.27:0/2844391896 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c005020 con 0x561fadb6eea0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc62dffb640 1 -- 172.21.15.27:0/2844391896 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fc63c0051c0 con 0x561fadb6eea0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc64f7a9740 1 -- 172.21.15.27:0/2844391896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc63809c0c0 con 0x561fadb6eea0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc62dffb640 1 -- 172.21.15.27:0/2844391896 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc63c022050 con 0x561fadb6eea0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc64f7a9740 10 client.17967 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.744+0000 7fc64f7a9740 1 -- 172.21.15.27:0/2844391896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561fad7e73f0 con 0x561fadb6eea0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc62dffb640 1 -- 172.21.15.27:0/2844391896 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc63809c0c0 con 0x561fadb6eea0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc62dffb640 1 -- 172.21.15.27:0/2844391896 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc63c0115d0 con 0x561fadb6eea0 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc64f7a9740 10 client.17967 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc64f7a9740 10 client.17967 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc64f7a9740 20 client.17967 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x561fadb89580 0x561fadb8b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc64f7a9740 4 client.17967 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc64f7a9740 1 -- 172.21.15.27:0/2844391896 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fadae0bb0 con 0x561fadb89580 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.745+0000 7fc64659c640 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x561fadb89580 0x561fadb8b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.746+0000 7fc62dffb640 10 client.17967 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.746+0000 7fc64659c640 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x561fadb89580 0x561fadb8b970 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4f67361bc4875024 server_cookie=73928b7265b9a564 in_seq=0 out_seq=0 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc62dffb640 1 -- 172.21.15.27:0/2844391896 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fadae0bb0 con 0x561fadb89580 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc62dffb640 10 client.17967 handle_command_reply: tid=0 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 client.17967 shutdown 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 -- 172.21.15.27:0/2844391896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 msgr2=0x561fadb761e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb761e0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fc63c00b8f0 tx=0x7fc63c00bdb0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 -- 172.21.15.27:0/2844391896 shutdown_connections 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x561fadb89580 0x561fadb8b970 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadb76760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc62dffb640 0 client.17967 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb761e0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/2844391896 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6e580 0x561fadb78ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 -- 172.21.15.27:0/2844391896 >> 172.21.15.27:0/2844391896 conn(0x561fadb6a8a0 msgr2=0x561fadb69b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 -- 172.21.15.27:0/2844391896 shutdown_connections 2024-04-25T01:36:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 1 -- 172.21.15.27:0/2844391896 wait complete. 2024-04-25T01:36:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.747+0000 7fc64f7a9740 20 client.17967 trim_cache size 0 max 16384 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.760+0000 7fc64f7a9740 1 Processor -- start 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.760+0000 7fc64f7a9740 1 -- start start 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.760+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb6e580 0x561fadb85160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.760+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb78a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.760+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb79570 0x561fadb91240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.760+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6481b37f0 con 0x561fadb6eea0 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.760+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc6481b10c0 con 0x561fadb79570 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.760+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6481aef30 con 0x561fadb6e580 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb78a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb78a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51130/0 (socket says 172.21.15.27:51130) 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 -- 172.21.15.27:0/1198239152 learned_addr learned my addr 172.21.15.27:0/1198239152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 -- 172.21.15.27:0/1198239152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb6e580 msgr2=0x561fadb85160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc64659c640 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb6e580 0x561fadb85160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc646d9d640 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb79570 0x561fadb91240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb6e580 0x561fadb85160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 -- 172.21.15.27:0/1198239152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb79570 msgr2=0x561fadb91240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb79570 0x561fadb91240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 -- 172.21.15.27:0/1198239152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fadb76ad0 con 0x561fadb6eea0 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc645d9b640 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb78a20 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc63c000c00 tx=0x7fc63c005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24dc2bd65213d6f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc62ffff640 1 -- 172.21.15.27:0/1198239152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c003d60 con 0x561fadb6eea0 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc62ffff640 1 -- 172.21.15.27:0/1198239152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc63c02a070 con 0x561fadb6eea0 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.761+0000 7fc62ffff640 1 -- 172.21.15.27:0/1198239152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c0053a0 con 0x561fadb6eea0 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- 172.21.15.27:0/1198239152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 msgr2=0x561fadb78a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb78a20 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc63c000c00 tx=0x7fc63c005fb0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- 172.21.15.27:0/1198239152 shutdown_connections 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb79570 0x561fadb91240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6eea0 0x561fadb78a20 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/1198239152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb6e580 0x561fadb85160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- 172.21.15.27:0/1198239152 >> 172.21.15.27:0/1198239152 conn(0x561fadb6a8a0 msgr2=0x561fadb71860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- 172.21.15.27:0/1198239152 shutdown_connections 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- 172.21.15.27:0/1198239152 wait complete. 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- start start 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 Processor -- start 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6e580 0x561fadb735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6eea0 0x561fadb73b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadbeeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6481af580 con 0x561fadb6e580 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc6481b09f0 con 0x561fadb6eea0 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.762+0000 7fc64f7a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6481afc50 con 0x561fadb79570 2024-04-25T01:36:17.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64f7a9740 10 client.0 _ll_register_callbacks cb 0x7fc63c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6e580 0x561fadb735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6e580 0x561fadb735c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51140/0 (socket says 172.21.15.27:51140) 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 -- 172.21.15.27:0/4203935840 learned_addr learned my addr 172.21.15.27:0/4203935840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc646d9d640 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadbeeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 -- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 msgr2=0x561fadbeeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadbeeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 -- 172.21.15.27:0/4203935840 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6eea0 msgr2=0x561fadb73b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6eea0 0x561fadb73b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 -- 172.21.15.27:0/4203935840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fadb7bcd0 con 0x561fadb6e580 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc646d9d640 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadbeeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64659c640 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6e580 0x561fadb735c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc6380ccca0 tx=0x7fc63809b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acea7d25f9e8dae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc62dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc62dffb640 1 -- 172.21.15.27:0/4203935840 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63806ce60 con 0x561fadb6e580 2024-04-25T01:36:17.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc62dffb640 1 -- 172.21.15.27:0/4203935840 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fc638099a60 con 0x561fadb6e580 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64f7a9740 1 -- 172.21.15.27:0/4203935840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc63c02a070 con 0x561fadb6e580 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc62dffb640 1 -- 172.21.15.27:0/4203935840 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc63809c0c0 con 0x561fadb6e580 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64f7a9740 10 client.17973 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc64f7a9740 1 -- 172.21.15.27:0/4203935840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561fad8b17d0 con 0x561fadb6e580 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc62dffb640 1 -- 172.21.15.27:0/4203935840 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc63c02a070 con 0x561fadb6e580 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.763+0000 7fc62dffb640 1 -- 172.21.15.27:0/4203935840 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc6380cce70 con 0x561fadb6e580 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc64f7a9740 10 client.17973 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc64f7a9740 10 client.17973 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc64f7a9740 20 client.17973 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x561fadbf0260 0x561fadbf2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc64f7a9740 4 client.17973 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc64f7a9740 1 -- 172.21.15.27:0/4203935840 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fadadaf00 con 0x561fadbf0260 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc645d9b640 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x561fadbf0260 0x561fadbf2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc645d9b640 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x561fadbf0260 0x561fadbf2670 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fea0a2a5cdadcd4e server_cookie=b5133245aa48df84 in_seq=0 out_seq=0 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.764+0000 7fc62dffb640 10 client.17973 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc62dffb640 1 -- 172.21.15.27:0/4203935840 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fadadaf00 con 0x561fadbf0260 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc62dffb640 10 client.17973 handle_command_reply: tid=0 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 client.17973 shutdown 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 -- 172.21.15.27:0/4203935840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6e580 msgr2=0x561fadb735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6e580 0x561fadb735c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc6380ccca0 tx=0x7fc63809b320 comp rx=0 tx=0).stop 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 -- 172.21.15.27:0/4203935840 shutdown_connections 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x561fadbf0260 0x561fadbf2670 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fadb79570 0x561fadbeeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc62dffb640 0 client.17973 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fadb6eea0 0x561fadb73b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 --2- 172.21.15.27:0/4203935840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fadb6e580 0x561fadb735c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 -- 172.21.15.27:0/4203935840 >> 172.21.15.27:0/4203935840 conn(0x561fadb6a8a0 msgr2=0x561fadb70d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 -- 172.21.15.27:0/4203935840 shutdown_connections 2024-04-25T01:36:17.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 1 -- 172.21.15.27:0/4203935840 wait complete. 2024-04-25T01:36:17.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64f7a9740 20 client.17973 trim_cache size 0 max 16384 2024-04-25T01:36:17.748 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 -- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6300a2d00 msgr2=0x7fc6300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6300a2d00 0x7fc6300a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc640009a60 tx=0x7fc64003e040 comp rx=0 tx=0).stop 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 -- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648101760 msgr2=0x7fc648073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648101760 0x7fc648073a40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc63800c920 tx=0x7fc63800cde0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 -- 172.21.15.27:0/4157306134 shutdown_connections 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc648107770 0x7fc6480744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc648102120 0x7fc648073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc6300a2d00 0x7fc6300a51c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 --2- 172.21.15.27:0/4157306134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc648101760 0x7fc648073a40 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.765+0000 7fc64ca9e640 1 -- 172.21.15.27:0/4157306134 >> 172.21.15.27:0/4157306134 conn(0x7fc6480fd250 msgr2=0x7fc6480fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.766+0000 7fc64ca9e640 1 -- 172.21.15.27:0/4157306134 shutdown_connections 2024-04-25T01:36:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.766+0000 7fc64ca9e640 1 -- 172.21.15.27:0/4157306134 wait complete. 2024-04-25T01:36:17.751 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:17 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2602817615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:17.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:36:17.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:17 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2602817615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:17.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:17 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2602817615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 -- 172.21.15.27:0/2177551804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c100fd0 msgr2=0x7f983c10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 --2- 172.21.15.27:0/2177551804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c100fd0 0x7f983c10d4e0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f9834009cd0 tx=0x7f983402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 -- 172.21.15.27:0/2177551804 shutdown_connections 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 --2- 172.21.15.27:0/2177551804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c100fd0 0x7f983c10d4e0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 --2- 172.21.15.27:0/2177551804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983c100610 0x7f983c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 --2- 172.21.15.27:0/2177551804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f983c1066b0 0x7f983c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 -- 172.21.15.27:0/2177551804 >> 172.21.15.27:0/2177551804 conn(0x7f983c077740 msgr2=0x7f983c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 -- 172.21.15.27:0/2177551804 shutdown_connections 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 -- 172.21.15.27:0/2177551804 wait complete. 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 Processor -- start 2024-04-25T01:36:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.882+0000 7f9842fb7640 1 -- start start 2024-04-25T01:36:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9842fb7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983c100610 0x7f983c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9842fb7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f983c100fd0 0x7f983c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9842fb7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c1066b0 0x7f983c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9842fb7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1109f0 con 0x7f983c1066b0 2024-04-25T01:36:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9842fb7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f983c110870 con 0x7f983c100fd0 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9842fb7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f983c110b70 con 0x7f983c100610 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c1066b0 0x7f983c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c1066b0 0x7f983c19c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51170/0 (socket says 172.21.15.27:51170) 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 -- 172.21.15.27:0/1984626892 learned_addr learned my addr 172.21.15.27:0/1984626892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9840d53640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983c100610 0x7f983c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f983bfff640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f983c100fd0 0x7f983c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 -- 172.21.15.27:0/1984626892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983c100610 msgr2=0x7f983c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983c100610 0x7f983c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 -- 172.21.15.27:0/1984626892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f983c100fd0 msgr2=0x7f983c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f983c100fd0 0x7f983c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 -- 172.21.15.27:0/1984626892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983c19cfc0 con 0x7f983c1066b0 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9841554640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c1066b0 0x7f983c19c890 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f9834005cc0 tx=0x7f9834003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9839ffb640 1 -- 172.21.15.27:0/1984626892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9834004040 con 0x7f983c1066b0 2024-04-25T01:36:17.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9839ffb640 1 -- 172.21.15.27:0/1984626892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98340046c0 con 0x7f983c1066b0 2024-04-25T01:36:17.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.883+0000 7f9842fb7640 1 -- 172.21.15.27:0/1984626892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f983c19d250 con 0x7f983c1066b0 2024-04-25T01:36:17.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.884+0000 7f9839ffb640 1 -- 172.21.15.27:0/1984626892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9834046480 con 0x7f983c1066b0 2024-04-25T01:36:17.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.884+0000 7f9842fb7640 1 -- 172.21.15.27:0/1984626892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f983c1a4b40 con 0x7f983c1066b0 2024-04-25T01:36:17.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9839ffb640 1 -- 172.21.15.27:0/1984626892 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f98340305a0 con 0x7f983c1066b0 2024-04-25T01:36:17.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9845da3740 1 Processor -- start 2024-04-25T01:36:17.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9845da3740 1 -- start start 2024-04-25T01:36:17.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9845da3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e53a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9845da3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e545030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9845da3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 0x55bb6e547960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9845da3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bb6e49f670 con 0x55bb6e53aea0 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9845da3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bb6e447970 con 0x55bb6e53a580 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9845da3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bb6e49b630 con 0x55bb6e545570 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.886+0000 7f9839ffb640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f982c0a2d80 0x7f982c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e545030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9841554640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f982c0a2d80 0x7f982c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e545030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51176/0 (socket says 172.21.15.27:51176) 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 -- 172.21.15.27:0/2537161577 learned_addr learned my addr 172.21.15.27:0/2537161577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9839ffb640 1 -- 172.21.15.27:0/1984626892 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f983402e030 con 0x7f983c1066b0 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f983bfff640 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e53a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f983bfff640 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 0x55bb6e547960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 -- 172.21.15.27:0/2537161577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 msgr2=0x55bb6e547960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 0x55bb6e547960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 -- 172.21.15.27:0/2537161577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 msgr2=0x55bb6e53a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e53a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 -- 172.21.15.27:0/2537161577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb6e46b400 con 0x55bb6e53aea0 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9841554640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f982c0a2d80 0x7f982c0a5240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f98340ee780 tx=0x7f98340dd040 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9840d53640 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e545030 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f983000bc30 tx=0x7f98300107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11de1cd1d2f6f0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9826ffd640 1 -- 172.21.15.27:0/2537161577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830004020 con 0x55bb6e53aea0 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9826ffd640 1 -- 172.21.15.27:0/2537161577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98300041c0 con 0x55bb6e53aea0 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.887+0000 7f9826ffd640 1 -- 172.21.15.27:0/2537161577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830011750 con 0x55bb6e53aea0 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 -- 172.21.15.27:0/2537161577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 msgr2=0x55bb6e545030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e545030 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f983000bc30 tx=0x7f98300107a0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 -- 172.21.15.27:0/2537161577 shutdown_connections 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 0x55bb6e547960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e545030 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 --2- 172.21.15.27:0/2537161577 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e53a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 -- 172.21.15.27:0/2537161577 >> 172.21.15.27:0/2537161577 conn(0x55bb6e5368a0 msgr2=0x55bb6e536c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 -- 172.21.15.27:0/2537161577 shutdown_connections 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 -- 172.21.15.27:0/2537161577 wait complete. 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 -- start start 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.888+0000 7f9845da3740 1 Processor -- start 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9845da3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53a580 0x55bb6e544f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9845da3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e542200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9845da3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e545570 0x55bb6e5427a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9845da3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bb6e548360 con 0x55bb6e53aea0 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9845da3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bb6e45b050 con 0x55bb6e545570 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9845da3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bb6e5484e0 con 0x55bb6e53a580 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9845da3740 10 client.0 _ll_register_callbacks cb 0x55bb6e1b2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e542200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e542200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51180/0 (socket says 172.21.15.27:51180) 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 -- 172.21.15.27:0/631124793 learned_addr learned my addr 172.21.15.27:0/631124793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f983bfff640 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53a580 0x55bb6e544f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f983bfff640 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e545570 0x55bb6e5427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 -- 172.21.15.27:0/631124793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53a580 msgr2=0x55bb6e544f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53a580 0x55bb6e544f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 -- 172.21.15.27:0/631124793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e545570 msgr2=0x55bb6e5427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e545570 0x55bb6e5427a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 -- 172.21.15.27:0/631124793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb6e42cdc0 con 0x55bb6e53aea0 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f9840d53640 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e542200 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f9830004180 tx=0x7f98300058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb55707d7f815aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.889+0000 7f981bfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f981bfff640 1 -- 172.21.15.27:0/631124793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983002a060 con 0x55bb6e53aea0 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f981bfff640 1 -- 172.21.15.27:0/631124793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f983000bd20 con 0x55bb6e53aea0 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 1 -- 172.21.15.27:0/631124793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98300041c0 con 0x55bb6e53aea0 2024-04-25T01:36:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f981bfff640 1 -- 172.21.15.27:0/631124793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9830031050 con 0x55bb6e53aea0 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 10 client.17985 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 1 -- 172.21.15.27:0/631124793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bb6e1b33f0 con 0x55bb6e53aea0 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f981bfff640 1 -- 172.21.15.27:0/631124793 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98300041c0 con 0x55bb6e53aea0 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f981bfff640 1 -- 172.21.15.27:0/631124793 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f98300046c0 con 0x55bb6e53aea0 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 10 client.17985 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 10 client.17985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 20 client.17985 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bb6e5555f0 0x55bb6e5579e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 4 client.17985 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.890+0000 7f9845da3740 1 -- 172.21.15.27:0/631124793 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb6e4acbb0 con 0x55bb6e5555f0 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.891+0000 7f983bfff640 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bb6e5555f0 0x55bb6e5579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.891+0000 7f981bfff640 10 client.17985 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.891+0000 7f983bfff640 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bb6e5555f0 0x55bb6e5579e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6787b448de7ce8e5 server_cookie=9cb263a81492747c in_seq=0 out_seq=0 2024-04-25T01:36:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f981bfff640 1 -- 172.21.15.27:0/631124793 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb6e4acbb0 con 0x55bb6e5555f0 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f981bfff640 10 client.17985 handle_command_reply: tid=0 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 client.17985 shutdown 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 -- 172.21.15.27:0/631124793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 msgr2=0x55bb6e542200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e542200 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f9830004180 tx=0x7f98300058f0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 -- 172.21.15.27:0/631124793 shutdown_connections 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bb6e5555f0 0x55bb6e5579e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e545570 0x55bb6e5427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e542200 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f981bfff640 0 client.17985 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 --2- 172.21.15.27:0/631124793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53a580 0x55bb6e544f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 -- 172.21.15.27:0/631124793 >> 172.21.15.27:0/631124793 conn(0x55bb6e5368a0 msgr2=0x55bb6e535b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 -- 172.21.15.27:0/631124793 shutdown_connections 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 1 -- 172.21.15.27:0/631124793 wait complete. 2024-04-25T01:36:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.892+0000 7f9845da3740 20 client.17985 trim_cache size 0 max 16384 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.905+0000 7f9845da3740 1 Processor -- start 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.905+0000 7f9845da3740 1 -- start start 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9845da3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e551220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9845da3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e544a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9845da3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 0x55bb6e55cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9845da3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1b1580 con 0x55bb6e53aea0 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9845da3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1aee50 con 0x55bb6e53a580 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9845da3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1accc0 con 0x55bb6e545570 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e544a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e544a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51196/0 (socket says 172.21.15.27:51196) 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 -- 172.21.15.27:0/629690934 learned_addr learned my addr 172.21.15.27:0/629690934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f983bfff640 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e551220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 -- 172.21.15.27:0/629690934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 msgr2=0x55bb6e55cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 0x55bb6e55cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 -- 172.21.15.27:0/629690934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 msgr2=0x55bb6e551220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e551220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 -- 172.21.15.27:0/629690934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb6e542b10 con 0x55bb6e53aea0 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f983bfff640 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e551220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f9840d53640 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e544a40 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9830004c30 tx=0x7f983000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9d5f85cb61f010 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f98267fc640 1 -- 172.21.15.27:0/629690934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983001f070 con 0x55bb6e53aea0 2024-04-25T01:36:17.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f98267fc640 1 -- 172.21.15.27:0/629690934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f983000b800 con 0x55bb6e53aea0 2024-04-25T01:36:17.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.906+0000 7f98267fc640 1 -- 172.21.15.27:0/629690934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983002a070 con 0x55bb6e53aea0 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 -- 172.21.15.27:0/629690934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 msgr2=0x55bb6e544a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e544a40 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9830004c30 tx=0x7f983000b390 comp rx=0 tx=0).stop 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 -- 172.21.15.27:0/629690934 shutdown_connections 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e545570 0x55bb6e55cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e53aea0 0x55bb6e544a40 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 --2- 172.21.15.27:0/629690934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e551220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 -- 172.21.15.27:0/629690934 >> 172.21.15.27:0/629690934 conn(0x55bb6e5368a0 msgr2=0x55bb6e53d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 -- 172.21.15.27:0/629690934 shutdown_connections 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 -- 172.21.15.27:0/629690934 wait complete. 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 -- start start 2024-04-25T01:36:17.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.907+0000 7f9845da3740 1 Processor -- start 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9845da3740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e54d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9845da3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53aea0 0x55bb6e548d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9845da3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e545570 0x55bb6e5492b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9845da3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1ad310 con 0x55bb6e545570 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9845da3740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1ae780 con 0x55bb6e53a580 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9845da3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1ad9e0 con 0x55bb6e53aea0 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9845da3740 10 client.0 _ll_register_callbacks cb 0x7f982800a990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e545570 0x55bb6e5492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e545570 0x55bb6e5492b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51206/0 (socket says 172.21.15.27:51206) 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 -- 172.21.15.27:0/1052024104 learned_addr learned my addr 172.21.15.27:0/1052024104 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9840d53640 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53aea0 0x55bb6e548d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e54d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 -- 172.21.15.27:0/1052024104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53aea0 msgr2=0x55bb6e548d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53aea0 0x55bb6e548d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 -- 172.21.15.27:0/1052024104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 msgr2=0x55bb6e54d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e54d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 -- 172.21.15.27:0/1052024104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb6e27d7d0 con 0x55bb6e545570 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f983bfff640 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e545570 0x55bb6e5492b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x55bb6e537a30 tx=0x7f9828004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5af362c6d9d9a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f981bfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f981bfff640 1 -- 172.21.15.27:0/1052024104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982801a040 con 0x55bb6e545570 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f981bfff640 1 -- 172.21.15.27:0/1052024104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f9828004e00 con 0x55bb6e545570 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.908+0000 7f9845da3740 1 -- 172.21.15.27:0/1052024104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f983000b800 con 0x55bb6e545570 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f981bfff640 1 -- 172.21.15.27:0/1052024104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9828030050 con 0x55bb6e545570 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9845da3740 10 client.17991 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9845da3740 1 -- 172.21.15.27:0/1052024104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bb6e5bb4a0 con 0x55bb6e545570 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f981bfff640 1 -- 172.21.15.27:0/1052024104 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f983000b800 con 0x55bb6e545570 2024-04-25T01:36:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f981bfff640 1 -- 172.21.15.27:0/1052024104 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9828020050 con 0x55bb6e545570 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9845da3740 10 client.17991 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9845da3740 10 client.17991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9845da3740 20 client.17991 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9845da3740 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bb6e5bbc20 0x55bb6e5be040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9845da3740 4 client.17991 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9845da3740 1 -- 172.21.15.27:0/1052024104 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb6e4a6f00 con 0x55bb6e5bbc20 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.909+0000 7f9840d53640 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bb6e5bbc20 0x55bb6e5be040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f981bfff640 10 client.17991 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9840d53640 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bb6e5bbc20 0x55bb6e5be040 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68eebbc2ff18c33f server_cookie=6488b22e82b2fc4f in_seq=0 out_seq=0 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f981bfff640 1 -- 172.21.15.27:0/1052024104 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb6e4a6f00 con 0x55bb6e5bbc20 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f981bfff640 10 client.17991 handle_command_reply: tid=0 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 client.17991 shutdown 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 -- 172.21.15.27:0/1052024104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e545570 msgr2=0x55bb6e5492b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e545570 0x55bb6e5492b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x55bb6e537a30 tx=0x7f9828004730 comp rx=0 tx=0).stop 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 -- 172.21.15.27:0/1052024104 shutdown_connections 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55bb6e5bbc20 0x55bb6e5be040 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bb6e545570 0x55bb6e5492b0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f981bfff640 0 client.17991 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bb6e53aea0 0x55bb6e548d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 --2- 172.21.15.27:0/1052024104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bb6e53a580 0x55bb6e54d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 -- 172.21.15.27:0/1052024104 >> 172.21.15.27:0/1052024104 conn(0x55bb6e5368a0 msgr2=0x55bb6e550210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 -- 172.21.15.27:0/1052024104 shutdown_connections 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 1 -- 172.21.15.27:0/1052024104 wait complete. 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.910+0000 7f9845da3740 20 client.17991 trim_cache size 0 max 16384 2024-04-25T01:36:17.893 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:36:17.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 -- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f982c0a2d80 msgr2=0x7f982c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f982c0a2d80 0x7f982c0a5240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f98340ee780 tx=0x7f98340dd040 comp rx=0 tx=0).stop 2024-04-25T01:36:17.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 -- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c1066b0 msgr2=0x7f983c19c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c1066b0 0x7f983c19c890 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f9834005cc0 tx=0x7f9834003970 comp rx=0 tx=0).stop 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 -- 172.21.15.27:0/1984626892 shutdown_connections 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f982c0a2d80 0x7f982c0a5240 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c1066b0 0x7f983c19c890 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f983c100fd0 0x7f983c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 --2- 172.21.15.27:0/1984626892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983c100610 0x7f983c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 -- 172.21.15.27:0/1984626892 >> 172.21.15.27:0/1984626892 conn(0x7f983c077740 msgr2=0x7f983c0ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 -- 172.21.15.27:0/1984626892 shutdown_connections 2024-04-25T01:36:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:17.911+0000 7f9842fb7640 1 -- 172.21.15.27:0/1984626892 wait complete. 2024-04-25T01:36:18.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:18 smithi153 ceph-mon[32621]: pgmap v339: 129 pgs: 129 active+clean; 5.3 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:18.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:18 smithi139 ceph-mon[32760]: pgmap v339: 129 pgs: 129 active+clean; 5.3 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:18.898 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:36:18.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:36:18.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:36:18.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:18 smithi027 ceph-mon[29197]: pgmap v339: 129 pgs: 129 active+clean; 5.3 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.041+0000 7ffbf16da640 1 -- 172.21.15.27:0/4167437416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec1002e0 msgr2=0x7ffbec1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.041+0000 7ffbf16da640 1 --2- 172.21.15.27:0/4167437416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec1002e0 0x7ffbec1006c0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009900 tx=0x7ffbdc02a410 comp rx=0 tx=0).stop 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.041+0000 7ffbf16da640 1 -- 172.21.15.27:0/4167437416 shutdown_connections 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.041+0000 7ffbf16da640 1 --2- 172.21.15.27:0/4167437416 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec106aa0 0x7ffbec10e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.041+0000 7ffbf16da640 1 --2- 172.21.15.27:0/4167437416 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbec100c90 0x7ffbec1063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.041+0000 7ffbf16da640 1 --2- 172.21.15.27:0/4167437416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec1002e0 0x7ffbec1006c0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.041+0000 7ffbf16da640 1 -- 172.21.15.27:0/4167437416 >> 172.21.15.27:0/4167437416 conn(0x7ffbec0758d0 msgr2=0x7ffbec075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 -- 172.21.15.27:0/4167437416 shutdown_connections 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 -- 172.21.15.27:0/4167437416 wait complete. 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 Processor -- start 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 -- start start 2024-04-25T01:36:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbec1002e0 0x7ffbec198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec106aa0 0x7ffbec198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec19c8e0 0x7ffbec19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec111be0 con 0x7ffbec19c8e0 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec111a60 con 0x7ffbec1002e0 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.042+0000 7ffbf16da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec111d60 con 0x7ffbec106aa0 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec19c8e0 0x7ffbec19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec19c8e0 0x7ffbec19cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51212/0 (socket says 172.21.15.27:51212) 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 -- 172.21.15.27:0/284881277 learned_addr learned my addr 172.21.15.27:0/284881277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbec1002e0 0x7ffbec198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec106aa0 0x7ffbec198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 -- 172.21.15.27:0/284881277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec106aa0 msgr2=0x7ffbec198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec106aa0 0x7ffbec198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 -- 172.21.15.27:0/284881277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbec1002e0 msgr2=0x7ffbec198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbec1002e0 0x7ffbec198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 -- 172.21.15.27:0/284881277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbec19d3d0 con 0x7ffbec19c8e0 2024-04-25T01:36:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbec1002e0 0x7ffbec198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbeb7fe640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec19c8e0 0x7ffbec19cd90 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400cad0 tx=0x7ffbe400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbe8ff9640 1 -- 172.21.15.27:0/284881277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe400ae20 con 0x7ffbec19c8e0 2024-04-25T01:36:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbe8ff9640 1 -- 172.21.15.27:0/284881277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbe4005020 con 0x7ffbec19c8e0 2024-04-25T01:36:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbf16da640 1 -- 172.21.15.27:0/284881277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbec19d6c0 con 0x7ffbec19c8e0 2024-04-25T01:36:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbe8ff9640 1 -- 172.21.15.27:0/284881277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe400f6f0 con 0x7ffbec19c8e0 2024-04-25T01:36:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.043+0000 7ffbf16da640 1 -- 172.21.15.27:0/284881277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbec1a4eb0 con 0x7ffbec19c8e0 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.046+0000 7ffbe8ff9640 1 -- 172.21.15.27:0/284881277 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ffbe40051d0 con 0x7ffbec19c8e0 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.046+0000 7ffbf43ef740 1 Processor -- start 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.046+0000 7ffbf43ef740 1 -- start start 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.046+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eef980 0x55efd3eedcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.046+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eee290 0x55efd3eee710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3efb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55efd3e522c0 con 0x55efd3eee290 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55efd3dfa910 con 0x55efd3ef9010 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55efd3e4e280 con 0x55efd3eef980 2024-04-25T01:36:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbe8ff9640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffbd80a2d00 0x7ffbd80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eee290 0x55efd3eee710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eee290 0x55efd3eee710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51222/0 (socket says 172.21.15.27:51222) 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 -- 172.21.15.27:0/3845122475 learned_addr learned my addr 172.21.15.27:0/3845122475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeb7fe640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffbd80a2d00 0x7ffbd80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eef980 0x55efd3eedcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbe8ff9640 1 -- 172.21.15.27:0/284881277 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7ffbe40cfbd0 con 0x7ffbec19c8e0 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3efb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 -- 172.21.15.27:0/3845122475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eef980 msgr2=0x55efd3eedcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eef980 0x55efd3eedcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 -- 172.21.15.27:0/3845122475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 msgr2=0x55efd3efb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3efb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 -- 172.21.15.27:0/3845122475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efd3e0bfb0 con 0x55efd3eee290 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3efb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeb7fe640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffbd80a2d00 0x7ffbd80a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffbe409ba00 tx=0x7ffbe406ac30 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eee290 0x55efd3eee710 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009900 tx=0x7ffbdc004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26c4261d945c87a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.047+0000 7ffbe17fa640 1 -- 172.21.15.27:0/3845122475 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc03f070 con 0x55efd3eee290 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbe17fa640 1 -- 172.21.15.27:0/3845122475 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc005500 con 0x55efd3eee290 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbe17fa640 1 -- 172.21.15.27:0/3845122475 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc035070 con 0x55efd3eee290 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 -- 172.21.15.27:0/3845122475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eee290 msgr2=0x55efd3eee710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eee290 0x55efd3eee710 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009900 tx=0x7ffbdc004200 comp rx=0 tx=0).stop 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 -- 172.21.15.27:0/3845122475 shutdown_connections 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3efb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eee290 0x55efd3eee710 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/3845122475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eef980 0x55efd3eedcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 -- 172.21.15.27:0/3845122475 >> 172.21.15.27:0/3845122475 conn(0x55efd3eb6950 msgr2=0x55efd3eea7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 -- 172.21.15.27:0/3845122475 shutdown_connections 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.048+0000 7ffbf43ef740 1 -- 172.21.15.27:0/3845122475 wait complete. 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 1 -- start start 2024-04-25T01:36:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 1 Processor -- start 2024-04-25T01:36:19.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3eee290 0x55efd3ef5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3ef9010 0x55efd3ef69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55efd3efbe40 con 0x55efd3eef980 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55efd3e0de90 con 0x55efd3eee290 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55efd3efbfc0 con 0x55efd3ef9010 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbf43ef740 10 client.0 _ll_register_callbacks cb 0x55efd3e49970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51230/0 (socket says 172.21.15.27:51230) 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbeaffd640 1 -- 172.21.15.27:0/622942172 learned_addr learned my addr 172.21.15.27:0/622942172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3eee290 0x55efd3ef5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3ef9010 0x55efd3ef69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbeaffd640 1 -- 172.21.15.27:0/622942172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3ef9010 msgr2=0x55efd3ef69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3ef9010 0x55efd3ef69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbeaffd640 1 -- 172.21.15.27:0/622942172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3eee290 msgr2=0x55efd3ef5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3eee290 0x55efd3ef5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbeaffd640 1 -- 172.21.15.27:0/622942172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efd3c2e7e0 con 0x55efd3eef980 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.049+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3ef9010 0x55efd3ef69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6470 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc02a930 tx=0x7ffbdc004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10faf7e95592877 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbd27fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbd27fc640 1 -- 172.21.15.27:0/622942172 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc03f070 con 0x55efd3eef980 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbd27fc640 1 -- 172.21.15.27:0/622942172 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7ffbdc0315b0 con 0x55efd3eef980 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 1 -- 172.21.15.27:0/622942172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbdc005500 con 0x55efd3eef980 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbd27fc640 1 -- 172.21.15.27:0/622942172 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffbdc035040 con 0x55efd3eef980 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 10 client.18003 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 1 -- 172.21.15.27:0/622942172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55efd3c8ef70 con 0x55efd3eef980 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbd27fc640 1 -- 172.21.15.27:0/622942172 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc005500 con 0x55efd3eef980 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbd27fc640 1 -- 172.21.15.27:0/622942172 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ffbdc003c60 con 0x55efd3eef980 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 10 client.18003 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 10 client.18003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 20 client.18003 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:19.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55efd3f08ed0 0x55efd3f0b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 4 client.18003 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.050+0000 7ffbf43ef740 1 -- 172.21.15.27:0/622942172 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efd3e265a0 con 0x55efd3f08ed0 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.051+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55efd3f08ed0 0x55efd3f0b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.051+0000 7ffbd27fc640 10 client.18003 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.051+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55efd3f08ed0 0x55efd3f0b2c0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25cd753c1062416b server_cookie=dc95c05864ba55d6 in_seq=0 out_seq=0 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbd27fc640 1 -- 172.21.15.27:0/622942172 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55efd3e265a0 con 0x55efd3f08ed0 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbd27fc640 10 client.18003 handle_command_reply: tid=0 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 client.18003 shutdown 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 -- 172.21.15.27:0/622942172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 msgr2=0x55efd3ef6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6470 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc02a930 tx=0x7ffbdc004bd0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 -- 172.21.15.27:0/622942172 shutdown_connections 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55efd3f08ed0 0x55efd3f0b2c0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3ef9010 0x55efd3ef69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6470 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/622942172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3eee290 0x55efd3ef5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbd27fc640 0 client.18003 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:19.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 -- 172.21.15.27:0/622942172 >> 172.21.15.27:0/622942172 conn(0x55efd3eb6950 msgr2=0x55efd3ef0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 -- 172.21.15.27:0/622942172 shutdown_connections 2024-04-25T01:36:19.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 1 -- 172.21.15.27:0/622942172 wait complete. 2024-04-25T01:36:19.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.052+0000 7ffbf43ef740 20 client.18003 trim_cache size 0 max 16384 2024-04-25T01:36:19.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.070+0000 7ffbf43ef740 1 Processor -- start 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.070+0000 7ffbf43ef740 1 -- start start 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.070+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 0x55efd3ee9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.070+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.070+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3ef31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.070+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec1b1550 con 0x55efd3eef980 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.070+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec1aee20 con 0x55efd3ef9010 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.070+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec1acc90 con 0x55efd3eee290 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51232/0 (socket says 172.21.15.27:51232) 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 -- 172.21.15.27:0/1819719523 learned_addr learned my addr 172.21.15.27:0/1819719523 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 -- 172.21.15.27:0/1819719523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 msgr2=0x55efd3ee9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/1819719523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 0x55efd3ee9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 -- 172.21.15.27:0/1819719523 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 msgr2=0x55efd3ef31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/1819719523 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3ef31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 -- 172.21.15.27:0/1819719523 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efd3b26170 con 0x55efd3eef980 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/1819719523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6600 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009900 tx=0x7ffbdc005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76094d5be14831ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbe0ff9640 1 -- 172.21.15.27:0/1819719523 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc03f040 con 0x55efd3eef980 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbe0ff9640 1 -- 172.21.15.27:0/1819719523 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc004e80 con 0x55efd3eef980 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.071+0000 7ffbe0ff9640 1 -- 172.21.15.27:0/1819719523 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc031810 con 0x55efd3eef980 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- 172.21.15.27:0/1819719523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 msgr2=0x55efd3ef6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/1819719523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6600 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009900 tx=0x7ffbdc005c90 comp rx=0 tx=0).stop 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- 172.21.15.27:0/1819719523 shutdown_connections 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/1819719523 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3ef31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/1819719523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3ef6600 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/1819719523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 0x55efd3ee9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- 172.21.15.27:0/1819719523 >> 172.21.15.27:0/1819719523 conn(0x55efd3eb6950 msgr2=0x55efd3ef5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- 172.21.15.27:0/1819719523 shutdown_connections 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- 172.21.15.27:0/1819719523 wait complete. 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- start start 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 Processor -- start 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 0x55efd3f0e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3f0c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3f0cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec1ad2e0 con 0x55efd3eef980 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec1ae750 con 0x55efd3ef9010 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec1ad9b0 con 0x55efd3eee290 2024-04-25T01:36:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.072+0000 7ffbf43ef740 10 client.0 _ll_register_callbacks cb 0x7ffbd400e4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3f0c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3f0c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51236/0 (socket says 172.21.15.27:51236) 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 -- 172.21.15.27:0/2463007719 learned_addr learned my addr 172.21.15.27:0/2463007719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3f0cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 0x55efd3f0e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 -- 172.21.15.27:0/2463007719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 msgr2=0x55efd3f0e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 0x55efd3f0e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 -- 172.21.15.27:0/2463007719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 msgr2=0x55efd3f0cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3f0cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 -- 172.21.15.27:0/2463007719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efd3f0d350 con 0x55efd3eef980 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbeaffd640 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3f0c850 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc02a930 tx=0x7ffbdc004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359d0c18dc970485 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbd27fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:19.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbd27fc640 1 -- 172.21.15.27:0/2463007719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc047070 con 0x55efd3eef980 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbd27fc640 1 -- 172.21.15.27:0/2463007719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7ffbdc0315b0 con 0x55efd3eef980 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbf43ef740 1 -- 172.21.15.27:0/2463007719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbdc004e80 con 0x55efd3eef980 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbd27fc640 1 -- 172.21.15.27:0/2463007719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffbdc03f040 con 0x55efd3eef980 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbf43ef740 10 client.18009 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.073+0000 7ffbf43ef740 1 -- 172.21.15.27:0/2463007719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55efd3f6fef0 con 0x55efd3eef980 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbd27fc640 1 -- 172.21.15.27:0/2463007719 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc004e80 con 0x55efd3eef980 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbd27fc640 1 -- 172.21.15.27:0/2463007719 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ffbdc003990 con 0x55efd3eef980 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbf43ef740 10 client.18009 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbf43ef740 10 client.18009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbf43ef740 20 client.18009 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55efd3f70550 0x55efd3f72940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbf43ef740 4 client.18009 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbf43ef740 1 -- 172.21.15.27:0/2463007719 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55efd3e5f800 con 0x55efd3f70550 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55efd3f70550 0x55efd3f72940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbd27fc640 10 client.18009 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:19.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.074+0000 7ffbe3fff640 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55efd3f70550 0x55efd3f72940 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89f4f9173c491eb3 server_cookie=91b2397c50f1b47b in_seq=0 out_seq=0 2024-04-25T01:36:19.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbd27fc640 1 -- 172.21.15.27:0/2463007719 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55efd3e5f800 con 0x55efd3f70550 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbd27fc640 10 client.18009 handle_command_reply: tid=0 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 client.18009 shutdown 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 -- 172.21.15.27:0/2463007719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 msgr2=0x55efd3f0c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3f0c850 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc02a930 tx=0x7ffbdc004e00 comp rx=0 tx=0).stop 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 -- 172.21.15.27:0/2463007719 shutdown_connections 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x55efd3f70550 0x55efd3f72940 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55efd3ef9010 0x55efd3f0cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efd3eef980 0x55efd3f0c850 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbd27fc640 0 client.18009 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 --2- 172.21.15.27:0/2463007719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55efd3eee290 0x55efd3f0e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 -- 172.21.15.27:0/2463007719 >> 172.21.15.27:0/2463007719 conn(0x55efd3eb6950 msgr2=0x55efd3ef6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 -- 172.21.15.27:0/2463007719 shutdown_connections 2024-04-25T01:36:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 1 -- 172.21.15.27:0/2463007719 wait complete. 2024-04-25T01:36:19.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.076+0000 7ffbf43ef740 20 client.18009 trim_cache size 0 max 16384 2024-04-25T01:36:19.060 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:36:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:36:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "4bbdb0e7-33cb-4c88-a8d9-376e5027be7b", 2024-04-25T01:36:19.060 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:36:19.060 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:36:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 -- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffbd80a2d00 msgr2=0x7ffbd80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffbd80a2d00 0x7ffbd80a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffbe409ba00 tx=0x7ffbe406ac30 comp rx=0 tx=0).stop 2024-04-25T01:36:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 -- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec19c8e0 msgr2=0x7ffbec19cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec19c8e0 0x7ffbec19cd90 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400cad0 tx=0x7ffbe400cf90 comp rx=0 tx=0).stop 2024-04-25T01:36:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 -- 172.21.15.27:0/284881277 shutdown_connections 2024-04-25T01:36:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ffbd80a2d00 0x7ffbd80a51c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbec19c8e0 0x7ffbec19cd90 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbec106aa0 0x7ffbec198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 --2- 172.21.15.27:0/284881277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbec1002e0 0x7ffbec198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 -- 172.21.15.27:0/284881277 >> 172.21.15.27:0/284881277 conn(0x7ffbec0758d0 msgr2=0x7ffbec0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 -- 172.21.15.27:0/284881277 shutdown_connections 2024-04-25T01:36:19.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.077+0000 7ffbf16da640 1 -- 172.21.15.27:0/284881277 wait complete. 2024-04-25T01:36:19.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:36:19.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:36:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 -- 172.21.15.27:0/2897421470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 msgr2=0x7fc348107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 --2- 172.21.15.27:0/2897421470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 0x7fc348107b10 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fc338009930 tx=0x7fc33802a420 comp rx=0 tx=0).stop 2024-04-25T01:36:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 -- 172.21.15.27:0/2897421470 shutdown_connections 2024-04-25T01:36:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 --2- 172.21.15.27:0/2897421470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34806a780 0x7fc34810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 --2- 172.21.15.27:0/2897421470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc348069dc0 0x7fc34806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 --2- 172.21.15.27:0/2897421470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 0x7fc348107b10 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 -- 172.21.15.27:0/2897421470 >> 172.21.15.27:0/2897421470 conn(0x7fc3480776a0 msgr2=0x7fc348077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 -- 172.21.15.27:0/2897421470 shutdown_connections 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.232+0000 7fc34c928640 1 -- 172.21.15.27:0/2897421470 wait complete. 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.233+0000 7fc34c928640 1 Processor -- start 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.233+0000 7fc34c928640 1 -- start start 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.233+0000 7fc34c928640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc348069dc0 0x7fc348198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.233+0000 7fc34c928640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34806a780 0x7fc3481985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.233+0000 7fc34c928640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 0x7fc34819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.233+0000 7fc34c928640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc348111b00 con 0x7fc348107730 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.233+0000 7fc34c928640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc348111980 con 0x7fc348069dc0 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.233+0000 7fc34c928640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc348111c80 con 0x7fc34806a780 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 0x7fc34819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 0x7fc34819c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51254/0 (socket says 172.21.15.27:51254) 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 -- 172.21.15.27:0/2233548289 learned_addr learned my addr 172.21.15.27:0/2233548289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc34659c640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc348069dc0 0x7fc348198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc345d9b640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34806a780 0x7fc3481985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 -- 172.21.15.27:0/2233548289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34806a780 msgr2=0x7fc3481985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34806a780 0x7fc3481985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 -- 172.21.15.27:0/2233548289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc348069dc0 msgr2=0x7fc348198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc348069dc0 0x7fc348198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 -- 172.21.15.27:0/2233548289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34819d000 con 0x7fc348107730 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc345d9b640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34806a780 0x7fc3481985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc346d9d640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 0x7fc34819c8d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fc340009fd0 tx=0x7fc34000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc3377fe640 1 -- 172.21.15.27:0/2233548289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340017070 con 0x7fc348107730 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc34c928640 1 -- 172.21.15.27:0/2233548289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc34819d2f0 con 0x7fc348107730 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc34c928640 1 -- 172.21.15.27:0/2233548289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3481a4b80 con 0x7fc348107730 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc3377fe640 1 -- 172.21.15.27:0/2233548289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc340003e50 con 0x7fc348107730 2024-04-25T01:36:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.234+0000 7fc3377fe640 1 -- 172.21.15.27:0/2233548289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3400106c0 con 0x7fc348107730 2024-04-25T01:36:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.236+0000 7fc3377fe640 1 -- 172.21.15.27:0/2233548289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc340003970 con 0x7fc348107730 2024-04-25T01:36:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34f631740 1 Processor -- start 2024-04-25T01:36:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34f631740 1 -- start start 2024-04-25T01:36:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34f631740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4dde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34f631740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34f631740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4eae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34f631740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56508f43e6b0 con 0x56508f4dda40 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34f631740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56508f3e6880 con 0x56508f4de360 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34f631740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56508f43a670 con 0x56508f4e8a30 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc345d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4dde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc345d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4dde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51256/0 (socket says 172.21.15.27:51256) 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc345d9b640 1 -- 172.21.15.27:0/1014387243 learned_addr learned my addr 172.21.15.27:0/1014387243 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc34659c640 1 --2- 172.21.15.27:0/1014387243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc3377fe640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3300a2d00 0x7fc3300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc345d9b640 1 -- 172.21.15.27:0/1014387243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 msgr2=0x56508f4eae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc346d9d640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3300a2d00 0x7fc3300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc345d9b640 1 --2- 172.21.15.27:0/1014387243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4eae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc345d9b640 1 -- 172.21.15.27:0/1014387243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 msgr2=0x56508f4e84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.237+0000 7fc345d9b640 1 --2- 172.21.15.27:0/1014387243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.238+0000 7fc345d9b640 1 -- 172.21.15.27:0/1014387243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56508f21c7d0 con 0x56508f4dda40 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.238+0000 7fc3377fe640 1 -- 172.21.15.27:0/2233548289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fc3400cf620 con 0x7fc348107730 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.238+0000 7fc345d9b640 1 --2- 172.21.15.27:0/1014387243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4dde20 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009cb0 tx=0x7fc33c0106e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49301670c3635fb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.238+0000 7fc334ff9640 1 -- 172.21.15.27:0/1014387243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c016070 con 0x56508f4dda40 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.238+0000 7fc346d9d640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3300a2d00 0x7fc3300a51c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc3400ccf80 tx=0x7fc340090be0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.238+0000 7fc334ff9640 1 -- 172.21.15.27:0/1014387243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc33c005500 con 0x56508f4dda40 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.238+0000 7fc334ff9640 1 -- 172.21.15.27:0/1014387243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c012420 con 0x56508f4dda40 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- 172.21.15.27:0/1014387243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 msgr2=0x56508f4dde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 --2- 172.21.15.27:0/1014387243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4dde20 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009cb0 tx=0x7fc33c0106e0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- 172.21.15.27:0/1014387243 shutdown_connections 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 --2- 172.21.15.27:0/1014387243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4eae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 --2- 172.21.15.27:0/1014387243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 --2- 172.21.15.27:0/1014387243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4dde20 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- 172.21.15.27:0/1014387243 >> 172.21.15.27:0/1014387243 conn(0x56508f4d9d60 msgr2=0x56508f4da140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- 172.21.15.27:0/1014387243 shutdown_connections 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- 172.21.15.27:0/1014387243 wait complete. 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- start start 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 Processor -- start 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 0x56508f4e5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4e5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56508f4eb820 con 0x56508f4de360 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56508f3f9f50 con 0x56508f4dda40 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.239+0000 7fc34f631740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56508f4eb9a0 con 0x56508f4e8a30 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4e5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34f631740 10 client.0 _ll_register_callbacks cb 0x56508f435000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4e5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51266/0 (socket says 172.21.15.27:51266) 2024-04-25T01:36:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 -- 172.21.15.27:0/1668806469 learned_addr learned my addr 172.21.15.27:0/1668806469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc345d9b640 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc345d9b640 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 0x56508f4e5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 -- 172.21.15.27:0/1668806469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 msgr2=0x56508f4e63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 -- 172.21.15.27:0/1668806469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 msgr2=0x56508f4e5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 0x56508f4e5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 -- 172.21.15.27:0/1668806469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56508f1523f0 con 0x56508f4de360 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc345d9b640 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34659c640 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4e5eb0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc338031550 tx=0x7fc338032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88149f15e360957d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc32dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc32dffb640 1 -- 172.21.15.27:0/1668806469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338040070 con 0x56508f4de360 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc32dffb640 1 -- 172.21.15.27:0/1668806469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fc338004040 con 0x56508f4de360 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34f631740 1 -- 172.21.15.27:0/1668806469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc33c005500 con 0x56508f4de360 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc32dffb640 1 -- 172.21.15.27:0/1668806469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc33804c050 con 0x56508f4de360 2024-04-25T01:36:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34f631740 10 client.18021 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.240+0000 7fc34f631740 1 -- 172.21.15.27:0/1668806469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56508f1120f0 con 0x56508f4de360 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc32dffb640 1 -- 172.21.15.27:0/1668806469 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc33c005500 con 0x56508f4de360 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc32dffb640 1 -- 172.21.15.27:0/1668806469 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc3380048c0 con 0x56508f4de360 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc34f631740 10 client.18021 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc34f631740 10 client.18021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc34f631740 20 client.18021 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc34f631740 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56508f4f8d00 0x56508f4fb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc34f631740 4 client.18021 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc34f631740 1 -- 172.21.15.27:0/1668806469 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56508f44bbf0 con 0x56508f4f8d00 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc345d9b640 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56508f4f8d00 0x56508f4fb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc345d9b640 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56508f4f8d00 0x56508f4fb170 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe8b7938f389ee server_cookie=c74b7a273b56a130 in_seq=0 out_seq=0 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.241+0000 7fc32dffb640 10 client.18021 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc32dffb640 1 -- 172.21.15.27:0/1668806469 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56508f44bbf0 con 0x56508f4f8d00 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc32dffb640 10 client.18021 handle_command_reply: tid=0 2024-04-25T01:36:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 client.18021 shutdown 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 -- 172.21.15.27:0/1668806469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 msgr2=0x56508f4e5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4e5eb0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc338031550 tx=0x7fc338032eb0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 -- 172.21.15.27:0/1668806469 shutdown_connections 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56508f4f8d00 0x56508f4fb170 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc32dffb640 0 client.18021 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4e5eb0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 --2- 172.21.15.27:0/1668806469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 0x56508f4e5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 -- 172.21.15.27:0/1668806469 >> 172.21.15.27:0/1668806469 conn(0x56508f4d9d60 msgr2=0x56508f4d4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 -- 172.21.15.27:0/1668806469 shutdown_connections 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 1 -- 172.21.15.27:0/1668806469 wait complete. 2024-04-25T01:36:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.242+0000 7fc34f631740 20 client.18021 trim_cache size 0 max 16384 2024-04-25T01:36:19.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34f631740 1 Processor -- start 2024-04-25T01:36:19.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34f631740 1 -- start start 2024-04-25T01:36:19.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34f631740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 0x56508f4e80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34f631740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4f4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34f631740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4da140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34f631740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc3481af470 con 0x56508f4de360 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34f631740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc3481acd40 con 0x56508f4dda40 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34f631740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc3481aabb0 con 0x56508f4e8a30 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4f4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51276/0 (socket says 172.21.15.27:51276) 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34659c640 1 -- 172.21.15.27:0/3015597920 learned_addr learned my addr 172.21.15.27:0/3015597920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc345d9b640 1 --2- 172.21.15.27:0/3015597920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4da140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34659c640 1 -- 172.21.15.27:0/3015597920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 msgr2=0x56508f4da140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34659c640 1 --2- 172.21.15.27:0/3015597920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4da140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34659c640 1 -- 172.21.15.27:0/3015597920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 msgr2=0x56508f4e80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34659c640 1 --2- 172.21.15.27:0/3015597920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 0x56508f4e80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.260+0000 7fc34659c640 1 -- 172.21.15.27:0/3015597920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56508f4e3ec0 con 0x56508f4de360 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34659c640 1 --2- 172.21.15.27:0/3015597920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4f4710 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc33802a460 tx=0x7fc338009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fccf44f126482d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc32ffff640 1 -- 172.21.15.27:0/3015597920 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338055070 con 0x56508f4de360 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc32ffff640 1 -- 172.21.15.27:0/3015597920 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc338049040 con 0x56508f4de360 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc32ffff640 1 -- 172.21.15.27:0/3015597920 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338040070 con 0x56508f4de360 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 -- 172.21.15.27:0/3015597920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 msgr2=0x56508f4f4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 --2- 172.21.15.27:0/3015597920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4f4710 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc33802a460 tx=0x7fc338009610 comp rx=0 tx=0).stop 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 -- 172.21.15.27:0/3015597920 shutdown_connections 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 --2- 172.21.15.27:0/3015597920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4da140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 --2- 172.21.15.27:0/3015597920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4de360 0x56508f4f4710 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 --2- 172.21.15.27:0/3015597920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4dda40 0x56508f4e80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 -- 172.21.15.27:0/3015597920 >> 172.21.15.27:0/3015597920 conn(0x56508f4d9d60 msgr2=0x56508f4d4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 -- 172.21.15.27:0/3015597920 shutdown_connections 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.261+0000 7fc34f631740 1 -- 172.21.15.27:0/3015597920 wait complete. 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 1 -- start start 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 1 Processor -- start 2024-04-25T01:36:19.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc3481ab200 con 0x56508f4dda40 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc3481ac670 con 0x56508f4de360 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc3481ab8d0 con 0x56508f4e8a30 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34f631740 10 client.0 _ll_register_callbacks cb 0x7fc338031400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc345d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc345d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4e28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51292/0 (socket says 172.21.15.27:51292) 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc345d9b640 1 -- 172.21.15.27:0/3475041296 learned_addr learned my addr 172.21.15.27:0/3475041296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34659c640 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc345d9b640 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34659c640 1 -- 172.21.15.27:0/3475041296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 msgr2=0x56508f4e3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34659c640 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34659c640 1 -- 172.21.15.27:0/3475041296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 msgr2=0x56508f4e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34659c640 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4e28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc34659c640 1 -- 172.21.15.27:0/3475041296 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56508f55e550 con 0x56508f4de360 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.262+0000 7fc345d9b640 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4e28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc34659c640 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e2df0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x56508f4d9750 tx=0x7fc338031bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfc5a51f32a3fd92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc32dffb640 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc32dffb640 1 -- 172.21.15.27:0/3475041296 <== mon.2 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338055040 con 0x56508f4de360 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc34f631740 1 -- 172.21.15.27:0/3475041296 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc338049040 con 0x56508f4de360 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc32dffb640 1 -- 172.21.15.27:0/3475041296 <== mon.2 v2:172.21.15.139:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fc3380048c0 con 0x56508f4de360 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc32dffb640 1 -- 172.21.15.27:0/3475041296 <== mon.2 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc338040040 con 0x56508f4de360 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc34f631740 10 client.26879 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc34f631740 1 -- 172.21.15.27:0/3475041296 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56508f55f840 con 0x56508f4de360 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc32dffb640 1 -- 172.21.15.27:0/3475041296 <== mon.2 v2:172.21.15.139:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc338049040 con 0x56508f4de360 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc32dffb640 1 -- 172.21.15.27:0/3475041296 <== mon.2 v2:172.21.15.139:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc3380054c0 con 0x56508f4de360 2024-04-25T01:36:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc34f631740 10 client.26879 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc34f631740 10 client.26879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.263+0000 7fc34f631740 20 client.26879 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56508f55fec0 0x56508f5622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 4 client.26879 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 -- 172.21.15.27:0/3475041296 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56508f445f40 con 0x56508f55fec0 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc345d9b640 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56508f55fec0 0x56508f5622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc32dffb640 10 client.26879 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc345d9b640 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56508f55fec0 0x56508f5622e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=331bc0dbf9b70b51 server_cookie=d11ab5ef9cd58cb5 in_seq=0 out_seq=0 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc32dffb640 1 -- 172.21.15.27:0/3475041296 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56508f445f40 con 0x56508f55fec0 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc32dffb640 10 client.26879 handle_command_reply: tid=0 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 client.26879 shutdown 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 -- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 msgr2=0x56508f4e2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e2df0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x56508f4d9750 tx=0x7fc338031bb0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 -- 172.21.15.27:0/3475041296 shutdown_connections 2024-04-25T01:36:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56508f55fec0 0x56508f5622e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56508f4e8a30 0x56508f4e3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56508f4de360 0x56508f4e2df0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 --2- 172.21.15.27:0/3475041296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56508f4dda40 0x56508f4e28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc32dffb640 0 client.26879 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 -- 172.21.15.27:0/3475041296 >> 172.21.15.27:0/3475041296 conn(0x56508f4d9d60 msgr2=0x56508f4d4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.264+0000 7fc34f631740 1 -- 172.21.15.27:0/3475041296 shutdown_connections 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34f631740 1 -- 172.21.15.27:0/3475041296 wait complete. 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34f631740 20 client.26879 trim_cache size 0 max 16384 2024-04-25T01:36:19.248 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout: "4bbdb0e7-33cb-4c88-a8d9-376e5027be7b": { 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "4bbdb0e7-33cb-4c88-a8d9-376e5027be7b", 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:36:19.249 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:36:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 -- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3300a2d00 msgr2=0x7fc3300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3300a2d00 0x7fc3300a51c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc3400ccf80 tx=0x7fc340090be0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 -- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 msgr2=0x7fc34819c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 0x7fc34819c8d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fc340009fd0 tx=0x7fc34000b260 comp rx=0 tx=0).stop 2024-04-25T01:36:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 -- 172.21.15.27:0/2233548289 shutdown_connections 2024-04-25T01:36:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc3300a2d00 0x7fc3300a51c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc348107730 0x7fc34819c8d0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34806a780 0x7fc3481985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 --2- 172.21.15.27:0/2233548289 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc348069dc0 0x7fc348198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:19.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 -- 172.21.15.27:0/2233548289 >> 172.21.15.27:0/2233548289 conn(0x7fc3480776a0 msgr2=0x7fc3480fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:19.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 -- 172.21.15.27:0/2233548289 shutdown_connections 2024-04-25T01:36:19.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:19.265+0000 7fc34c928640 1 -- 172.21.15.27:0/2233548289 wait complete. 2024-04-25T01:36:19.252 INFO:tasks.cephfs.filesystem:scrub status for tag:4bbdb0e7-33cb-4c88-a8d9-376e5027be7b - {'path': '/', 'tag': '4bbdb0e7-33cb-4c88-a8d9-376e5027be7b', 'options': 'recursive,force'} 2024-04-25T01:36:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:20 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:36:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:20 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:20 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:36:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:20 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:36:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:20 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:20 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:20 smithi153 ceph-mon[32621]: pgmap v340: 129 pgs: 129 active+clean; 5.3 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T01:36:20.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:20 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:36:20.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:20 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:20 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:36:20.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:20 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:36:20.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:20 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:20 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:20 smithi139 ceph-mon[32760]: pgmap v340: 129 pgs: 129 active+clean; 5.3 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T01:36:20.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:20 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:36:20.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:20 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:20 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:36:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:20 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:36:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:20 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:20 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:20 smithi027 ceph-mon[29197]: pgmap v340: 129 pgs: 129 active+clean; 5.3 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T01:36:22.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:22 smithi153 ceph-mon[32621]: pgmap v341: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:22.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:22 smithi139 ceph-mon[32760]: pgmap v341: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:22.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:22 smithi027 ceph-mon[29197]: pgmap v341: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:24.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:24 smithi153 ceph-mon[32621]: pgmap v342: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:24.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:24 smithi139 ceph-mon[32760]: pgmap v342: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:24.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:24 smithi027 ceph-mon[29197]: pgmap v342: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:26.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:26 smithi153 ceph-mon[32621]: pgmap v343: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:26.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:26 smithi139 ceph-mon[32760]: pgmap v343: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:26.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:26 smithi027 ceph-mon[29197]: pgmap v343: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-25T01:36:28.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:28 smithi153 ceph-mon[32621]: pgmap v344: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-04-25T01:36:28.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:28 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:36:28.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:28 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:36:28.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:28 smithi139 ceph-mon[32760]: pgmap v344: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-04-25T01:36:28.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:28 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:36:28.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:28 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:36:28.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:28 smithi027 ceph-mon[29197]: pgmap v344: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 15 op/s 2024-04-25T01:36:28.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:28 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:36:28.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:28 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:36:30.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:30 smithi153 ceph-mon[32621]: pgmap v345: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:36:30.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:30 smithi139 ceph-mon[32760]: pgmap v345: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:36:30.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:30 smithi027 ceph-mon[29197]: pgmap v345: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T01:36:32.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:32 smithi153 ceph-mon[32621]: pgmap v346: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:36:32.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:32 smithi139 ceph-mon[32760]: pgmap v346: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:36:32.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:32 smithi027 ceph-mon[29197]: pgmap v346: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:36:34.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:34 smithi153 ceph-mon[32621]: pgmap v347: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:36:34.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:34 smithi139 ceph-mon[32760]: pgmap v347: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:36:34.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:34 smithi027 ceph-mon[29197]: pgmap v347: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:36:36.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:36 smithi153 ceph-mon[32621]: pgmap v348: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:36:36.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:36 smithi139 ceph-mon[32760]: pgmap v348: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:36:36.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:36 smithi027 ceph-mon[29197]: pgmap v348: 129 pgs: 129 active+clean; 5.4 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.8 KiB/s wr, 9 op/s 2024-04-25T01:36:38.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:38 smithi153 ceph-mon[32621]: pgmap v349: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:36:38.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:38 smithi139 ceph-mon[32760]: pgmap v349: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:36:38.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:38 smithi027 ceph-mon[29197]: pgmap v349: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:36:40.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:40 smithi153 ceph-mon[32621]: pgmap v350: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:40.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:40 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:40.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:40 smithi139 ceph-mon[32760]: pgmap v350: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:40.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:40 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:40.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:40 smithi027 ceph-mon[29197]: pgmap v350: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:40.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:40 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:42.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: pgmap v351: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:36:42.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:42.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:42.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:41 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.271 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.271 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.272 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.272 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.272 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.272 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.272 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: pgmap v351: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:36:42.272 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:42.272 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:42.272 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:41 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: pgmap v351: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:42.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:41 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:36:44.521 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:44 smithi153 ceph-mon[32621]: pgmap v352: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:44.538 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:44 smithi139 ceph-mon[32760]: pgmap v352: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:44.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:44 smithi027 ceph-mon[29197]: pgmap v352: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:46.521 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:46 smithi153 ceph-mon[32621]: pgmap v353: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:46.538 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:46 smithi139 ceph-mon[32760]: pgmap v353: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:46.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:46 smithi027 ceph-mon[29197]: pgmap v353: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:48.521 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:48 smithi153 ceph-mon[32621]: pgmap v354: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:36:48.538 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:48 smithi139 ceph-mon[32760]: pgmap v354: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:36:48.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:48 smithi027 ceph-mon[29197]: pgmap v354: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T01:36:49.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:36:49.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:36:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 -- 172.21.15.27:0/3388345195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 msgr2=0x7fd1d010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 --2- 172.21.15.27:0/3388345195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 0x7fd1d010b7c0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009900 tx=0x7fd1b802a510 comp rx=0 tx=0).stop 2024-04-25T01:36:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 -- 172.21.15.27:0/3388345195 shutdown_connections 2024-04-25T01:36:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 --2- 172.21.15.27:0/3388345195 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1d010bd00 0x7fd1d010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 --2- 172.21.15.27:0/3388345195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 0x7fd1d010b7c0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 --2- 172.21.15.27:0/3388345195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d01029c0 0x7fd1d0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 -- 172.21.15.27:0/3388345195 >> 172.21.15.27:0/3388345195 conn(0x7fd1d0077740 msgr2=0x7fd1d0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 -- 172.21.15.27:0/3388345195 shutdown_connections 2024-04-25T01:36:49.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 -- 172.21.15.27:0/3388345195 wait complete. 2024-04-25T01:36:49.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.396+0000 7fd1d529e640 1 Processor -- start 2024-04-25T01:36:49.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1d529e640 1 -- start start 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1d529e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1d01029c0 0x7fd1d0100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1d529e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 0x7fd1d0100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1d529e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d010bd00 0x7fd1d0101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1d529e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d0111b40 con 0x7fd1d01032e0 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1d529e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d01119c0 con 0x7fd1d01029c0 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1d529e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d0111cc0 con 0x7fd1d010bd00 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 0x7fd1d0100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 0x7fd1d0100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46828/0 (socket says 172.21.15.27:46828) 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1031235585 learned_addr learned my addr 172.21.15.27:0/1031235585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1d01029c0 0x7fd1d0100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d010bd00 0x7fd1d0101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1031235585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d010bd00 msgr2=0x7fd1d0101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d010bd00 0x7fd1d0101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1031235585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1d01029c0 msgr2=0x7fd1d0100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1d01029c0 0x7fd1d0100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.397+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1031235585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d01a65a0 con 0x7fd1d01032e0 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.398+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 0x7fd1d0100b50 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8006010 tx=0x7fd1b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.398+0000 7fd1c7fff640 1 -- 172.21.15.27:0/1031235585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b803f030 con 0x7fd1d01032e0 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.398+0000 7fd1d529e640 1 -- 172.21.15.27:0/1031235585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d01a6830 con 0x7fd1d01032e0 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.398+0000 7fd1c7fff640 1 -- 172.21.15.27:0/1031235585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1b8003ce0 con 0x7fd1d01032e0 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.398+0000 7fd1c7fff640 1 -- 172.21.15.27:0/1031235585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8035070 con 0x7fd1d01032e0 2024-04-25T01:36:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.398+0000 7fd1d529e640 1 -- 172.21.15.27:0/1031235585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d01a6d70 con 0x7fd1d01032e0 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.400+0000 7fd1c7fff640 1 -- 172.21.15.27:0/1031235585 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd1b8042030 con 0x7fd1d01032e0 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.400+0000 7fd1d7faa740 1 Processor -- start 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.400+0000 7fd1d7faa740 1 -- start start 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1c7fff640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1bc0a2d00 0x7fd1bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a81650a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a8165b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a8165b620 0x564a8165da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a815b56b0 con 0x564a8165b620 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564a8155d880 con 0x564a81650630 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a815b1670 con 0x564a81650f50 2024-04-25T01:36:49.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1bc0a2d00 0x7fd1bc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a8165b620 0x564a8165da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1c7fff640 1 -- 172.21.15.27:0/1031235585 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fd1b80eff20 con 0x7fd1d01032e0 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a8165b620 0x564a8165da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46836/0 (socket says 172.21.15.27:46836) 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 -- 172.21.15.27:0/3031526256 learned_addr learned my addr 172.21.15.27:0/3031526256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a81650a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a8165b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 -- 172.21.15.27:0/3031526256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 msgr2=0x564a8165b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a8165b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 -- 172.21.15.27:0/3031526256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 msgr2=0x564a81650a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a81650a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1cf7fe640 1 -- 172.21.15.27:0/3031526256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a813937d0 con 0x564a8165b620 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.401+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a81650a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.402+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a8165b620 0x564a8165da10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0013db0 tx=0x7fd1c0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc120bf4d98d966 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.402+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1bc0a2d00 0x7fd1bc0a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009fd0 tx=0x7fd1c0015000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.402+0000 7fd1c57fa640 1 -- 172.21.15.27:0/3031526256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c000dbb0 con 0x564a8165b620 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.402+0000 7fd1c57fa640 1 -- 172.21.15.27:0/3031526256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c0009b80 con 0x564a8165b620 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.402+0000 7fd1c57fa640 1 -- 172.21.15.27:0/3031526256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c001d650 con 0x564a8165b620 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- 172.21.15.27:0/3031526256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a8165b620 msgr2=0x564a8165da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a8165b620 0x564a8165da10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0013db0 tx=0x7fd1c0010740 comp rx=0 tx=0).stop 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- 172.21.15.27:0/3031526256 shutdown_connections 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a8165b620 0x564a8165da10 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a8165b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/3031526256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a81650a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- 172.21.15.27:0/3031526256 >> 172.21.15.27:0/3031526256 conn(0x564a8164c950 msgr2=0x564a8164cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- 172.21.15.27:0/3031526256 shutdown_connections 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- 172.21.15.27:0/3031526256 wait complete. 2024-04-25T01:36:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- start start 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 Processor -- start 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650630 0x564a81658600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81658b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 0x564a81659080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a8165e3b0 con 0x564a81650f50 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x564a81570f50 con 0x564a8165b620 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a8165e530 con 0x564a81650630 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.403+0000 7fd1d7faa740 10 client.0 _ll_register_callbacks cb 0x564a815ac000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81658b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81658b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46846/0 (socket says 172.21.15.27:46846) 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1303623978 learned_addr learned my addr 172.21.15.27:0/1303623978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 0x564a81659080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650630 0x564a81658600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1303623978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650630 msgr2=0x564a81658600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650630 0x564a81658600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1303623978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 msgr2=0x564a81659080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 0x564a81659080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1303623978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a812c93f0 con 0x564a81650f50 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81658b40 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80ed880 tx=0x7fd1b80bbee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4dda6b5b49e913 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1303623978 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8049030 con 0x564a81650f50 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1303623978 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fd1b80c0120 con 0x564a81650f50 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1303623978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c0009b80 con 0x564a81650f50 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1303623978 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd1b803bcb0 con 0x564a81650f50 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1d7faa740 10 client.18039 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.404+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1303623978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564a812890f0 con 0x564a81650f50 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1303623978 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c0009b80 con 0x564a81650f50 2024-04-25T01:36:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1303623978 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd1b808c420 con 0x564a81650f50 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1d7faa740 10 client.18039 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1d7faa740 10 client.18039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1d7faa740 20 client.18039 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a8166b930 0x564a8166dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1d7faa740 4 client.18039 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1303623978 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a815c2bf0 con 0x564a8166b930 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a8166b930 0x564a8166dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1b67fc640 10 client.18039 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.405+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a8166b930 0x564a8166dd20 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55c176e9c9c77c34 server_cookie=91ac19432caef755 in_seq=0 out_seq=0 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1303623978 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a815c2bf0 con 0x564a8166b930 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1b67fc640 10 client.18039 handle_command_reply: tid=0 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 client.18039 shutdown 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1303623978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 msgr2=0x564a81658b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81658b40 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80ed880 tx=0x7fd1b80bbee0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1303623978 shutdown_connections 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a8166b930 0x564a8166dd20 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 0x564a81659080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1b67fc640 0 client.18039 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81658b40 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1303623978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650630 0x564a81658600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1303623978 >> 172.21.15.27:0/1303623978 conn(0x564a8164c950 msgr2=0x564a81652750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1303623978 shutdown_connections 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.406+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1303623978 wait complete. 2024-04-25T01:36:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.407+0000 7fd1d7faa740 20 client.18039 trim_cache size 0 max 16384 2024-04-25T01:36:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.424+0000 7fd1d7faa740 1 Processor -- start 2024-04-25T01:36:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.424+0000 7fd1d7faa740 1 -- start start 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a8165ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81667270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8165b620 0x564a816524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d01b3740 con 0x564a81650f50 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d01b1010 con 0x564a81650630 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d01aee80 con 0x564a8165b620 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81667270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81667270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46858/0 (socket says 172.21.15.27:46858) 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1398108378 learned_addr learned my addr 172.21.15.27:0/1398108378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a8165ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8165b620 0x564a816524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1398108378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8165b620 msgr2=0x564a816524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8165b620 0x564a816524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1398108378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 msgr2=0x564a8165ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a8165ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 -- 172.21.15.27:0/1398108378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a8166e090 con 0x564a81650f50 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8165b620 0x564a816524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81667270 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80bc0a0 tx=0x7fd1b80bc010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bf0f82dc2e4df3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1c4ff9640 1 -- 172.21.15.27:0/1398108378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8085020 con 0x564a81650f50 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1c4ff9640 1 -- 172.21.15.27:0/1398108378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1b803b580 con 0x564a81650f50 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.425+0000 7fd1c4ff9640 1 -- 172.21.15.27:0/1398108378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8049050 con 0x564a81650f50 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1398108378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 msgr2=0x564a81667270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81667270 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80bc0a0 tx=0x7fd1b80bc010 comp rx=0 tx=0).stop 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1398108378 shutdown_connections 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8165b620 0x564a816524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650f50 0x564a81667270 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1398108378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a81650630 0x564a8165ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1398108378 >> 172.21.15.27:0/1398108378 conn(0x564a8164c950 msgr2=0x564a81651dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1398108378 shutdown_connections 2024-04-25T01:36:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1398108378 wait complete. 2024-04-25T01:36:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.426+0000 7fd1d7faa740 1 -- start start 2024-04-25T01:36:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1d7faa740 1 Processor -- start 2024-04-25T01:36:49.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650630 0x564a81655420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a81655960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1d7faa740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 0x564a81655ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d01af4d0 con 0x564a81650630 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d01b0940 con 0x564a8165b620 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1d7faa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d01afba0 con 0x564a81650f50 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1d7faa740 10 client.0 _ll_register_callbacks cb 0x7fd1b8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650630 0x564a81655420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650630 0x564a81655420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46862/0 (socket says 172.21.15.27:46862) 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 -- 172.21.15.27:0/1484185016 learned_addr learned my addr 172.21.15.27:0/1484185016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1cf7fe640 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 0x564a81655ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a81655960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 -- 172.21.15.27:0/1484185016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 msgr2=0x564a81655960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a81655960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 -- 172.21.15.27:0/1484185016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 msgr2=0x564a81655ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 0x564a81655ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 -- 172.21.15.27:0/1484185016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a816d0fc0 con 0x564a81650630 2024-04-25T01:36:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a81655960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1ceffd640 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650630 0x564a81655420 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800bdd0 tx=0x7fd1c80157f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675b8df1ba1d5bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.427+0000 7fd1b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1484185016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8015e60 con 0x564a81650630 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1484185016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fd1c80145c0 con 0x564a81650630 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1484185016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564a815bcf40 con 0x564a81650630 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1484185016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd1c802d050 con 0x564a81650630 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 10 client.18045 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1484185016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564a816d2250 con 0x564a81650630 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1484185016 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564a815bcf40 con 0x564a81650630 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1484185016 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd1c8024a50 con 0x564a81650630 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 10 client.18045 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 10 client.18045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 20 client.18045 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a8166b930 0x564a816d49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 4 client.18045 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1484185016 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564a8153a6f0 con 0x564a8166b930 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.428+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a8166b930 0x564a816d49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1b67fc640 10 client.18045 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1ce7fc640 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a8166b930 0x564a816d49f0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70e02299d2808a8b server_cookie=5d90bd719cc97f87 in_seq=0 out_seq=0 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1b67fc640 1 -- 172.21.15.27:0/1484185016 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564a8153a6f0 con 0x564a8166b930 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1b67fc640 10 client.18045 handle_command_reply: tid=0 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 client.18045 shutdown 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1484185016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650630 msgr2=0x564a81655420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650630 0x564a81655420 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800bdd0 tx=0x7fd1c80157f0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1484185016 shutdown_connections 2024-04-25T01:36:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x564a8166b930 0x564a816d49f0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x564a8165b620 0x564a81655ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a81650f50 0x564a81655960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1b67fc640 0 client.18045 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 --2- 172.21.15.27:0/1484185016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a81650630 0x564a81655420 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1484185016 >> 172.21.15.27:0/1484185016 conn(0x564a8164c950 msgr2=0x564a81658e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1484185016 shutdown_connections 2024-04-25T01:36:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 1 -- 172.21.15.27:0/1484185016 wait complete. 2024-04-25T01:36:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.429+0000 7fd1d7faa740 20 client.18045 trim_cache size 0 max 16384 2024-04-25T01:36:49.414 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:36:49.414 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:36:49.415 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:36:49.415 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:36:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 -- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1bc0a2d00 msgr2=0x7fd1bc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1bc0a2d00 0x7fd1bc0a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009fd0 tx=0x7fd1c0015000 comp rx=0 tx=0).stop 2024-04-25T01:36:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 -- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 msgr2=0x7fd1d0100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 0x7fd1d0100b50 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8006010 tx=0x7fd1b80042f0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 -- 172.21.15.27:0/1031235585 shutdown_connections 2024-04-25T01:36:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1d010bd00 0x7fd1d0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1bc0a2d00 0x7fd1bc0a51c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d01032e0 0x7fd1d0100b50 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 --2- 172.21.15.27:0/1031235585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1d01029c0 0x7fd1d0100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 -- 172.21.15.27:0/1031235585 >> 172.21.15.27:0/1031235585 conn(0x7fd1d0077740 msgr2=0x7fd1d00fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 -- 172.21.15.27:0/1031235585 shutdown_connections 2024-04-25T01:36:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.430+0000 7fd1d529e640 1 -- 172.21.15.27:0/1031235585 wait complete. 2024-04-25T01:36:49.417 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:36:49.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:36:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.543+0000 7fc7c3190640 1 -- 172.21.15.27:0/725243642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100620 msgr2=0x7fc7bc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.543+0000 7fc7c3190640 1 --2- 172.21.15.27:0/725243642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100620 0x7fc7bc100aa0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009930 tx=0x7fc7b002a420 comp rx=0 tx=0).stop 2024-04-25T01:36:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.543+0000 7fc7c3190640 1 -- 172.21.15.27:0/725243642 shutdown_connections 2024-04-25T01:36:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 --2- 172.21.15.27:0/725243642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7bc100fe0 0x7fc7bc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 --2- 172.21.15.27:0/725243642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100620 0x7fc7bc100aa0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 --2- 172.21.15.27:0/725243642 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7bc106630 0x7fc7bc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 -- 172.21.15.27:0/725243642 >> 172.21.15.27:0/725243642 conn(0x7fc7bc077740 msgr2=0x7fc7bc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 -- 172.21.15.27:0/725243642 shutdown_connections 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 -- 172.21.15.27:0/725243642 wait complete. 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 Processor -- start 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 -- start start 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7bc100620 0x7fc7bc198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100fe0 0x7fc7bc1993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7bc106630 0x7fc7bc1a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc110a00 con 0x7fc7bc100fe0 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc110880 con 0x7fc7bc100620 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7c3190640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc110b80 con 0x7fc7bc106630 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.544+0000 7fc7bbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100fe0 0x7fc7bc1993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7bbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100fe0 0x7fc7bc1993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46890/0 (socket says 172.21.15.27:46890) 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7bbfff640 1 -- 172.21.15.27:0/2731811137 learned_addr learned my addr 172.21.15.27:0/2731811137 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7c0f2c640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7bc100620 0x7fc7bc198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7c172d640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7bc106630 0x7fc7bc1a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7bbfff640 1 -- 172.21.15.27:0/2731811137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7bc106630 msgr2=0x7fc7bc1a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7bbfff640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7bc106630 0x7fc7bc1a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7bbfff640 1 -- 172.21.15.27:0/2731811137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7bc100620 msgr2=0x7fc7bc198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7bbfff640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7bc100620 0x7fc7bc198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7bbfff640 1 -- 172.21.15.27:0/2731811137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc1a4440 con 0x7fc7bc100fe0 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7bbfff640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100fe0 0x7fc7bc1993c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009230 tx=0x7fc7b0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7b9ffb640 1 -- 172.21.15.27:0/2731811137 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b003f030 con 0x7fc7bc100fe0 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7b9ffb640 1 -- 172.21.15.27:0/2731811137 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7b0003ad0 con 0x7fc7bc100fe0 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7b9ffb640 1 -- 172.21.15.27:0/2731811137 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0035070 con 0x7fc7bc100fe0 2024-04-25T01:36:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7bc1a4610 con 0x7fc7bc100fe0 2024-04-25T01:36:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.545+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7bc1a4a70 con 0x7fc7bc100fe0 2024-04-25T01:36:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.546+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7bc068560 con 0x7fc7bc100fe0 2024-04-25T01:36:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.547+0000 7fc7b9ffb640 1 -- 172.21.15.27:0/2731811137 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc7b0042030 con 0x7fc7bc100fe0 2024-04-25T01:36:49.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.548+0000 7fc7b9ffb640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc7a40a2d00 0x7fc7a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.548+0000 7fc7c0f2c640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc7a40a2d00 0x7fc7a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.548+0000 7fc7b9ffb640 1 -- 172.21.15.27:0/2731811137 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fc7b00efdd0 con 0x7fc7bc100fe0 2024-04-25T01:36:49.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.549+0000 7fc7c0f2c640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc7a40a2d00 0x7fc7a40a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4005fd0 tx=0x7fc7b4009500 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.553+0000 7fc7b9ffb640 1 -- 172.21.15.27:0/2731811137 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc7b00bbfb0 con 0x7fc7bc100fe0 2024-04-25T01:36:49.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.777+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7bc108bb0 con 0x7fc7bc100fe0 2024-04-25T01:36:49.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.778+0000 7fc7b9ffb640 1 -- 172.21.15.27:0/2731811137 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11882 (secure 0 0 0) 0x7fc7b00c02b0 con 0x7fc7bc100fe0 2024-04-25T01:36:49.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:36:49.753 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:28:52.435981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24505,24517,24521]},"id":1}]} 2024-04-25T01:36:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-25T01:36:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc7a40a2d00 msgr2=0x7fc7a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc7a40a2d00 0x7fc7a40a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4005fd0 tx=0x7fc7b4009500 comp rx=0 tx=0).stop 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100fe0 msgr2=0x7fc7bc1993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100fe0 0x7fc7bc1993c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009230 tx=0x7fc7b0004270 comp rx=0 tx=0).stop 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 shutdown_connections 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7bc106630 0x7fc7bc1a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc7a40a2d00 0x7fc7a40a51c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc100fe0 0x7fc7bc1993c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 --2- 172.21.15.27:0/2731811137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7bc100620 0x7fc7bc198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 >> 172.21.15.27:0/2731811137 conn(0x7fc7bc077740 msgr2=0x7fc7bc102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 shutdown_connections 2024-04-25T01:36:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.780+0000 7fc7c3190640 1 -- 172.21.15.27:0/2731811137 wait complete. 2024-04-25T01:36:49.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 -- 172.21.15.27:0/699780658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 msgr2=0x7fec48106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 --2- 172.21.15.27:0/699780658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 0x7fec48106a90 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fec40009930 tx=0x7fec4002a420 comp rx=0 tx=0).stop 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 -- 172.21.15.27:0/699780658 shutdown_connections 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 --2- 172.21.15.27:0/699780658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48100fd0 0x7fec4810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 --2- 172.21.15.27:0/699780658 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec48100610 0x7fec48100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 --2- 172.21.15.27:0/699780658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 0x7fec48106a90 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 -- 172.21.15.27:0/699780658 >> 172.21.15.27:0/699780658 conn(0x7fec48077740 msgr2=0x7fec48077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 -- 172.21.15.27:0/699780658 shutdown_connections 2024-04-25T01:36:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.897+0000 7fec4d77e640 1 -- 172.21.15.27:0/699780658 wait complete. 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec4d77e640 1 Processor -- start 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec4d77e640 1 -- start start 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec4d77e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec48100610 0x7fec48197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec4d77e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48100fd0 0x7fec48198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec4d77e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 0x7fec4819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec4d77e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fec48111b30 con 0x7fec481066b0 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec4d77e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fec481119b0 con 0x7fec48100610 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec4d77e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec48111cb0 con 0x7fec48100fd0 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.898+0000 7fec477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 0x7fec4819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 0x7fec4819c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46894/0 (socket says 172.21.15.27:46894) 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec477fe640 1 -- 172.21.15.27:0/43616315 learned_addr learned my addr 172.21.15.27:0/43616315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec467fc640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48100fd0 0x7fec48198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec46ffd640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec48100610 0x7fec48197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec477fe640 1 -- 172.21.15.27:0/43616315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48100fd0 msgr2=0x7fec48198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec477fe640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48100fd0 0x7fec48198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec477fe640 1 -- 172.21.15.27:0/43616315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec48100610 msgr2=0x7fec48197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec477fe640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec48100610 0x7fec48197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec477fe640 1 -- 172.21.15.27:0/43616315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4819cfe0 con 0x7fec481066b0 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec477fe640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 0x7fec4819c8b0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fec3800cad0 tx=0x7fec3800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec2ffff640 1 -- 172.21.15.27:0/43616315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3800ae20 con 0x7fec481066b0 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec2ffff640 1 -- 172.21.15.27:0/43616315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec38005020 con 0x7fec481066b0 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec2ffff640 1 -- 172.21.15.27:0/43616315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3800f6f0 con 0x7fec481066b0 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec4d77e640 1 -- 172.21.15.27:0/43616315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4819d2d0 con 0x7fec481066b0 2024-04-25T01:36:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.899+0000 7fec4d77e640 1 -- 172.21.15.27:0/43616315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec481a4b60 con 0x7fec481066b0 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec2ffff640 1 -- 172.21.15.27:0/43616315 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fec380051d0 con 0x7fec481066b0 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec5056f740 1 Processor -- start 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec5056f740 1 -- start start 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec5056f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132580 0x563e03132960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec5056f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec5056f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 0x563e0313f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec5056f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563e03097670 con 0x563e03132ea0 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec5056f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563e0303f970 con 0x563e0313d570 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec5056f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563e03093630 con 0x563e03132580 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec2ffff640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fec300a2d00 0x7fec300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec46ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec46ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46908/0 (socket says 172.21.15.27:46908) 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec46ffd640 1 -- 172.21.15.27:0/1374045910 learned_addr learned my addr 172.21.15.27:0/1374045910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec467fc640 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 0x563e0313f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec477fe640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fec300a2d00 0x7fec300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.902+0000 7fec467fc640 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132580 0x563e03132960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec2ffff640 1 -- 172.21.15.27:0/43616315 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fec380cf900 con 0x7fec481066b0 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec46ffd640 1 -- 172.21.15.27:0/1374045910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132580 msgr2=0x563e03132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec46ffd640 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132580 0x563e03132960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec46ffd640 1 -- 172.21.15.27:0/1374045910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 msgr2=0x563e0313f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec46ffd640 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 0x563e0313f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec46ffd640 1 -- 172.21.15.27:0/1374045910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e03063400 con 0x563e03132ea0 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec467fc640 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132580 0x563e03132960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec46ffd640 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313d030 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fec40000c00 tx=0x7fec40032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f096fa76d025c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec477fe640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fec300a2d00 0x7fec300a51c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fec380ccd10 tx=0x7fec3806b890 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec2d7fa640 1 -- 172.21.15.27:0/1374045910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4003f070 con 0x563e03132ea0 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec2d7fa640 1 -- 172.21.15.27:0/1374045910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec40031870 con 0x563e03132ea0 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.903+0000 7fec2d7fa640 1 -- 172.21.15.27:0/1374045910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec40035070 con 0x563e03132ea0 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- 172.21.15.27:0/1374045910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 msgr2=0x563e0313d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313d030 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fec40000c00 tx=0x7fec40032d60 comp rx=0 tx=0).stop 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- 172.21.15.27:0/1374045910 shutdown_connections 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 0x563e0313f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313d030 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 --2- 172.21.15.27:0/1374045910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132580 0x563e03132960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- 172.21.15.27:0/1374045910 >> 172.21.15.27:0/1374045910 conn(0x563e0312e8a0 msgr2=0x563e0312ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- 172.21.15.27:0/1374045910 shutdown_connections 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- 172.21.15.27:0/1374045910 wait complete. 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- start start 2024-04-25T01:36:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 Processor -- start 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132580 0x563e0313ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e0313a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563e03140300 con 0x563e03132ea0 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563e03053050 con 0x563e03132580 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563e03140480 con 0x563e0313d570 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec5056f740 10 client.0 _ll_register_callbacks cb 0x563e02daabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.904+0000 7fec46ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec46ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46914/0 (socket says 172.21.15.27:46914) 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec46ffd640 1 -- 172.21.15.27:0/4204436089 learned_addr learned my addr 172.21.15.27:0/4204436089 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec467fc640 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132580 0x563e0313ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec467fc640 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e0313a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec46ffd640 1 -- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 msgr2=0x563e0313a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec46ffd640 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e0313a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec46ffd640 1 -- 172.21.15.27:0/4204436089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132580 msgr2=0x563e0313ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec46ffd640 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132580 0x563e0313ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec46ffd640 1 -- 172.21.15.27:0/4204436089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e03024dc0 con 0x563e03132ea0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec467fc640 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e0313a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec46ffd640 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313a1a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fec40006010 tx=0x7fec40005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b29e6884b16e7a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec2aff5640 1 -- 172.21.15.27:0/4204436089 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4003f040 con 0x563e03132ea0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec2aff5640 1 -- 172.21.15.27:0/4204436089 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fec40003800 con 0x563e03132ea0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec5056f740 1 -- 172.21.15.27:0/4204436089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec40031870 con 0x563e03132ea0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec2aff5640 1 -- 172.21.15.27:0/4204436089 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec40056050 con 0x563e03132ea0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec5056f740 10 client.18063 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec5056f740 1 -- 172.21.15.27:0/4204436089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563e02dab3f0 con 0x563e03132ea0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.905+0000 7fec2aff5640 1 -- 172.21.15.27:0/4204436089 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec40031870 con 0x563e03132ea0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec2aff5640 1 -- 172.21.15.27:0/4204436089 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fec4003b9c0 con 0x563e03132ea0 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec5056f740 10 client.18063 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec5056f740 10 client.18063 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec5056f740 20 client.18063 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec5056f740 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x563e0314d370 0x563e0314f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec5056f740 4 client.18063 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec5056f740 1 -- 172.21.15.27:0/4204436089 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e030a4bb0 con 0x563e0314d370 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec467fc640 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x563e0314d370 0x563e0314f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec467fc640 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x563e0314d370 0x563e0314f760 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5453d0d244177172 server_cookie=3363560d2fc03ec4 in_seq=0 out_seq=0 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.906+0000 7fec2aff5640 10 client.18063 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec2aff5640 1 -- 172.21.15.27:0/4204436089 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e030a4bb0 con 0x563e0314d370 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec2aff5640 10 client.18063 handle_command_reply: tid=0 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec5056f740 1 client.18063 shutdown 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec5056f740 1 -- 172.21.15.27:0/4204436089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 msgr2=0x563e0313a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec5056f740 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313a1a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fec40006010 tx=0x7fec40005750 comp rx=0 tx=0).stop 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec5056f740 1 -- 172.21.15.27:0/4204436089 shutdown_connections 2024-04-25T01:36:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec5056f740 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x563e0314d370 0x563e0314f760 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec5056f740 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e0313a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec5056f740 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132ea0 0x563e0313a1a0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec2aff5640 0 client.18063 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:36:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.907+0000 7fec5056f740 1 --2- 172.21.15.27:0/4204436089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132580 0x563e0313ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.908+0000 7fec5056f740 1 -- 172.21.15.27:0/4204436089 >> 172.21.15.27:0/4204436089 conn(0x563e0312e8a0 msgr2=0x563e0312db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.908+0000 7fec5056f740 1 -- 172.21.15.27:0/4204436089 shutdown_connections 2024-04-25T01:36:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.908+0000 7fec5056f740 1 -- 172.21.15.27:0/4204436089 wait complete. 2024-04-25T01:36:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.908+0000 7fec5056f740 20 client.18063 trim_cache size 0 max 16384 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec5056f740 1 Processor -- start 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec5056f740 1 -- start start 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec5056f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031491f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec5056f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132ea0 0x563e03154d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec5056f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 0x563e031552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec5056f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fec481b15d0 con 0x563e03132580 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec5056f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fec481aeea0 con 0x563e0313d570 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec5056f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec481acd10 con 0x563e03132ea0 2024-04-25T01:36:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031491f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46924/0 (socket says 172.21.15.27:46924) 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec467fc640 1 -- 172.21.15.27:0/785145488 learned_addr learned my addr 172.21.15.27:0/785145488 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec46ffd640 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132ea0 0x563e03154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec467fc640 1 -- 172.21.15.27:0/785145488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132ea0 msgr2=0x563e03154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec467fc640 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132ea0 0x563e03154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec467fc640 1 -- 172.21.15.27:0/785145488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 msgr2=0x563e031552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.921+0000 7fec467fc640 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 0x563e031552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec467fc640 1 -- 172.21.15.27:0/785145488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e02e757d0 con 0x563e03132580 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec46ffd640 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132ea0 0x563e03154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec467fc640 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031491f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fec3c000c00 tx=0x7fec3c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c1f4bf94507fea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec2cff9640 1 -- 172.21.15.27:0/785145488 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c005020 con 0x563e03132580 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec2cff9640 1 -- 172.21.15.27:0/785145488 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec3c0047a0 con 0x563e03132580 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec2cff9640 1 -- 172.21.15.27:0/785145488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c021320 con 0x563e03132580 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 -- 172.21.15.27:0/785145488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 msgr2=0x563e031491f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031491f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fec3c000c00 tx=0x7fec3c00b870 comp rx=0 tx=0).stop 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 -- 172.21.15.27:0/785145488 shutdown_connections 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e0313d570 0x563e031552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e03132ea0 0x563e03154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 --2- 172.21.15.27:0/785145488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031491f0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 -- 172.21.15.27:0/785145488 >> 172.21.15.27:0/785145488 conn(0x563e0312e8a0 msgr2=0x563e0312db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 -- 172.21.15.27:0/785145488 shutdown_connections 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.922+0000 7fec5056f740 1 -- 172.21.15.27:0/785145488 wait complete. 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 1 -- start start 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 1 Processor -- start 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031459f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132ea0 0x563e03142e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e03143360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fec481ad360 con 0x563e03132580 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fec481ae7d0 con 0x563e03132ea0 2024-04-25T01:36:49.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec481ada30 con 0x563e0313d570 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec5056f740 10 client.0 _ll_register_callbacks cb 0x7fec3c00b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031459f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46926/0 (socket says 172.21.15.27:46926) 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 -- 172.21.15.27:0/1304268637 learned_addr learned my addr 172.21.15.27:0/1304268637 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec46ffd640 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132ea0 0x563e03142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e03143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 -- 172.21.15.27:0/1304268637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 msgr2=0x563e03143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e03143360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 -- 172.21.15.27:0/1304268637 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132ea0 msgr2=0x563e03142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132ea0 0x563e03142e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.923+0000 7fec467fc640 1 -- 172.21.15.27:0/1304268637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e03143aa0 con 0x563e03132580 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec467fc640 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031459f0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x563e03140590 tx=0x7fec3c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4081352bc38fe9d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec2aff5640 1 -- 172.21.15.27:0/1304268637 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c034060 con 0x563e03132580 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec2aff5640 1 -- 172.21.15.27:0/1304268637 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fec3c005430 con 0x563e03132580 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 1 -- 172.21.15.27:0/1304268637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec3c0047a0 con 0x563e03132580 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec2aff5640 1 -- 172.21.15.27:0/1304268637 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec3c0210c0 con 0x563e03132580 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 10 client.18069 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 1 -- 172.21.15.27:0/1304268637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563e031b34a0 con 0x563e03132580 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec2aff5640 1 -- 172.21.15.27:0/1304268637 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec3c0047a0 con 0x563e03132580 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec2aff5640 1 -- 172.21.15.27:0/1304268637 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fec3c029970 con 0x563e03132580 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 10 client.18069 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 10 client.18069 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 20 client.18069 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x563e031b3ae0 0x563e031b5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 4 client.18069 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.924+0000 7fec5056f740 1 -- 172.21.15.27:0/1304268637 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e0309ef00 con 0x563e031b3ae0 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec46ffd640 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x563e031b3ae0 0x563e031b5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec2aff5640 10 client.18069 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec46ffd640 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x563e031b3ae0 0x563e031b5f00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8050cef77146878f server_cookie=e704707f8cdd8275 in_seq=0 out_seq=0 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec2aff5640 1 -- 172.21.15.27:0/1304268637 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e0309ef00 con 0x563e031b3ae0 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec2aff5640 10 client.18069 handle_command_reply: tid=0 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 client.18069 shutdown 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 -- 172.21.15.27:0/1304268637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 msgr2=0x563e031459f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031459f0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x563e03140590 tx=0x7fec3c00b390 comp rx=0 tx=0).stop 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 -- 172.21.15.27:0/1304268637 shutdown_connections 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x563e031b3ae0 0x563e031b5f00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563e0313d570 0x563e03143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563e03132ea0 0x563e03142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 --2- 172.21.15.27:0/1304268637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e03132580 0x563e031459f0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec2aff5640 0 client.18069 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:36:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 -- 172.21.15.27:0/1304268637 >> 172.21.15.27:0/1304268637 conn(0x563e0312e8a0 msgr2=0x563e031481e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 -- 172.21.15.27:0/1304268637 shutdown_connections 2024-04-25T01:36:49.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.925+0000 7fec5056f740 1 -- 172.21.15.27:0/1304268637 wait complete. 2024-04-25T01:36:49.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec5056f740 20 client.18069 trim_cache size 0 max 16384 2024-04-25T01:36:49.908 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 -- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fec300a2d00 msgr2=0x7fec300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fec300a2d00 0x7fec300a51c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fec380ccd10 tx=0x7fec3806b890 comp rx=0 tx=0).stop 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 -- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 msgr2=0x7fec4819c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 0x7fec4819c8b0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fec3800cad0 tx=0x7fec3800cf90 comp rx=0 tx=0).stop 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 -- 172.21.15.27:0/43616315 shutdown_connections 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fec300a2d00 0x7fec300a51c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec481066b0 0x7fec4819c8b0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec48100fd0 0x7fec48198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 --2- 172.21.15.27:0/43616315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec48100610 0x7fec48197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 -- 172.21.15.27:0/43616315 >> 172.21.15.27:0/43616315 conn(0x7fec48077740 msgr2=0x7fec480ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:49.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 -- 172.21.15.27:0/43616315 shutdown_connections 2024-04-25T01:36:49.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:49.926+0000 7fec4d77e640 1 -- 172.21.15.27:0/43616315 wait complete. 2024-04-25T01:36:49.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:36:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.040+0000 7f125ee4d640 1 -- 172.21.15.27:0/704066804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258107730 msgr2=0x7f1258107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.040+0000 7f125ee4d640 1 --2- 172.21.15.27:0/704066804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258107730 0x7f1258107b10 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1250009930 tx=0x7f125002a420 comp rx=0 tx=0).stop 2024-04-25T01:36:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 -- 172.21.15.27:0/704066804 shutdown_connections 2024-04-25T01:36:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 --2- 172.21.15.27:0/704066804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f125806a780 0x7f125810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 --2- 172.21.15.27:0/704066804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1258069dc0 0x7f125806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 --2- 172.21.15.27:0/704066804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258107730 0x7f1258107b10 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 -- 172.21.15.27:0/704066804 >> 172.21.15.27:0/704066804 conn(0x7f12580776a0 msgr2=0x7f1258077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 -- 172.21.15.27:0/704066804 shutdown_connections 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 -- 172.21.15.27:0/704066804 wait complete. 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 Processor -- start 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 -- start start 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258069dc0 0x7f12581980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f125806a780 0x7f12581985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1258107730 0x7f125819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1258111b40 con 0x7f1258069dc0 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f12581119c0 con 0x7f125806a780 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.041+0000 7f125ee4d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1258111cc0 con 0x7f1258107730 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258069dc0 0x7f12581980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258069dc0 0x7f12581980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46948/0 (socket says 172.21.15.27:46948) 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 -- 172.21.15.27:0/3757270424 learned_addr learned my addr 172.21.15.27:0/3757270424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f1257fff640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f125806a780 0x7f12581985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125d3ea640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1258107730 0x7f125819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 -- 172.21.15.27:0/3757270424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1258107730 msgr2=0x7f125819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1258107730 0x7f125819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 -- 172.21.15.27:0/3757270424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f125806a780 msgr2=0x7f12581985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f125806a780 0x7f12581985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 -- 172.21.15.27:0/3757270424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125819d090 con 0x7f1258069dc0 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f1257fff640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f125806a780 0x7f12581985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125cbe9640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258069dc0 0x7f12581980a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f1250009230 tx=0x7f1250004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f1255ffb640 1 -- 172.21.15.27:0/3757270424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125003f030 con 0x7f1258069dc0 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f1255ffb640 1 -- 172.21.15.27:0/3757270424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1250004440 con 0x7f1258069dc0 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125ee4d640 1 -- 172.21.15.27:0/3757270424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125819d320 con 0x7f1258069dc0 2024-04-25T01:36:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f1255ffb640 1 -- 172.21.15.27:0/3757270424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250035070 con 0x7f1258069dc0 2024-04-25T01:36:50.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.042+0000 7f125ee4d640 1 -- 172.21.15.27:0/3757270424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12581a4c10 con 0x7f1258069dc0 2024-04-25T01:36:50.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1255ffb640 1 -- 172.21.15.27:0/3757270424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1250042030 con 0x7f1258069dc0 2024-04-25T01:36:50.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1261c3a740 1 Processor -- start 2024-04-25T01:36:50.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1261c3a740 1 -- start start 2024-04-25T01:36:50.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecdf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaecee440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecee980 0x55bbaecf0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1261c3a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bbaec44670 con 0x55bbaecdfc60 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1261c3a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bbaebec970 con 0x55bbaecee980 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1261c3a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bbaec40630 con 0x55bbaecdf2b0 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f125cbe9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaecee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1255ffb640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12440a2d00 0x7f12440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f125cbe9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaecee440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46956/0 (socket says 172.21.15.27:46956) 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f125cbe9640 1 -- 172.21.15.27:0/260703957 learned_addr learned my addr 172.21.15.27:0/260703957 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f1257fff640 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecee980 0x55bbaecf0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f125d3ea640 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecdf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.045+0000 7f125d3ea640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12440a2d00 0x7f12440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f1255ffb640 1 -- 172.21.15.27:0/3757270424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f12500efc90 con 0x7f1258069dc0 2024-04-25T01:36:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f125cbe9640 1 -- 172.21.15.27:0/260703957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 msgr2=0x55bbaecdf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f125cbe9640 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecdf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f125cbe9640 1 -- 172.21.15.27:0/260703957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecee980 msgr2=0x55bbaecf0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f125cbe9640 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecee980 0x55bbaecf0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f125cbe9640 1 -- 172.21.15.27:0/260703957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbaec10400 con 0x55bbaecdfc60 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f125cbe9640 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaecee440 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f12500c4500 tx=0x7f1250009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b44440bd0a0623 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f125d3ea640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12440a2d00 0x7f12440a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1240009fd0 tx=0x7f1240005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f124effd640 1 -- 172.21.15.27:0/260703957 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250049070 con 0x55bbaecdfc60 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f124effd640 1 -- 172.21.15.27:0/260703957 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12500bc4c0 con 0x55bbaecdfc60 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.046+0000 7f124effd640 1 -- 172.21.15.27:0/260703957 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250031dd0 con 0x55bbaecdfc60 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 -- 172.21.15.27:0/260703957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 msgr2=0x55bbaecee440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaecee440 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f12500c4500 tx=0x7f1250009680 comp rx=0 tx=0).stop 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 -- 172.21.15.27:0/260703957 shutdown_connections 2024-04-25T01:36:50.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecee980 0x55bbaecf0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaecee440 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 --2- 172.21.15.27:0/260703957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecdf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 -- 172.21.15.27:0/260703957 >> 172.21.15.27:0/260703957 conn(0x55bbaecdb8a0 msgr2=0x55bbaecdbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 -- 172.21.15.27:0/260703957 shutdown_connections 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 -- 172.21.15.27:0/260703957 wait complete. 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 -- start start 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.047+0000 7f1261c3a740 1 Processor -- start 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 0x55bbaeceb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaecebb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1261c3a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bbaec44670 con 0x55bbaecee980 2024-04-25T01:36:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1261c3a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55bbaec00050 con 0x55bbaecdfc60 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1261c3a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bbaecf1500 con 0x55bbaecdf2b0 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1261c3a740 10 client.0 _ll_register_callbacks cb 0x55bbae957be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaecebb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaecebb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46964/0 (socket says 172.21.15.27:46964) 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 -- 172.21.15.27:0/1392596932 learned_addr learned my addr 172.21.15.27:0/1392596932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1257fff640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 0x55bbaeceb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125cbe9640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 -- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 msgr2=0x55bbaecee2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecee2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 -- 172.21.15.27:0/1392596932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 msgr2=0x55bbaeceb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 0x55bbaeceb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 -- 172.21.15.27:0/1392596932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbaebd1dc0 con 0x55bbaecee980 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125cbe9640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecee2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1257fff640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 0x55bbaeceb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f125d3ea640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaecebb80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f124000df40 tx=0x7f12400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0461d857587da9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1237fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1237fff640 1 -- 172.21.15.27:0/1392596932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240012e60 con 0x55bbaecee980 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1237fff640 1 -- 172.21.15.27:0/1392596932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f1240005020 con 0x55bbaecee980 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.048+0000 7f1261c3a740 1 -- 172.21.15.27:0/1392596932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12500bc4c0 con 0x55bbaecee980 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1237fff640 1 -- 172.21.15.27:0/1392596932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1240005220 con 0x55bbaecee980 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1261c3a740 10 client.18081 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1261c3a740 1 -- 172.21.15.27:0/1392596932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bbae9583f0 con 0x55bbaecee980 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1237fff640 1 -- 172.21.15.27:0/1392596932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12500bc4c0 con 0x55bbaecee980 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1237fff640 1 -- 172.21.15.27:0/1392596932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1240027030 con 0x55bbaecee980 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1261c3a740 10 client.18081 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1261c3a740 10 client.18081 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1261c3a740 20 client.18081 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bbaecfe760 0x55bbaed00b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1261c3a740 4 client.18081 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f1261c3a740 1 -- 172.21.15.27:0/1392596932 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbaec51bb0 con 0x55bbaecfe760 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.049+0000 7f125cbe9640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bbaecfe760 0x55bbaed00b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.050+0000 7f1237fff640 10 client.18081 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.050+0000 7f125cbe9640 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bbaecfe760 0x55bbaed00b50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fcd985fc6674c5fc server_cookie=2e6bbdfb0d5d0e78 in_seq=0 out_seq=0 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.050+0000 7f1237fff640 1 -- 172.21.15.27:0/1392596932 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bbaec51bb0 con 0x55bbaecfe760 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.050+0000 7f1237fff640 10 client.18081 handle_command_reply: tid=0 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.050+0000 7f1261c3a740 1 client.18081 shutdown 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 -- 172.21.15.27:0/1392596932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 msgr2=0x55bbaecebb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaecebb80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f124000df40 tx=0x7f12400046c0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 -- 172.21.15.27:0/1392596932 shutdown_connections 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bbaecfe760 0x55bbaed00b50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaecebb80 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 0x55bbaeceb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1237fff640 0 client.18081 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:36:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1392596932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecee2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 -- 172.21.15.27:0/1392596932 >> 172.21.15.27:0/1392596932 conn(0x55bbaecdb8a0 msgr2=0x55bbaece16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 -- 172.21.15.27:0/1392596932 shutdown_connections 2024-04-25T01:36:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 1 -- 172.21.15.27:0/1392596932 wait complete. 2024-04-25T01:36:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.051+0000 7f1261c3a740 20 client.18081 trim_cache size 0 max 16384 2024-04-25T01:36:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f1261c3a740 1 Processor -- start 2024-04-25T01:36:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f1261c3a740 1 -- start start 2024-04-25T01:36:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecfa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 0x55bbaeceaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaece1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f1261c3a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f12581af360 con 0x55bbaecee980 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f1261c3a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f12581acc30 con 0x55bbaecdfc60 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f1261c3a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12581aaaa0 con 0x55bbaecdf2b0 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f125d3ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaece1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f125d3ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaece1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46978/0 (socket says 172.21.15.27:46978) 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.064+0000 7f125d3ea640 1 -- 172.21.15.27:0/434731274 learned_addr learned my addr 172.21.15.27:0/434731274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f125d3ea640 1 -- 172.21.15.27:0/434731274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 msgr2=0x55bbaecfa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f125d3ea640 1 --2- 172.21.15.27:0/434731274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecfa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f125d3ea640 1 -- 172.21.15.27:0/434731274 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 msgr2=0x55bbaeceaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f125d3ea640 1 --2- 172.21.15.27:0/434731274 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 0x55bbaeceaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f125d3ea640 1 -- 172.21.15.27:0/434731274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbaea227d0 con 0x55bbaecee980 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f125d3ea640 1 --2- 172.21.15.27:0/434731274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaece1570 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f124000d7b0 tx=0x7f124000daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c131c2364eeb866 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f124e7fc640 1 -- 172.21.15.27:0/434731274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240012e60 con 0x55bbaecee980 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f124e7fc640 1 -- 172.21.15.27:0/434731274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1240012570 con 0x55bbaecee980 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f124e7fc640 1 -- 172.21.15.27:0/434731274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240004990 con 0x55bbaecee980 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f1261c3a740 1 -- 172.21.15.27:0/434731274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 msgr2=0x55bbaece1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f1261c3a740 1 --2- 172.21.15.27:0/434731274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaece1570 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f124000d7b0 tx=0x7f124000daf0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f1261c3a740 1 -- 172.21.15.27:0/434731274 shutdown_connections 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f1261c3a740 1 --2- 172.21.15.27:0/434731274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecee980 0x55bbaece1570 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f1261c3a740 1 --2- 172.21.15.27:0/434731274 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdfc60 0x55bbaeceaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f1261c3a740 1 --2- 172.21.15.27:0/434731274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecdf2b0 0x55bbaecfa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.065+0000 7f1261c3a740 1 -- 172.21.15.27:0/434731274 >> 172.21.15.27:0/434731274 conn(0x55bbaecdb8a0 msgr2=0x55bbaece0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 -- 172.21.15.27:0/434731274 shutdown_connections 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 -- 172.21.15.27:0/434731274 wait complete. 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 -- start start 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 Processor -- start 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdf2b0 0x55bbaed5d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaed5d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecee980 0x55bbaecf1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f12581ab0f0 con 0x55bbaecdfc60 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f12581ac560 con 0x55bbaecdf2b0 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12581ab7c0 con 0x55bbaecee980 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1261c3a740 10 client.0 _ll_register_callbacks cb 0x7f124000dbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1257fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaed5d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1257fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaed5d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46986/0 (socket says 172.21.15.27:46986) 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.066+0000 7f1257fff640 1 -- 172.21.15.27:0/1247689419 learned_addr learned my addr 172.21.15.27:0/1247689419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f125d3ea640 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecee980 0x55bbaecf1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f125cbe9640 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdf2b0 0x55bbaed5d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1257fff640 1 -- 172.21.15.27:0/1247689419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecee980 msgr2=0x55bbaecf1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1257fff640 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecee980 0x55bbaecf1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1257fff640 1 -- 172.21.15.27:0/1247689419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdf2b0 msgr2=0x55bbaed5d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1257fff640 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdf2b0 0x55bbaed5d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1257fff640 1 -- 172.21.15.27:0/1247689419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbaecf2250 con 0x55bbaecdfc60 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1257fff640 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaed5d580 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1248011ce0 tx=0x7f1248010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872ebf43ff65ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1237fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1237fff640 1 -- 172.21.15.27:0/1247689419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248004b20 con 0x55bbaecdfc60 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1237fff640 1 -- 172.21.15.27:0/1247689419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f1248004cc0 con 0x55bbaecdfc60 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1261c3a740 1 -- 172.21.15.27:0/1247689419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1240012570 con 0x55bbaecdfc60 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1237fff640 1 -- 172.21.15.27:0/1247689419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1248025050 con 0x55bbaecdfc60 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1261c3a740 10 client.18087 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1261c3a740 1 -- 172.21.15.27:0/1247689419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bbaed64870 con 0x55bbaecdfc60 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1237fff640 1 -- 172.21.15.27:0/1247689419 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1240012570 con 0x55bbaecdfc60 2024-04-25T01:36:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.067+0000 7f1237fff640 1 -- 172.21.15.27:0/1247689419 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f124801a070 con 0x55bbaecdfc60 2024-04-25T01:36:50.047 INFO:tasks.workunit.client.0.smithi027.stdout:561/600: open fd = 3 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:562/600: open fd = 3 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:563/600: open fd = 3 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:564/600: open fd = 3 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.048 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:565/600: open fd = 3 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:566/600: open fd = 3 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:567/600: open fd = 3 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:568/600: open fd = 3 2024-04-25T01:36:50.049 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:569/600: open fd = 3 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:570/600: open fd = 3 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:571/600: open fd = 3 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.050 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:572/600: open fd = 3 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:573/600: open fd = 3 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:574/600: open fd = 3 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:575/600: open fd = 3 2024-04-25T01:36:50.051 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:576/600: open fd = 3 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:577/600: open fd = 3 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:578/600: open fd = 3 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:579/600: open fd = 3 2024-04-25T01:36:50.052 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:580/600: open fd = 3 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:581/600: open fd = 3 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:582/600: open fd = 3 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.053 INFO:tasks.workunit.client.0.smithi027.stdout:583/600: open fd = 3 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:584/600: open fd = 3 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:585/600: open fd = 3 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:586/600: open fd = 3 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.054 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:587/600: open fd = 3 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:588/600: open fd = 3 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:589/600: open fd = 3 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:590/600: open fd = 3 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.055 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:591/600: open fd = 3 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:592/600: open fd = 3 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:593/600: open fd = 3 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:594/600: open fd = 3 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.056 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:595/600: open fd = 3 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:596/600: open fd = 3 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:597/600: open fd = 3 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:598/600: open fd = 3 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.057 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:599/600: open fd = 3 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:600/600: open fd = 3 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:write ret = 32 2024-04-25T01:36:50.058 INFO:tasks.workunit.client.0.smithi027.stdout:pread ret = 64 2024-04-25T01:36:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 10 client.18087 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 10 client.18087 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 20 client.18087 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bbaed65050 0x55bbaed67470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 4 client.18087 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 1 -- 172.21.15.27:0/1247689419 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bbaec4bf00 con 0x55bbaed65050 2024-04-25T01:36:50.059 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo OK 2024-04-25T01:36:50.060 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-04-25T01:36:50.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f125cbe9640 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bbaed65050 0x55bbaed67470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.061 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T01:36:50.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T01:36:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f125cbe9640 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bbaed65050 0x55bbaed67470 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=44e7e8a11714eed1 server_cookie=fd7ce2aa2e1a9af0 in_seq=0 out_seq=0 2024-04-25T01:36:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1237fff640 10 client.18087 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1237fff640 1 -- 172.21.15.27:0/1247689419 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bbaec4bf00 con 0x55bbaed65050 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1237fff640 10 client.18087 handle_command_reply: tid=0 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 1 client.18087 shutdown 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 1 -- 172.21.15.27:0/1247689419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 msgr2=0x55bbaed5d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.068+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaed5d580 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1248011ce0 tx=0x7f1248010800 comp rx=0 tx=0).stop 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 1 -- 172.21.15.27:0/1247689419 shutdown_connections 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55bbaed65050 0x55bbaed67470 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bbaecee980 0x55bbaecf1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bbaecdfc60 0x55bbaed5d580 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1237fff640 0 client.18087 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:36:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 1 --2- 172.21.15.27:0/1247689419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55bbaecdf2b0 0x55bbaed5d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 1 -- 172.21.15.27:0/1247689419 >> 172.21.15.27:0/1247689419 conn(0x55bbaecdb8a0 msgr2=0x55bbaeced6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 1 -- 172.21.15.27:0/1247689419 shutdown_connections 2024-04-25T01:36:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 1 -- 172.21.15.27:0/1247689419 wait complete. 2024-04-25T01:36:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f1261c3a740 20 client.18087 trim_cache size 0 max 16384 2024-04-25T01:36:50.064 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 -- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12440a2d00 msgr2=0x7f12440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12440a2d00 0x7f12440a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1240009fd0 tx=0x7f1240005f90 comp rx=0 tx=0).stop 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 -- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258069dc0 msgr2=0x7f12581980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258069dc0 0x7f12581980a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f1250009230 tx=0x7f1250004040 comp rx=0 tx=0).stop 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 -- 172.21.15.27:0/3757270424 shutdown_connections 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1258107730 0x7f125819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f125806a780 0x7f12581985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f12440a2d00 0x7f12440a51c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 --2- 172.21.15.27:0/3757270424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1258069dc0 0x7f12581980a0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 -- 172.21.15.27:0/3757270424 >> 172.21.15.27:0/3757270424 conn(0x7f12580776a0 msgr2=0x7f12580fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 -- 172.21.15.27:0/3757270424 shutdown_connections 2024-04-25T01:36:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.069+0000 7f125ee4d640 1 -- 172.21.15.27:0/3757270424 wait complete. 2024-04-25T01:36:50.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:36:50.151 INFO:tasks.workunit:Stopping ['fs/test_o_trunc.sh'] on client.0... 2024-04-25T01:36:50.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T01:36:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 -- 172.21.15.27:0/1973305485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 msgr2=0x7fe97810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 --2- 172.21.15.27:0/1973305485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 0x7fe97810d4e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe970009cd0 tx=0x7fe97002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 -- 172.21.15.27:0/1973305485 shutdown_connections 2024-04-25T01:36:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 --2- 172.21.15.27:0/1973305485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 0x7fe97810d4e0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 --2- 172.21.15.27:0/1973305485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe978100610 0x7fe978100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 --2- 172.21.15.27:0/1973305485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9781066b0 0x7fe978106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 -- 172.21.15.27:0/1973305485 >> 172.21.15.27:0/1973305485 conn(0x7fe978077740 msgr2=0x7fe978077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 -- 172.21.15.27:0/1973305485 shutdown_connections 2024-04-25T01:36:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 -- 172.21.15.27:0/1973305485 wait complete. 2024-04-25T01:36:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.236+0000 7fe97cb84640 1 Processor -- start 2024-04-25T01:36:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe97cb84640 1 -- start start 2024-04-25T01:36:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe97cb84640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe978100610 0x7fe978197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe97cb84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 0x7fe978198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe97cb84640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9781066b0 0x7fe97819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe97cb84640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe978110a30 con 0x7fe978100fd0 2024-04-25T01:36:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe97cb84640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe9781108b0 con 0x7fe978100610 2024-04-25T01:36:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe97cb84640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe978110bb0 con 0x7fe9781066b0 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe975d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 0x7fe978198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe975d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 0x7fe978198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47000/0 (socket says 172.21.15.27:47000) 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe975d9b640 1 -- 172.21.15.27:0/486615684 learned_addr learned my addr 172.21.15.27:0/486615684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe97659c640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe978100610 0x7fe978197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe976d9d640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9781066b0 0x7fe97819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe975d9b640 1 -- 172.21.15.27:0/486615684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9781066b0 msgr2=0x7fe97819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe975d9b640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9781066b0 0x7fe97819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe975d9b640 1 -- 172.21.15.27:0/486615684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe978100610 msgr2=0x7fe978197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe975d9b640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe978100610 0x7fe978197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe975d9b640 1 -- 172.21.15.27:0/486615684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97819cea0 con 0x7fe978100fd0 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.237+0000 7fe976d9d640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9781066b0 0x7fe97819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.238+0000 7fe975d9b640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 0x7fe978198440 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe96800c920 tx=0x7fe96800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.238+0000 7fe9677fe640 1 -- 172.21.15.27:0/486615684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968007c70 con 0x7fe978100fd0 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.238+0000 7fe9677fe640 1 -- 172.21.15.27:0/486615684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe968007e10 con 0x7fe978100fd0 2024-04-25T01:36:50.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.238+0000 7fe9677fe640 1 -- 172.21.15.27:0/486615684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968018890 con 0x7fe978100fd0 2024-04-25T01:36:50.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.238+0000 7fe97cb84640 1 -- 172.21.15.27:0/486615684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97819d190 con 0x7fe978100fd0 2024-04-25T01:36:50.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.238+0000 7fe97cb84640 1 -- 172.21.15.27:0/486615684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9781a4a20 con 0x7fe978100fd0 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.240+0000 7fe9677fe640 1 -- 172.21.15.27:0/486615684 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe968018a70 con 0x7fe978100fd0 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.240+0000 7fe97f968740 1 Processor -- start 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe97f968740 1 -- start start 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe97f968740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbab960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe97f968740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe97f968740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe97f968740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56499db10670 con 0x56499dbb6570 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe97f968740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56499dab8970 con 0x56499dbabea0 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe97f968740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56499db0c630 con 0x56499dbab580 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe975d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe975d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47012/0 (socket says 172.21.15.27:47012) 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe975d9b640 1 -- 172.21.15.27:0/790969597 learned_addr learned my addr 172.21.15.27:0/790969597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe976d9d640 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe97659c640 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe9677fe640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9600a2d00 0x7fe9600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe976d9d640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9600a2d00 0x7fe9600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe9677fe640 1 -- 172.21.15.27:0/486615684 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fe9680cf990 con 0x7fe978100fd0 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe975d9b640 1 -- 172.21.15.27:0/790969597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 msgr2=0x56499dbab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe975d9b640 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe975d9b640 1 -- 172.21.15.27:0/790969597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 msgr2=0x56499dbb6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.241+0000 7fe975d9b640 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.242+0000 7fe975d9b640 1 -- 172.21.15.27:0/790969597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56499dadc400 con 0x56499dbb6570 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.242+0000 7fe97659c640 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:50.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.242+0000 7fe975d9b640 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb8960 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe968098490 tx=0x7fe968000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e6aeb4a0159843 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.242+0000 7fe964ff9640 1 -- 172.21.15.27:0/790969597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96809c1c0 con 0x56499dbb6570 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.242+0000 7fe964ff9640 1 -- 172.21.15.27:0/790969597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9680d2ff0 con 0x56499dbb6570 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.242+0000 7fe964ff9640 1 -- 172.21.15.27:0/790969597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9680ccf70 con 0x56499dbb6570 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.242+0000 7fe976d9d640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9600a2d00 0x7fe9600a51c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe970006010 tx=0x7fe97002a8e0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- 172.21.15.27:0/790969597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 msgr2=0x56499dbb8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb8960 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe968098490 tx=0x7fe968000bc0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- 172.21.15.27:0/790969597 shutdown_connections 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb8960 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 --2- 172.21.15.27:0/790969597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- 172.21.15.27:0/790969597 >> 172.21.15.27:0/790969597 conn(0x56499dba78a0 msgr2=0x56499dba7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- 172.21.15.27:0/790969597 shutdown_connections 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- 172.21.15.27:0/790969597 wait complete. 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- start start 2024-04-25T01:36:50.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 Processor -- start 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbb5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56499dbb9300 con 0x56499dbb6570 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x56499dacc050 con 0x56499dbabea0 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.243+0000 7fe97f968740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56499dbb9480 con 0x56499dbab580 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe97f968740 10 client.0 _ll_register_callbacks cb 0x56499d823be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47014/0 (socket says 172.21.15.27:47014) 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 -- 172.21.15.27:0/831532920 learned_addr learned my addr 172.21.15.27:0/831532920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe97659c640 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbb5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe975d9b640 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 -- 172.21.15.27:0/831532920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 msgr2=0x56499dbb5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbb5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 -- 172.21.15.27:0/831532920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 msgr2=0x56499dbb3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 -- 172.21.15.27:0/831532920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499da9ddc0 con 0x56499dbb6570 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe976d9d640 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb3860 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fe970009690 tx=0x7fe9700095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a596874364bd1968 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe95dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe95dffb640 1 -- 172.21.15.27:0/831532920 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97004e070 con 0x56499dbb6570 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe97f968740 1 -- 172.21.15.27:0/831532920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9680d2ff0 con 0x56499dbb6570 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe95dffb640 1 -- 172.21.15.27:0/831532920 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fe970004510 con 0x56499dbb6570 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe95dffb640 1 -- 172.21.15.27:0/831532920 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe970004710 con 0x56499dbb6570 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.244+0000 7fe97f968740 10 client.18099 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe97f968740 1 -- 172.21.15.27:0/831532920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56499d8243f0 con 0x56499dbb6570 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe95dffb640 1 -- 172.21.15.27:0/831532920 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9680d2ff0 con 0x56499dbb6570 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe95dffb640 1 -- 172.21.15.27:0/831532920 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe970030070 con 0x56499dbb6570 2024-04-25T01:36:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe97f968740 10 client.18099 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe97f968740 10 client.18099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe97f968740 20 client.18099 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe97f968740 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56499dbc65e0 0x56499dbc89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe97f968740 4 client.18099 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe97f968740 1 -- 172.21.15.27:0/831532920 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56499db1dbb0 con 0x56499dbc65e0 2024-04-25T01:36:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.245+0000 7fe97659c640 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56499dbc65e0 0x56499dbc89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe95dffb640 10 client.18099 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97659c640 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56499dbc65e0 0x56499dbc89f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ce1f6ef5aaf4531 server_cookie=39958b302b78216 in_seq=0 out_seq=0 2024-04-25T01:36:50.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe95dffb640 1 -- 172.21.15.27:0/831532920 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56499db1dbb0 con 0x56499dbc65e0 2024-04-25T01:36:50.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe95dffb640 10 client.18099 handle_command_reply: tid=0 2024-04-25T01:36:50.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 client.18099 shutdown 2024-04-25T01:36:50.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 -- 172.21.15.27:0/831532920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 msgr2=0x56499dbb3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb3860 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fe970009690 tx=0x7fe9700095c0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 -- 172.21.15.27:0/831532920 shutdown_connections 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56499dbc65e0 0x56499dbc89f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbb3860 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbb3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe95dffb640 0 client.18099 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 --2- 172.21.15.27:0/831532920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbb5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.246+0000 7fe97f968740 1 -- 172.21.15.27:0/831532920 >> 172.21.15.27:0/831532920 conn(0x56499dba78a0 msgr2=0x56499dba6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.247+0000 7fe97f968740 1 -- 172.21.15.27:0/831532920 shutdown_connections 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.247+0000 7fe97f968740 1 -- 172.21.15.27:0/831532920 wait complete. 2024-04-25T01:36:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.247+0000 7fe97f968740 20 client.18099 trim_cache size 0 max 16384 2024-04-25T01:36:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97f968740 1 Processor -- start 2024-04-25T01:36:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97f968740 1 -- start start 2024-04-25T01:36:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97f968740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbc21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97f968740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97f968740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97f968740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe9781b1590 con 0x56499dbb6570 2024-04-25T01:36:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97f968740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe9781aee60 con 0x56499dbabea0 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97f968740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9781accd0 con 0x56499dbab580 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47028/0 (socket says 172.21.15.27:47028) 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 -- 172.21.15.27:0/3884461409 learned_addr learned my addr 172.21.15.27:0/3884461409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe97659c640 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbc21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe975d9b640 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 -- 172.21.15.27:0/3884461409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 msgr2=0x56499dbc21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbc21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 -- 172.21.15.27:0/3884461409 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 msgr2=0x56499dbce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 -- 172.21.15.27:0/3884461409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56499d8ee7d0 con 0x56499dbb6570 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.261+0000 7fe976d9d640 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbce800 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fe970009bf0 tx=0x7fe970009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b7f1241591c544 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe95ffff640 1 -- 172.21.15.27:0/3884461409 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97004e040 con 0x56499dbb6570 2024-04-25T01:36:50.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe95ffff640 1 -- 172.21.15.27:0/3884461409 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe970064070 con 0x56499dbb6570 2024-04-25T01:36:50.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe95ffff640 1 -- 172.21.15.27:0/3884461409 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe970004310 con 0x56499dbb6570 2024-04-25T01:36:50.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe97f968740 1 -- 172.21.15.27:0/3884461409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 msgr2=0x56499dbce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe97f968740 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbce800 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fe970009bf0 tx=0x7fe970009360 comp rx=0 tx=0).stop 2024-04-25T01:36:50.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe97f968740 1 -- 172.21.15.27:0/3884461409 shutdown_connections 2024-04-25T01:36:50.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe97f968740 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbce800 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe97f968740 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe97f968740 1 --2- 172.21.15.27:0/3884461409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbc21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.262+0000 7fe97f968740 1 -- 172.21.15.27:0/3884461409 >> 172.21.15.27:0/3884461409 conn(0x56499dba78a0 msgr2=0x56499dba6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 -- 172.21.15.27:0/3884461409 shutdown_connections 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 -- 172.21.15.27:0/3884461409 wait complete. 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 -- start start 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 Processor -- start 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbbe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbbbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbbc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe9781ad320 con 0x56499dbb6570 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe9781ae790 con 0x56499dbabea0 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9781ad9f0 con 0x56499dbab580 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe97f968740 10 client.0 _ll_register_callbacks cb 0x7fe970031dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:50.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe976d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbbc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe976d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbbc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47044/0 (socket says 172.21.15.27:47044) 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe976d9d640 1 -- 172.21.15.27:0/3128400859 learned_addr learned my addr 172.21.15.27:0/3128400859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe975d9b640 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbbbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe976d9d640 1 -- 172.21.15.27:0/3128400859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 msgr2=0x56499dbbe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.263+0000 7fe976d9d640 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbbe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe976d9d640 1 -- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 msgr2=0x56499dbbbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe976d9d640 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbbbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe976d9d640 1 -- 172.21.15.27:0/3128400859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499dbbc870 con 0x56499dbb6570 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe975d9b640 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbbbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe976d9d640 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbbc280 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe970009860 tx=0x7fe97003b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed15e42cbc1a7fb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe95dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe95dffb640 1 -- 172.21.15.27:0/3128400859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97004e030 con 0x56499dbb6570 2024-04-25T01:36:50.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe97f968740 1 -- 172.21.15.27:0/3128400859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe970064070 con 0x56499dbb6570 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe95dffb640 1 -- 172.21.15.27:0/3128400859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fe9700305a0 con 0x56499dbb6570 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe95dffb640 1 -- 172.21.15.27:0/3128400859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe970003d40 con 0x56499dbb6570 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe97f968740 10 client.18105 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe97f968740 1 -- 172.21.15.27:0/3128400859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56499dc2c940 con 0x56499dbb6570 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe95dffb640 1 -- 172.21.15.27:0/3128400859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe970064070 con 0x56499dbb6570 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.264+0000 7fe95dffb640 1 -- 172.21.15.27:0/3128400859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe970070030 con 0x56499dbb6570 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 10 client.18105 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 10 client.18105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 20 client.18105 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56499dc2ce40 0x56499dc2f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 4 client.18105 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 1 -- 172.21.15.27:0/3128400859 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56499db17f00 con 0x56499dc2ce40 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97659c640 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56499dc2ce40 0x56499dc2f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe95dffb640 10 client.18105 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97659c640 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56499dc2ce40 0x56499dc2f260 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e2ee0c796956d8e server_cookie=dc8db89c4672e576 in_seq=0 out_seq=0 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe95dffb640 1 -- 172.21.15.27:0/3128400859 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56499db17f00 con 0x56499dc2ce40 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe95dffb640 10 client.18105 handle_command_reply: tid=0 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 1 client.18105 shutdown 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 1 -- 172.21.15.27:0/3128400859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 msgr2=0x56499dbbc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbbc280 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe970009860 tx=0x7fe97003b5e0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 1 -- 172.21.15.27:0/3128400859 shutdown_connections 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x56499dc2ce40 0x56499dc2f260 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe95dffb640 0 client.18105 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.265+0000 7fe97f968740 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56499dbb6570 0x56499dbbc280 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97f968740 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x56499dbabea0 0x56499dbbbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97f968740 1 --2- 172.21.15.27:0/3128400859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499dbab580 0x56499dbbe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97f968740 1 -- 172.21.15.27:0/3128400859 >> 172.21.15.27:0/3128400859 conn(0x56499dba78a0 msgr2=0x56499dbc11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97f968740 1 -- 172.21.15.27:0/3128400859 shutdown_connections 2024-04-25T01:36:50.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97f968740 1 -- 172.21.15.27:0/3128400859 wait complete. 2024-04-25T01:36:50.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97f968740 20 client.18105 trim_cache size 0 max 16384 2024-04-25T01:36:50.248 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 -- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9600a2d00 msgr2=0x7fe9600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9600a2d00 0x7fe9600a51c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe970006010 tx=0x7fe97002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 -- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 msgr2=0x7fe978198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 0x7fe978198440 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe96800c920 tx=0x7fe96800cde0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 -- 172.21.15.27:0/486615684 shutdown_connections 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9781066b0 0x7fe97819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9600a2d00 0x7fe9600a51c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe978100fd0 0x7fe978198440 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 --2- 172.21.15.27:0/486615684 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe978100610 0x7fe978197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 -- 172.21.15.27:0/486615684 >> 172.21.15.27:0/486615684 conn(0x7fe978077740 msgr2=0x7fe9780ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 -- 172.21.15.27:0/486615684 shutdown_connections 2024-04-25T01:36:50.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.266+0000 7fe97cb84640 1 -- 172.21.15.27:0/486615684 wait complete. 2024-04-25T01:36:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:50 smithi027 ceph-mon[29197]: pgmap v355: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:50.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:50 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2731811137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:50.503 DEBUG:teuthology.parallel:result is None 2024-04-25T01:36:50.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T01:36:50.521 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:50 smithi153 ceph-mon[32621]: pgmap v355: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:50.521 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:50 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2731811137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:50.536 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T01:36:50.536 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T01:36:50.547 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T01:36:50.547 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:36:50.547 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:36:50.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T01:36:50.548 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:50 smithi139 ceph-mon[32760]: pgmap v355: 129 pgs: 129 active+clean; 5.5 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 9 op/s 2024-04-25T01:36:50.548 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:50 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2731811137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:50.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T01:36:50.668 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T01:36:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 -- 172.21.15.27:0/130627933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c103310 msgr2=0x7fce8c10b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 --2- 172.21.15.27:0/130627933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c103310 0x7fce8c10b7f0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fce80009900 tx=0x7fce8002a480 comp rx=0 tx=0).stop 2024-04-25T01:36:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 -- 172.21.15.27:0/130627933 shutdown_connections 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 --2- 172.21.15.27:0/130627933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce8c10bdc0 0x7fce8c10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 --2- 172.21.15.27:0/130627933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c103310 0x7fce8c10b7f0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 --2- 172.21.15.27:0/130627933 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce8c1029f0 0x7fce8c102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 -- 172.21.15.27:0/130627933 >> 172.21.15.27:0/130627933 conn(0x7fce8c0769c0 msgr2=0x7fce8c076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 -- 172.21.15.27:0/130627933 shutdown_connections 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 -- 172.21.15.27:0/130627933 wait complete. 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 Processor -- start 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.836+0000 7fce946ee640 1 -- start start 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce946ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c1029f0 0x7fce8c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce946ee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce8c103310 0x7fce8c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce946ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce8c10bdc0 0x7fce8c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce946ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fce8c111b50 con 0x7fce8c1029f0 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce946ee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fce8c1119d0 con 0x7fce8c103310 2024-04-25T01:36:50.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce946ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fce8c111cd0 con 0x7fce8c10bdc0 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c1029f0 0x7fce8c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c1029f0 0x7fce8c19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47072/0 (socket says 172.21.15.27:47072) 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 -- 172.21.15.27:0/988926315 learned_addr learned my addr 172.21.15.27:0/988926315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce92c8b640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce8c10bdc0 0x7fce8c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce91c89640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce8c103310 0x7fce8c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 -- 172.21.15.27:0/988926315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce8c10bdc0 msgr2=0x7fce8c19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce8c10bdc0 0x7fce8c19ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 -- 172.21.15.27:0/988926315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce8c103310 msgr2=0x7fce8c19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce8c103310 0x7fce8c19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 -- 172.21.15.27:0/988926315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce8c19f140 con 0x7fce8c1029f0 2024-04-25T01:36:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce9248a640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c1029f0 0x7fce8c19a150 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fce8800c890 tx=0x7fce8800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce7f7fe640 1 -- 172.21.15.27:0/988926315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce88004220 con 0x7fce8c1029f0 2024-04-25T01:36:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce7f7fe640 1 -- 172.21.15.27:0/988926315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce880043c0 con 0x7fce8c1029f0 2024-04-25T01:36:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce8c19f430 con 0x7fce8c1029f0 2024-04-25T01:36:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.837+0000 7fce7f7fe640 1 -- 172.21.15.27:0/988926315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce880049e0 con 0x7fce8c1029f0 2024-04-25T01:36:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.838+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce8c1a6cc0 con 0x7fce8c1029f0 2024-04-25T01:36:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.839+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce8c04f500 con 0x7fce8c1029f0 2024-04-25T01:36:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.840+0000 7fce7f7fe640 1 -- 172.21.15.27:0/988926315 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fce880079b0 con 0x7fce8c1029f0 2024-04-25T01:36:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.840+0000 7fce7f7fe640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce780a2da0 0x7fce780a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.840+0000 7fce7f7fe640 1 -- 172.21.15.27:0/988926315 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fce880cf950 con 0x7fce8c1029f0 2024-04-25T01:36:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.840+0000 7fce91c89640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce780a2da0 0x7fce780a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.841+0000 7fce91c89640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce780a2da0 0x7fce780a5260 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0ff3d0 tx=0x7fce80031490 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:50.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:50.844+0000 7fce7f7fe640 1 -- 172.21.15.27:0/988926315 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fce88014030 con 0x7fce8c1029f0 2024-04-25T01:36:51.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.052+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fce8c100c80 con 0x7fce780a2da0 2024-04-25T01:36:51.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.060+0000 7fce7f7fe640 1 -- 172.21.15.27:0/988926315 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fce8c100c80 con 0x7fce780a2da0 2024-04-25T01:36:51.035 INFO:teuthology.orchestra.run.smithi027.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T01:25:33", "first": "2024-04-25T01:26:00", "last": "2024-04-25T01:36:00", "last_pruned": "2024-04-25T01:36:00", "created_count": 11, "pruned_count": 5, "active": true} 2024-04-25T01:36:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce780a2da0 msgr2=0x7fce780a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce780a2da0 0x7fce780a5260 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0ff3d0 tx=0x7fce80031490 comp rx=0 tx=0).stop 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c1029f0 msgr2=0x7fce8c19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c1029f0 0x7fce8c19a150 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fce8800c890 tx=0x7fce8800cd50 comp rx=0 tx=0).stop 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 shutdown_connections 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce8c10bdc0 0x7fce8c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fce8c103310 0x7fce8c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fce780a2da0 0x7fce780a5260 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 --2- 172.21.15.27:0/988926315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8c1029f0 0x7fce8c19a150 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 >> 172.21.15.27:0/988926315 conn(0x7fce8c0769c0 msgr2=0x7fce8c100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 shutdown_connections 2024-04-25T01:36:51.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.062+0000 7fce946ee640 1 -- 172.21.15.27:0/988926315 wait complete. 2024-04-25T01:36:51.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T01:36:51.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/4204788611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 msgr2=0x7fbf88106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/4204788611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 0x7fbf88106a30 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009cd0 tx=0x7fbf8002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/4204788611 shutdown_connections 2024-04-25T01:36:51.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/4204788611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf88101000 0x7fbf8810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/4204788611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf88100640 0x7fbf88100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/4204788611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 0x7fbf88106a30 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/4204788611 >> 172.21.15.27:0/4204788611 conn(0x7fbf880777c0 msgr2=0x7fbf88077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/4204788611 shutdown_connections 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/4204788611 wait complete. 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.177+0000 7fbf8d6ee640 1 Processor -- start 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf8d6ee640 1 -- start start 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf8d6ee640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf88100640 0x7fbf88197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf8d6ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf88101000 0x7fbf881984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf8d6ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 0x7fbf8819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf8d6ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbf88110a60 con 0x7fbf88106650 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf8d6ee640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbf881108e0 con 0x7fbf88100640 2024-04-25T01:36:51.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf8d6ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf88110be0 con 0x7fbf88101000 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 0x7fbf8819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 0x7fbf8819c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47088/0 (socket says 172.21.15.27:47088) 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf877fe640 1 -- 172.21.15.27:0/296835688 learned_addr learned my addr 172.21.15.27:0/296835688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf86ffd640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf88100640 0x7fbf88197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf867fc640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf88101000 0x7fbf881984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf877fe640 1 -- 172.21.15.27:0/296835688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf88101000 msgr2=0x7fbf881984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf877fe640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf88101000 0x7fbf881984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf877fe640 1 -- 172.21.15.27:0/296835688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf88100640 msgr2=0x7fbf88197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf877fe640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf88100640 0x7fbf88197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf877fe640 1 -- 172.21.15.27:0/296835688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8819cf60 con 0x7fbf88106650 2024-04-25T01:36:51.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.178+0000 7fbf867fc640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf88101000 0x7fbf881984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:51.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.179+0000 7fbf877fe640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 0x7fbf8819c830 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800d750 tx=0x7fbf7800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.179+0000 7fbf7ffff640 1 -- 172.21.15.27:0/296835688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78017070 con 0x7fbf88106650 2024-04-25T01:36:51.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.179+0000 7fbf7ffff640 1 -- 172.21.15.27:0/296835688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf78005500 con 0x7fbf88106650 2024-04-25T01:36:51.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.179+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8819d250 con 0x7fbf88106650 2024-04-25T01:36:51.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.179+0000 7fbf7ffff640 1 -- 172.21.15.27:0/296835688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78004d90 con 0x7fbf88106650 2024-04-25T01:36:51.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.179+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf881a4ae0 con 0x7fbf88106650 2024-04-25T01:36:51.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.180+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf8804f500 con 0x7fbf88106650 2024-04-25T01:36:51.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.181+0000 7fbf7ffff640 1 -- 172.21.15.27:0/296835688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbf78004f30 con 0x7fbf88106650 2024-04-25T01:36:51.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.182+0000 7fbf7ffff640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbf740a2e20 0x7fbf740a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.182+0000 7fbf86ffd640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbf740a2e20 0x7fbf740a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.182+0000 7fbf7ffff640 1 -- 172.21.15.27:0/296835688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fbf780cf500 con 0x7fbf88106650 2024-04-25T01:36:51.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.182+0000 7fbf86ffd640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbf740a2e20 0x7fbf740a52e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbf80006010 tx=0x7fbf800093f0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.185+0000 7fbf7ffff640 1 -- 172.21.15.27:0/296835688 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbf7809be50 con 0x7fbf88106650 2024-04-25T01:36:51.253 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:36:51.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:36:51.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:36:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 -- 172.21.15.27:0/3142621518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f01032e0 msgr2=0x7f27f010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 --2- 172.21.15.27:0/3142621518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f01032e0 0x7f27f010b7c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009900 tx=0x7f27e002a510 comp rx=0 tx=0).stop 2024-04-25T01:36:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 -- 172.21.15.27:0/3142621518 shutdown_connections 2024-04-25T01:36:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 --2- 172.21.15.27:0/3142621518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f010bd00 0x7f27f010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 --2- 172.21.15.27:0/3142621518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f01032e0 0x7f27f010b7c0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 --2- 172.21.15.27:0/3142621518 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f01029c0 0x7f27f0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 -- 172.21.15.27:0/3142621518 >> 172.21.15.27:0/3142621518 conn(0x7f27f0077740 msgr2=0x7f27f0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 -- 172.21.15.27:0/3142621518 shutdown_connections 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.384+0000 7f27f5a38640 1 -- 172.21.15.27:0/3142621518 wait complete. 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.385+0000 7f27f5a38640 1 Processor -- start 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.385+0000 7f27f5a38640 1 -- start start 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.385+0000 7f27f5a38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f01029c0 0x7f27f0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.385+0000 7f27f5a38640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f01032e0 0x7f27f01984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.385+0000 7f27f5a38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f010bd00 0x7f27f019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.385+0000 7f27f5a38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0111b00 con 0x7f27f010bd00 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.385+0000 7f27f5a38640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0111980 con 0x7f27f01032e0 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.385+0000 7f27f5a38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0111c80 con 0x7f27f01029c0 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f010bd00 0x7f27f019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f010bd00 0x7f27f019c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47104/0 (socket says 172.21.15.27:47104) 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 -- 172.21.15.27:0/716778230 learned_addr learned my addr 172.21.15.27:0/716778230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27eeffd640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f01029c0 0x7f27f0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f01032e0 0x7f27f01984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 -- 172.21.15.27:0/716778230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f01029c0 msgr2=0x7f27f0197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f01029c0 0x7f27f0197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 -- 172.21.15.27:0/716778230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f01032e0 msgr2=0x7f27f01984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f01032e0 0x7f27f01984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 -- 172.21.15.27:0/716778230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f019cf20 con 0x7f27f010bd00 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27ef7fe640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f010bd00 0x7f27f019c7f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f27e800cce0 tx=0x7f27e800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27f4a36640 1 -- 172.21.15.27:0/716778230 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e800f450 con 0x7f27f010bd00 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27f4a36640 1 -- 172.21.15.27:0/716778230 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e800fad0 con 0x7f27f010bd00 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27f4a36640 1 -- 172.21.15.27:0/716778230 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8018b60 con 0x7f27f010bd00 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27f5a38640 1 -- 172.21.15.27:0/716778230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f019d210 con 0x7f27f010bd00 2024-04-25T01:36:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.386+0000 7f27f5a38640 1 -- 172.21.15.27:0/716778230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f01a4aa0 con 0x7f27f010bd00 2024-04-25T01:36:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.389+0000 7f27f8826740 1 Processor -- start 2024-04-25T01:36:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.389+0000 7f27f4a36640 1 -- 172.21.15.27:0/716778230 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f27e800fc80 con 0x7f27f010bd00 2024-04-25T01:36:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.389+0000 7f27f8826740 1 -- start start 2024-04-25T01:36:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27f8826740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27f8826740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1ae290 0x5635aa1ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27f8826740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1b9010 0x5635aa1bb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27f8826740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5635aa1122c0 con 0x5635aa1af980 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27f8826740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5635aa0ba910 con 0x5635aa1b9010 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27f8826740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5635aa10e280 con 0x5635aa1ae290 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47118/0 (socket says 172.21.15.27:47118) 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 -- 172.21.15.27:0/1180662758 learned_addr learned my addr 172.21.15.27:0/1180662758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27eeffd640 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1ae290 0x5635aa1ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1b9010 0x5635aa1bb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27f4a36640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f27dc0a2d80 0x7f27dc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 -- 172.21.15.27:0/1180662758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1ae290 msgr2=0x5635aa1ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1ae290 0x5635aa1ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 -- 172.21.15.27:0/1180662758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1b9010 msgr2=0x5635aa1bb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1b9010 0x5635aa1bb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 -- 172.21.15.27:0/1180662758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635aa0cbfb0 con 0x5635aa1af980 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ef7fe640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f27dc0a2d80 0x7f27dc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27f4a36640 1 -- 172.21.15.27:0/716778230 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f27e80cfc70 con 0x7f27f010bd00 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.390+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1adcc0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f27e0005ab0 tx=0x7f27e00058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e6e723f10999f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27e5ffb640 1 -- 172.21.15.27:0/1180662758 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0043020 con 0x5635aa1af980 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27e5ffb640 1 -- 172.21.15.27:0/1180662758 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e0005500 con 0x5635aa1af980 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27e5ffb640 1 -- 172.21.15.27:0/1180662758 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0035050 con 0x5635aa1af980 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27f8826740 1 -- 172.21.15.27:0/1180662758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 msgr2=0x5635aa1adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27f8826740 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1adcc0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f27e0005ab0 tx=0x7f27e00058b0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27ef7fe640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f27dc0a2d80 0x7f27dc0a5240 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f27e80cd080 tx=0x7f27e8091230 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27f8826740 1 -- 172.21.15.27:0/1180662758 shutdown_connections 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27f8826740 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1b9010 0x5635aa1bb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27f8826740 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1ae290 0x5635aa1ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27f8826740 1 --2- 172.21.15.27:0/1180662758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1adcc0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.391+0000 7f27f8826740 1 -- 172.21.15.27:0/1180662758 >> 172.21.15.27:0/1180662758 conn(0x5635aa176950 msgr2=0x5635aa1aa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 -- 172.21.15.27:0/1180662758 shutdown_connections 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 -- 172.21.15.27:0/1180662758 wait complete. 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 -- start start 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 Processor -- start 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1ae290 0x5635aa1b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1b6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1b6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5635aa1bbe40 con 0x5635aa1af980 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5635aa0cde90 con 0x5635aa1ae290 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5635aa1bbfc0 con 0x5635aa1b9010 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27f8826740 10 client.0 _ll_register_callbacks cb 0x5635aa109970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1b6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47122/0 (socket says 172.21.15.27:47122) 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.392+0000 7f27eeffd640 1 -- 172.21.15.27:0/1218525437 learned_addr learned my addr 172.21.15.27:0/1218525437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1ae290 0x5635aa1b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27eeffd640 1 -- 172.21.15.27:0/1218525437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 msgr2=0x5635aa1b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27eeffd640 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1b6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27eeffd640 1 -- 172.21.15.27:0/1218525437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1ae290 msgr2=0x5635aa1b5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27eeffd640 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1ae290 0x5635aa1b5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27eeffd640 1 -- 172.21.15.27:0/1218525437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635a9eee7e0 con 0x5635aa1af980 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27eeffd640 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1b6430 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f27d80127d0 tx=0x7f27d8012c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4facdbe87c80e308 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27ceffd640 1 -- 172.21.15.27:0/1218525437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d801f060 con 0x5635aa1af980 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27ceffd640 1 -- 172.21.15.27:0/1218525437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f27d8010dc0 con 0x5635aa1af980 2024-04-25T01:36:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27f8826740 1 -- 172.21.15.27:0/1218525437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27e0005500 con 0x5635aa1af980 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27ceffd640 1 -- 172.21.15.27:0/1218525437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f27d8026050 con 0x5635aa1af980 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27f8826740 10 client.18129 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27f8826740 1 -- 172.21.15.27:0/1218525437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5635a9f4ef70 con 0x5635aa1af980 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.393+0000 7f27ceffd640 1 -- 172.21.15.27:0/1218525437 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e0005500 con 0x5635aa1af980 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27ceffd640 1 -- 172.21.15.27:0/1218525437 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f27d800d5a0 con 0x5635aa1af980 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27f8826740 10 client.18129 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27f8826740 10 client.18129 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27f8826740 20 client.18129 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27f8826740 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5635aa1c90f0 0x5635aa1cb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27f8826740 4 client.18129 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27f8826740 1 -- 172.21.15.27:0/1218525437 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635aa0e65a0 con 0x5635aa1c90f0 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5635aa1c90f0 0x5635aa1cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5635aa1c90f0 0x5635aa1cb520 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb64852a3b365d42 server_cookie=1e4749d13b61ebc5 in_seq=0 out_seq=0 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.394+0000 7f27ceffd640 10 client.18129 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27ceffd640 1 -- 172.21.15.27:0/1218525437 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635aa0e65a0 con 0x5635aa1c90f0 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27ceffd640 10 client.18129 handle_command_reply: tid=0 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 client.18129 shutdown 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 -- 172.21.15.27:0/1218525437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 msgr2=0x5635aa1b6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1b6430 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f27d80127d0 tx=0x7f27d8012c90 comp rx=0 tx=0).stop 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 -- 172.21.15.27:0/1218525437 shutdown_connections 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5635aa1c90f0 0x5635aa1cb520 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1af980 0x5635aa1b6430 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27ceffd640 0 client.18129 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 --2- 172.21.15.27:0/1218525437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1ae290 0x5635aa1b5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.395+0000 7f27f8826740 1 -- 172.21.15.27:0/1218525437 >> 172.21.15.27:0/1218525437 conn(0x5635aa176950 msgr2=0x5635aa1b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.396+0000 7f27f8826740 1 -- 172.21.15.27:0/1218525437 shutdown_connections 2024-04-25T01:36:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.396+0000 7f27f8826740 1 -- 172.21.15.27:0/1218525437 wait complete. 2024-04-25T01:36:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.396+0000 7f27f8826740 20 client.18129 trim_cache size 0 max 16384 2024-04-25T01:36:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.396+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 --> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fbf88061b80 con 0x7fbf740a2e20 2024-04-25T01:36:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.403+0000 7fbf7ffff640 1 -- 172.21.15.27:0/296835688 <== mgr.15480 v2:172.21.15.27:6800/1825554759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fbf88061b80 con 0x7fbf740a2e20 2024-04-25T01:36:51.378 INFO:teuthology.orchestra.run.smithi027.stdout:/ 1m 6m3h 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbf740a2e20 msgr2=0x7fbf740a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbf740a2e20 0x7fbf740a52e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbf80006010 tx=0x7fbf800093f0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 msgr2=0x7fbf8819c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 0x7fbf8819c830 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800d750 tx=0x7fbf7800dc10 comp rx=0 tx=0).stop 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 shutdown_connections 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbf740a2e20 0x7fbf740a52e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf88106650 0x7fbf8819c830 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf88101000 0x7fbf881984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 --2- 172.21.15.27:0/296835688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbf88100640 0x7fbf88197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.405+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 >> 172.21.15.27:0/296835688 conn(0x7fbf880777c0 msgr2=0x7fbf8806a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.406+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 shutdown_connections 2024-04-25T01:36:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.406+0000 7fbf8d6ee640 1 -- 172.21.15.27:0/296835688 wait complete. 2024-04-25T01:36:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27f8826740 1 Processor -- start 2024-04-25T01:36:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27f8826740 1 -- start start 2024-04-25T01:36:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27f8826740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27f8826740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27f8826740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27f8826740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01b13d0 con 0x5635aa1ae290 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27f8826740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01aeca0 con 0x5635aa1af980 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27f8826740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01acb10 con 0x5635aa1b9010 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47138/0 (socket says 172.21.15.27:47138) 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27ee7fc640 1 -- 172.21.15.27:0/2443817943 learned_addr learned my addr 172.21.15.27:0/2443817943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27eeffd640 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27ee7fc640 1 -- 172.21.15.27:0/2443817943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 msgr2=0x5635aa1d0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27ee7fc640 1 -- 172.21.15.27:0/2443817943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 msgr2=0x5635aa1b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1b7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27ee7fc640 1 -- 172.21.15.27:0/2443817943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635a9de6170 con 0x5635aa1ae290 2024-04-25T01:36:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.415+0000 7f27eeffd640 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.416+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c4d70 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009900 tx=0x7f27e0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f94f4347a0628c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.416+0000 7f27e57fa640 1 -- 172.21.15.27:0/2443817943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0043040 con 0x5635aa1ae290 2024-04-25T01:36:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.416+0000 7f27e57fa640 1 -- 172.21.15.27:0/2443817943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e0052070 con 0x5635aa1ae290 2024-04-25T01:36:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.416+0000 7f27e57fa640 1 -- 172.21.15.27:0/2443817943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0035040 con 0x5635aa1ae290 2024-04-25T01:36:51.395 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T01:36:51.405 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T01:36:51.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 -- 172.21.15.27:0/2443817943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 msgr2=0x5635aa1c4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c4d70 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009900 tx=0x7f27e0005fd0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 -- 172.21.15.27:0/2443817943 shutdown_connections 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1d0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 --2- 172.21.15.27:0/2443817943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c4d70 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 -- 172.21.15.27:0/2443817943 >> 172.21.15.27:0/2443817943 conn(0x5635aa176950 msgr2=0x5635aa1b7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 -- 172.21.15.27:0/2443817943 shutdown_connections 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 -- 172.21.15.27:0/2443817943 wait complete. 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 -- start start 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.417+0000 7f27f8826740 1 Processor -- start 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1c09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01ad160 con 0x5635aa1ae290 2024-04-25T01:36:51.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01ae5d0 con 0x5635aa1af980 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01ad830 con 0x5635aa1b9010 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 10 client.0 _ll_register_callbacks cb 0x7f27d800a690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47140/0 (socket says 172.21.15.27:47140) 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 -- 172.21.15.27:0/1445012582 learned_addr learned my addr 172.21.15.27:0/1445012582 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27eeffd640 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1c09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 -- 172.21.15.27:0/1445012582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 msgr2=0x5635aa1c0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1c0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 -- 172.21.15.27:0/1445012582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 msgr2=0x5635aa1c09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1c09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 -- 172.21.15.27:0/1445012582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635aa1c1760 con 0x5635aa1ae290 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ee7fc640 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c0470 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f27e0031a40 tx=0x7f27e0003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2007a8012d6a2f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:51.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ceffd640 1 -- 172.21.15.27:0/1445012582 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0043070 con 0x5635aa1ae290 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ceffd640 1 -- 172.21.15.27:0/1445012582 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f27e00037f0 con 0x5635aa1ae290 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 1 -- 172.21.15.27:0/1445012582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27e0052070 con 0x5635aa1ae290 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27ceffd640 1 -- 172.21.15.27:0/1445012582 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f27e0035070 con 0x5635aa1ae290 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.418+0000 7f27f8826740 10 client.18135 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27f8826740 1 -- 172.21.15.27:0/1445012582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5635aa234800 con 0x5635aa1ae290 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27ceffd640 1 -- 172.21.15.27:0/1445012582 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e0052070 con 0x5635aa1ae290 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27ceffd640 1 -- 172.21.15.27:0/1445012582 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f27e0048040 con 0x5635aa1ae290 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27f8826740 10 client.18135 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27f8826740 10 client.18135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:51.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27f8826740 20 client.18135 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27f8826740 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5635aa234a30 0x5635aa236e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27f8826740 4 client.18135 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27f8826740 1 -- 172.21.15.27:0/1445012582 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5635aa11f800 con 0x5635aa234a30 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.419+0000 7f27eeffd640 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5635aa234a30 0x5635aa236e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.420+0000 7f27eeffd640 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5635aa234a30 0x5635aa236e20 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d35195c9553ed1ff server_cookie=2c05952d14f3d676 in_seq=0 out_seq=0 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.420+0000 7f27ceffd640 10 client.18135 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27ceffd640 1 -- 172.21.15.27:0/1445012582 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5635aa11f800 con 0x5635aa234a30 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27ceffd640 10 client.18135 handle_command_reply: tid=0 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 client.18135 shutdown 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 -- 172.21.15.27:0/1445012582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 msgr2=0x5635aa1c0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c0470 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f27e0031a40 tx=0x7f27e0003f70 comp rx=0 tx=0).stop 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 -- 172.21.15.27:0/1445012582 shutdown_connections 2024-04-25T01:36:51.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5635aa234a30 0x5635aa236e20 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5635aa1b9010 0x5635aa1c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5635aa1af980 0x5635aa1c09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 --2- 172.21.15.27:0/1445012582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635aa1ae290 0x5635aa1c0470 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 -- 172.21.15.27:0/1445012582 >> 172.21.15.27:0/1445012582 conn(0x5635aa176950 msgr2=0x5635aa1a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27ceffd640 0 client.18135 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 -- 172.21.15.27:0/1445012582 shutdown_connections 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 1 -- 172.21.15.27:0/1445012582 wait complete. 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.421+0000 7f27f8826740 20 client.18135 trim_cache size 0 max 16384 2024-04-25T01:36:51.415 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:36:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T01:36:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "df52f62d-479d-465a-bf49-0b98888e1f2e", 2024-04-25T01:36:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T01:36:51.416 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:36:51.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 -- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f27dc0a2d80 msgr2=0x7f27dc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f27dc0a2d80 0x7f27dc0a5240 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f27e80cd080 tx=0x7f27e8091230 comp rx=0 tx=0).stop 2024-04-25T01:36:51.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 -- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f010bd00 msgr2=0x7f27f019c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f010bd00 0x7f27f019c7f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f27e800cce0 tx=0x7f27e800a620 comp rx=0 tx=0).stop 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 -- 172.21.15.27:0/716778230 shutdown_connections 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f27dc0a2d80 0x7f27dc0a5240 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f010bd00 0x7f27f019c7f0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27f01032e0 0x7f27f01984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 --2- 172.21.15.27:0/716778230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f01029c0 0x7f27f0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 -- 172.21.15.27:0/716778230 >> 172.21.15.27:0/716778230 conn(0x7f27f0077740 msgr2=0x7f27f00fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 -- 172.21.15.27:0/716778230 shutdown_connections 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.422+0000 7f27f5a38640 1 -- 172.21.15.27:0/716778230 wait complete. 2024-04-25T01:36:51.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:36:51.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:36:51.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 -- 172.21.15.27:0/589641361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01032e0 msgr2=0x7fd1e010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/589641361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01032e0 0x7fd1e010b7c0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009900 tx=0x7fd1d002a510 comp rx=0 tx=0).stop 2024-04-25T01:36:51.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 -- 172.21.15.27:0/589641361 shutdown_connections 2024-04-25T01:36:51.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/589641361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1e010bd00 0x7fd1e010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/589641361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01032e0 0x7fd1e010b7c0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/589641361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e01029c0 0x7fd1e0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 -- 172.21.15.27:0/589641361 >> 172.21.15.27:0/589641361 conn(0x7fd1e0077740 msgr2=0x7fd1e0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 -- 172.21.15.27:0/589641361 shutdown_connections 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.587+0000 7fd1e7f53640 1 -- 172.21.15.27:0/589641361 wait complete. 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e7f53640 1 Processor -- start 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e7f53640 1 -- start start 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e7f53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01029c0 0x7fd1e019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e7f53640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e01032e0 0x7fd1e019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e7f53640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1e010bd00 0x7fd1e019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e7f53640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0111b00 con 0x7fd1e01029c0 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e7f53640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0111980 con 0x7fd1e010bd00 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e7f53640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0111c80 con 0x7fd1e01032e0 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e5cef640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01029c0 0x7fd1e019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e5cef640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01029c0 0x7fd1e019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47158/0 (socket says 172.21.15.27:47158) 2024-04-25T01:36:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e5cef640 1 -- 172.21.15.27:0/3073380547 learned_addr learned my addr 172.21.15.27:0/3073380547 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e01032e0 0x7fd1e019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.588+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1e010bd00 0x7fd1e019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1e5cef640 1 -- 172.21.15.27:0/3073380547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e01032e0 msgr2=0x7fd1e019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e01032e0 0x7fd1e019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1e5cef640 1 -- 172.21.15.27:0/3073380547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1e010bd00 msgr2=0x7fd1e019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1e010bd00 0x7fd1e019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1e5cef640 1 -- 172.21.15.27:0/3073380547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e019f220 con 0x7fd1e01029c0 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01029c0 0x7fd1e019a230 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d770 tx=0x7fd1dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1d6ffd640 1 -- 172.21.15.27:0/3073380547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc017070 con 0x7fd1e01029c0 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1e7f53640 1 -- 172.21.15.27:0/3073380547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e019f510 con 0x7fd1e01029c0 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1d6ffd640 1 -- 172.21.15.27:0/3073380547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc005500 con 0x7fd1e01029c0 2024-04-25T01:36:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1d6ffd640 1 -- 172.21.15.27:0/3073380547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc004e00 con 0x7fd1e01029c0 2024-04-25T01:36:51.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.589+0000 7fd1e7f53640 1 -- 172.21.15.27:0/3073380547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e01a6da0 con 0x7fd1e01029c0 2024-04-25T01:36:51.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.591+0000 7fd1d6ffd640 1 -- 172.21.15.27:0/3073380547 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd1dc004fa0 con 0x7fd1e01029c0 2024-04-25T01:36:51.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1eac5e740 1 Processor -- start 2024-04-25T01:36:51.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1eac5e740 1 -- start start 2024-04-25T01:36:51.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb019a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb019f50 0x5646eb0240e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb024620 0x5646eb026a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5646eaf7e6b0 con 0x5646eb019630 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5646eaf26880 con 0x5646eb019f50 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5646eaf7a670 con 0x5646eb024620 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1d6ffd640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1cc0a2d00 0x7fd1cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1e64f0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb019a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1e64f0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb019a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47164/0 (socket says 172.21.15.27:47164) 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1e64f0640 1 -- 172.21.15.27:0/2795360752 learned_addr learned my addr 172.21.15.27:0/2795360752 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1cc0a2d00 0x7fd1cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb024620 0x5646eb026a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.592+0000 7fd1d6ffd640 1 -- 172.21.15.27:0/3073380547 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fd1dc0cf2c0 con 0x7fd1e01029c0 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb019f50 0x5646eb0240e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e64f0640 1 -- 172.21.15.27:0/2795360752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb024620 msgr2=0x5646eb026a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb024620 0x5646eb026a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e64f0640 1 -- 172.21.15.27:0/2795360752 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb019f50 msgr2=0x5646eb0240e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb019f50 0x5646eb0240e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e64f0640 1 -- 172.21.15.27:0/2795360752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646ead5c7d0 con 0x5646eb019630 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb019f50 0x5646eb0240e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb019a10 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80094d0 tx=0x7fd1d80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97d1538a008a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1cc0a2d00 0x7fd1cc0a51c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8005c40 tx=0x7fd1d801d000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1cbfff640 1 -- 172.21.15.27:0/2795360752 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8012840 con 0x5646eb019630 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1cbfff640 1 -- 172.21.15.27:0/2795360752 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1d8003e50 con 0x5646eb019630 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.593+0000 7fd1cbfff640 1 -- 172.21.15.27:0/2795360752 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800de90 con 0x5646eb019630 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 -- 172.21.15.27:0/2795360752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 msgr2=0x5646eb019a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb019a10 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80094d0 tx=0x7fd1d80107a0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 -- 172.21.15.27:0/2795360752 shutdown_connections 2024-04-25T01:36:51.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb024620 0x5646eb026a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb019f50 0x5646eb0240e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/2795360752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb019a10 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 -- 172.21.15.27:0/2795360752 >> 172.21.15.27:0/2795360752 conn(0x5646eb015950 msgr2=0x5646eb015d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 -- 172.21.15.27:0/2795360752 shutdown_connections 2024-04-25T01:36:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 -- 172.21.15.27:0/2795360752 wait complete. 2024-04-25T01:36:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 -- start start 2024-04-25T01:36:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.594+0000 7fd1eac5e740 1 Processor -- start 2024-04-25T01:36:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb021620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019f50 0x5646eb021b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb0220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5646eb0273b0 con 0x5646eb019630 2024-04-25T01:36:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5646eaf39f50 con 0x5646eb024620 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5646eb027530 con 0x5646eb019f50 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1eac5e740 10 client.0 _ll_register_callbacks cb 0x5646eaf75000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb021620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb021620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47178/0 (socket says 172.21.15.27:47178) 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 -- 172.21.15.27:0/1023511408 learned_addr learned my addr 172.21.15.27:0/1023511408 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019f50 0x5646eb021b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb0220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 -- 172.21.15.27:0/1023511408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019f50 msgr2=0x5646eb021b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019f50 0x5646eb021b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 -- 172.21.15.27:0/1023511408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 msgr2=0x5646eb0220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb0220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 -- 172.21.15.27:0/1023511408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646eac923f0 con 0x5646eb019630 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb021620 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009610 tx=0x7fd1dc09af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df91598e00f52b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.595+0000 7fd1c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1c97fa640 1 -- 172.21.15.27:0/1023511408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0b4060 con 0x5646eb019630 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1c97fa640 1 -- 172.21.15.27:0/1023511408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fd1dc0cc8a0 con 0x5646eb019630 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 1 -- 172.21.15.27:0/1023511408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1d8003e50 con 0x5646eb019630 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1c97fa640 1 -- 172.21.15.27:0/1023511408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd1dc020040 con 0x5646eb019630 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 10 client.18147 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 1 -- 172.21.15.27:0/1023511408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5646eac520f0 con 0x5646eb019630 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1c97fa640 1 -- 172.21.15.27:0/1023511408 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1d8003e50 con 0x5646eb019630 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1c97fa640 1 -- 172.21.15.27:0/1023511408 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd1dc097a60 con 0x5646eb019630 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 10 client.18147 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 10 client.18147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 20 client.18147 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5646eb0347c0 0x5646eb036bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 4 client.18147 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.596+0000 7fd1eac5e740 1 -- 172.21.15.27:0/1023511408 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646eaf8bbf0 con 0x5646eb0347c0 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.597+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5646eb0347c0 0x5646eb036bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.597+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5646eb0347c0 0x5646eb036bd0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7873a806006b6788 server_cookie=561bb74b92cffd3f in_seq=0 out_seq=0 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.597+0000 7fd1c97fa640 10 client.18147 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1c97fa640 1 -- 172.21.15.27:0/1023511408 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646eaf8bbf0 con 0x5646eb0347c0 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1c97fa640 10 client.18147 handle_command_reply: tid=0 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 client.18147 shutdown 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 -- 172.21.15.27:0/1023511408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 msgr2=0x5646eb021620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb021620 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009610 tx=0x7fd1dc09af30 comp rx=0 tx=0).stop 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 -- 172.21.15.27:0/1023511408 shutdown_connections 2024-04-25T01:36:51.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5646eb0347c0 0x5646eb036bd0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb0220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1c97fa640 0 client.18147 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:51.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019f50 0x5646eb021b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/1023511408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019630 0x5646eb021620 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 -- 172.21.15.27:0/1023511408 >> 172.21.15.27:0/1023511408 conn(0x5646eb015950 msgr2=0x5646eb01b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 -- 172.21.15.27:0/1023511408 shutdown_connections 2024-04-25T01:36:51.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 1 -- 172.21.15.27:0/1023511408 wait complete. 2024-04-25T01:36:51.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.598+0000 7fd1eac5e740 20 client.18147 trim_cache size 0 max 16384 2024-04-25T01:36:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.615+0000 7fd1eac5e740 1 Processor -- start 2024-04-25T01:36:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1eac5e740 1 -- start start 2024-04-25T01:36:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 0x5646eb030270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb023450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb03c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01b3770 con 0x5646eb019f50 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01b1040 con 0x5646eb024620 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01aeeb0 con 0x5646eb019630 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb023450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb023450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47194/0 (socket says 172.21.15.27:47194) 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 -- 172.21.15.27:0/4033086566 learned_addr learned my addr 172.21.15.27:0/4033086566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 0x5646eb030270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 -- 172.21.15.27:0/4033086566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 msgr2=0x5646eb030270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 0x5646eb030270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 -- 172.21.15.27:0/4033086566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 msgr2=0x5646eb03c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb03c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb03c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 -- 172.21.15.27:0/4033086566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646eb026a10 con 0x5646eb019f50 2024-04-25T01:36:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb023450 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd1d002aa30 tx=0x7fd1d00319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100543b2532a3b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1cb7fe640 1 -- 172.21.15.27:0/4033086566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d004c030 con 0x5646eb019f50 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1cb7fe640 1 -- 172.21.15.27:0/4033086566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1d0048420 con 0x5646eb019f50 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.616+0000 7fd1cb7fe640 1 -- 172.21.15.27:0/4033086566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0035040 con 0x5646eb019f50 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 -- 172.21.15.27:0/4033086566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 msgr2=0x5646eb023450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb023450 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd1d002aa30 tx=0x7fd1d00319a0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 -- 172.21.15.27:0/4033086566 shutdown_connections 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb03c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb023450 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/4033086566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 0x5646eb030270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 -- 172.21.15.27:0/4033086566 >> 172.21.15.27:0/4033086566 conn(0x5646eb015950 msgr2=0x5646eb022d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 -- 172.21.15.27:0/4033086566 shutdown_connections 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 -- 172.21.15.27:0/4033086566 wait complete. 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 -- start start 2024-04-25T01:36:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.617+0000 7fd1eac5e740 1 Processor -- start 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 0x5646eb02cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb027e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1eac5e740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb0283d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01af500 con 0x5646eb019f50 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01b0970 con 0x5646eb024620 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1eac5e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01afbd0 con 0x5646eb019630 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1eac5e740 10 client.0 _ll_register_callbacks cb 0x7fd1dc00d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb027e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb027e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47210/0 (socket says 172.21.15.27:47210) 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 -- 172.21.15.27:0/43902575 learned_addr learned my addr 172.21.15.27:0/43902575 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 0x5646eb02cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e64f0640 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb0283d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 -- 172.21.15.27:0/43902575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 msgr2=0x5646eb02cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 0x5646eb02cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 -- 172.21.15.27:0/43902575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 msgr2=0x5646eb0283d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb0283d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 -- 172.21.15.27:0/43902575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646eb028c00 con 0x5646eb019f50 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1e54ee640 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb027e90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009900 tx=0x7fd1d00315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed775441349f6381 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.618+0000 7fd1c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1c97fa640 1 -- 172.21.15.27:0/43902575 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0055030 con 0x5646eb019f50 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1c97fa640 1 -- 172.21.15.27:0/43902575 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fd1d0005260 con 0x5646eb019f50 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 1 -- 172.21.15.27:0/43902575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1d0048420 con 0x5646eb019f50 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1c97fa640 1 -- 172.21.15.27:0/43902575 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd1d0062050 con 0x5646eb019f50 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 10 client.18153 fetch_fsmap learned FSMap version 27 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 1 -- 172.21.15.27:0/43902575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5646eb09ad10 con 0x5646eb019f50 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1c97fa640 1 -- 172.21.15.27:0/43902575 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1d0048420 con 0x5646eb019f50 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1c97fa640 1 -- 172.21.15.27:0/43902575 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd1d0051500 con 0x5646eb019f50 2024-04-25T01:36:51.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 10 client.18153 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 10 client.18153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 20 client.18153 populate_metadata read hostname 'smithi027' 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5646eb09b350 0x5646eb09d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 4 client.18153 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1eac5e740 1 -- 172.21.15.27:0/43902575 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5646eaf85f40 con 0x5646eb09b350 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.619+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5646eb09b350 0x5646eb09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1e5cef640 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5646eb09b350 0x5646eb09d770 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25842bc2fa9a1926 server_cookie=a1f6eb552bcf38e5 in_seq=0 out_seq=0 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1c97fa640 10 client.18153 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1c97fa640 1 -- 172.21.15.27:0/43902575 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5646eaf85f40 con 0x5646eb09b350 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1c97fa640 10 client.18153 handle_command_reply: tid=0 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 client.18153 shutdown 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 -- 172.21.15.27:0/43902575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 msgr2=0x5646eb027e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb027e90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009900 tx=0x7fd1d00315b0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 -- 172.21.15.27:0/43902575 shutdown_connections 2024-04-25T01:36:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5646eb09b350 0x5646eb09d770 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5646eb024620 0x5646eb0283d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5646eb019f50 0x5646eb027e90 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 --2- 172.21.15.27:0/43902575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646eb019630 0x5646eb02cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1c97fa640 0 client.18153 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 -- 172.21.15.27:0/43902575 >> 172.21.15.27:0/43902575 conn(0x5646eb015950 msgr2=0x5646eb021e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 -- 172.21.15.27:0/43902575 shutdown_connections 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 1 -- 172.21.15.27:0/43902575 wait complete. 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.620+0000 7fd1eac5e740 20 client.18153 trim_cache size 0 max 16384 2024-04-25T01:36:51.602 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout: "df52f62d-479d-465a-bf49-0b98888e1f2e": { 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "df52f62d-479d-465a-bf49-0b98888e1f2e", 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:36:51.603 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:36:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 -- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1cc0a2d00 msgr2=0x7fd1cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1cc0a2d00 0x7fd1cc0a51c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8005c40 tx=0x7fd1d801d000 comp rx=0 tx=0).stop 2024-04-25T01:36:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 -- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01029c0 msgr2=0x7fd1e019a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01029c0 0x7fd1e019a230 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d770 tx=0x7fd1dc00dc30 comp rx=0 tx=0).stop 2024-04-25T01:36:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 -- 172.21.15.27:0/3073380547 shutdown_connections 2024-04-25T01:36:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1e010bd00 0x7fd1e019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e01032e0 0x7fd1e019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fd1cc0a2d00 0x7fd1cc0a51c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 --2- 172.21.15.27:0/3073380547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1e01029c0 0x7fd1e019a230 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:51.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 -- 172.21.15.27:0/3073380547 >> 172.21.15.27:0/3073380547 conn(0x7fd1e0077740 msgr2=0x7fd1e0107bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:51.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 -- 172.21.15.27:0/3073380547 shutdown_connections 2024-04-25T01:36:51.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:36:51.621+0000 7fd1e7f53640 1 -- 172.21.15.27:0/3073380547 wait complete. 2024-04-25T01:36:51.607 INFO:tasks.cephfs.filesystem:scrub status for tag:df52f62d-479d-465a-bf49-0b98888e1f2e - {'path': '/', 'tag': 'df52f62d-479d-465a-bf49-0b98888e1f2e', 'options': 'recursive,force'} 2024-04-25T01:36:52.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:52 smithi153 ceph-mon[32621]: from='client.18111 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:36:52.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:52 smithi153 ceph-mon[32621]: pgmap v356: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-25T01:36:52.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:52 smithi153 ceph-mon[32621]: from='client.18117 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:36:52.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:52 smithi153 ceph-mon[32621]: scrub queued for path: / 2024-04-25T01:36:52.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:52 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:52.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:52 smithi153 ceph-mon[32621]: scrub summary: active paths [/] 2024-04-25T01:36:52.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:52 smithi153 ceph-mon[32621]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:52 smithi139 ceph-mon[32760]: from='client.18111 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:36:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:52 smithi139 ceph-mon[32760]: pgmap v356: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-25T01:36:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:52 smithi139 ceph-mon[32760]: from='client.18117 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:36:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:52 smithi139 ceph-mon[32760]: scrub queued for path: / 2024-04-25T01:36:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:52 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:52 smithi139 ceph-mon[32760]: scrub summary: active paths [/] 2024-04-25T01:36:52.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:52 smithi139 ceph-mon[32760]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:52.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:52 smithi027 ceph-mon[29197]: from='client.18111 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:36:52.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:52 smithi027 ceph-mon[29197]: pgmap v356: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-25T01:36:52.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:52 smithi027 ceph-mon[29197]: from='client.18117 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:36:52.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:52 smithi027 ceph-mon[29197]: scrub queued for path: / 2024-04-25T01:36:52.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:52 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:52.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:52 smithi027 ceph-mon[29197]: scrub summary: active paths [/] 2024-04-25T01:36:52.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:52 smithi027 ceph-mon[29197]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:53.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:53 smithi153 ceph-mon[32621]: scrub summary: idle 2024-04-25T01:36:53.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:53 smithi139 ceph-mon[32760]: scrub summary: idle 2024-04-25T01:36:53.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:53 smithi027 ceph-mon[29197]: scrub summary: idle 2024-04-25T01:36:54.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:54 smithi153 ceph-mon[32621]: pgmap v357: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-25T01:36:54.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:54 smithi153 ceph-mon[32621]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:36:54.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:54 smithi139 ceph-mon[32760]: pgmap v357: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-25T01:36:54.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:54 smithi139 ceph-mon[32760]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:36:54.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:54 smithi027 ceph-mon[29197]: pgmap v357: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-25T01:36:54.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:54 smithi027 ceph-mon[29197]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:36:56.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:56 smithi153 ceph-mon[32621]: pgmap v358: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-25T01:36:56.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:56 smithi139 ceph-mon[32760]: pgmap v358: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-25T01:36:56.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:56 smithi027 ceph-mon[29197]: pgmap v358: 129 pgs: 129 active+clean; 5.6 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-25T01:36:58.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:36:58 smithi153 ceph-mon[32621]: pgmap v359: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 16 op/s 2024-04-25T01:36:58.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:36:58 smithi139 ceph-mon[32760]: pgmap v359: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 16 op/s 2024-04-25T01:36:58.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:36:58 smithi027 ceph-mon[29197]: pgmap v359: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 16 op/s 2024-04-25T01:37:00.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:00 smithi153 ceph-mon[32621]: pgmap v360: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-25T01:37:00.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:00 smithi139 ceph-mon[32760]: pgmap v360: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-25T01:37:00.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:00 smithi027 ceph-mon[29197]: pgmap v360: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-25T01:37:02.771 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:02 smithi153 ceph-mon[32621]: pgmap v361: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 62 KiB/s wr, 32 op/s 2024-04-25T01:37:02.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:02 smithi139 ceph-mon[32760]: pgmap v361: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 62 KiB/s wr, 32 op/s 2024-04-25T01:37:02.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:02 smithi027 ceph-mon[29197]: pgmap v361: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 62 KiB/s wr, 32 op/s 2024-04-25T01:37:04.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:04 smithi153 ceph-mon[32621]: pgmap v362: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:37:04.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:04 smithi139 ceph-mon[32760]: pgmap v362: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:37:04.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:04 smithi027 ceph-mon[29197]: pgmap v362: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:37:06.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:06 smithi153 ceph-mon[32621]: pgmap v363: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:37:06.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:06 smithi139 ceph-mon[32760]: pgmap v363: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:37:06.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:06 smithi027 ceph-mon[29197]: pgmap v363: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 KiB/s wr, 26 op/s 2024-04-25T01:37:08.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:08 smithi153 ceph-mon[32621]: pgmap v364: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 52 KiB/s wr, 29 op/s 2024-04-25T01:37:08.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:08 smithi153 ceph-mon[32621]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T01:37:08.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:08 smithi139 ceph-mon[32760]: pgmap v364: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 52 KiB/s wr, 29 op/s 2024-04-25T01:37:08.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:08 smithi139 ceph-mon[32760]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T01:37:08.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:08 smithi027 ceph-mon[29197]: pgmap v364: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 52 KiB/s wr, 29 op/s 2024-04-25T01:37:08.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:08 smithi027 ceph-mon[29197]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T01:37:10.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:10 smithi153 ceph-mon[32621]: pgmap v366: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 60 KiB/s wr, 28 op/s 2024-04-25T01:37:10.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:10 smithi139 ceph-mon[32760]: pgmap v366: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 60 KiB/s wr, 28 op/s 2024-04-25T01:37:10.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:10 smithi027 ceph-mon[29197]: pgmap v366: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 60 KiB/s wr, 28 op/s 2024-04-25T01:37:12.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:12 smithi153 ceph-mon[32621]: pgmap v367: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:12.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:12 smithi139 ceph-mon[32760]: pgmap v367: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:12.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:12 smithi027 ceph-mon[29197]: pgmap v367: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:14.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:14 smithi153 ceph-mon[32621]: pgmap v368: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:14.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:14 smithi139 ceph-mon[32760]: pgmap v368: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:14.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:14 smithi027 ceph-mon[29197]: pgmap v368: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:16.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:16 smithi153 ceph-mon[32621]: pgmap v369: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:16.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:16 smithi153 ceph-mon[32621]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T01:37:16.788 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:16 smithi139 ceph-mon[32760]: pgmap v369: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:16.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:16 smithi139 ceph-mon[32760]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T01:37:16.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:16 smithi027 ceph-mon[29197]: pgmap v369: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-25T01:37:16.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:16 smithi027 ceph-mon[29197]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T01:37:18.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:18 smithi153 ceph-mon[32621]: pgmap v371: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:37:18.807 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:18 smithi139 ceph-mon[32760]: pgmap v371: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:37:18.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:18 smithi027 ceph-mon[29197]: pgmap v371: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:37:20.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:20 smithi153 ceph-mon[32621]: pgmap v372: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:37:20.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:20 smithi139 ceph-mon[32760]: pgmap v372: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:37:20.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:20 smithi027 ceph-mon[29197]: pgmap v372: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-25T01:37:21.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:37:21.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:37:21.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.749+0000 7f5f58f6a640 1 -- 172.21.15.27:0/2864722034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f54101710 msgr2=0x7f5f54101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.749+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/2864722034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f54101710 0x7f5f54101b90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f5f48009930 tx=0x7f5f4802a420 comp rx=0 tx=0).stop 2024-04-25T01:37:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 -- 172.21.15.27:0/2864722034 shutdown_connections 2024-04-25T01:37:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/2864722034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f541020d0 0x7f5f5410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/2864722034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f54101710 0x7f5f54101b90 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/2864722034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f54107720 0x7f5f54107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 -- 172.21.15.27:0/2864722034 >> 172.21.15.27:0/2864722034 conn(0x7f5f540fd180 msgr2=0x7f5f540ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 -- 172.21.15.27:0/2864722034 shutdown_connections 2024-04-25T01:37:21.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 -- 172.21.15.27:0/2864722034 wait complete. 2024-04-25T01:37:21.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 Processor -- start 2024-04-25T01:37:21.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.750+0000 7f5f58f6a640 1 -- start start 2024-04-25T01:37:21.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f58f6a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f54101710 0x7f5f541980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f58f6a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f541020d0 0x7f5f54198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f58f6a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f54107720 0x7f5f5419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f58f6a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f54111b30 con 0x7f5f541020d0 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f58f6a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5f541119b0 con 0x7f5f54107720 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f58f6a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f54111cb0 con 0x7f5f54101710 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f541020d0 0x7f5f54198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f541020d0 0x7f5f54198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34760/0 (socket says 172.21.15.27:34760) 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 -- 172.21.15.27:0/4174123477 learned_addr learned my addr 172.21.15.27:0/4174123477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f5259c640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f54101710 0x7f5f541980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f54107720 0x7f5f5419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 -- 172.21.15.27:0/4174123477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f54101710 msgr2=0x7f5f541980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f54101710 0x7f5f541980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 -- 172.21.15.27:0/4174123477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f54107720 msgr2=0x7f5f5419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f54107720 0x7f5f5419c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 -- 172.21.15.27:0/4174123477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5419d0c0 con 0x7f5f541020d0 2024-04-25T01:37:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f54107720 0x7f5f5419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.751+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f541020d0 0x7f5f54198610 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f5f48009230 tx=0x7f5f48004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.752+0000 7f5f437fe640 1 -- 172.21.15.27:0/4174123477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4803f030 con 0x7f5f541020d0 2024-04-25T01:37:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.752+0000 7f5f58f6a640 1 -- 172.21.15.27:0/4174123477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f5419d350 con 0x7f5f541020d0 2024-04-25T01:37:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.752+0000 7f5f437fe640 1 -- 172.21.15.27:0/4174123477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f48003ad0 con 0x7f5f541020d0 2024-04-25T01:37:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.752+0000 7f5f437fe640 1 -- 172.21.15.27:0/4174123477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48035070 con 0x7f5f541020d0 2024-04-25T01:37:21.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.752+0000 7f5f58f6a640 1 -- 172.21.15.27:0/4174123477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f541a4c40 con 0x7f5f541020d0 2024-04-25T01:37:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.754+0000 7f5f437fe640 1 -- 172.21.15.27:0/4174123477 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5f48042030 con 0x7f5f541020d0 2024-04-25T01:37:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.754+0000 7f5f5bc70740 1 Processor -- start 2024-04-25T01:37:21.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f5bc70740 1 -- start start 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa6198a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198f50 0x563fa61a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 0x563fa61a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563fa60fd6b0 con 0x563fa6198630 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563fa60a5880 con 0x563fa6198f50 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563fa60f9670 con 0x563fa61a3620 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f437fe640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f3c0a2d00 0x7f5f3c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa6198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa6198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34770/0 (socket says 172.21.15.27:34770) 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 -- 172.21.15.27:0/1478852185 learned_addr learned my addr 172.21.15.27:0/1478852185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f5259c640 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198f50 0x563fa61a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f3c0a2d00 0x7f5f3c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f437fe640 1 -- 172.21.15.27:0/4174123477 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f5f480efe80 con 0x7f5f541020d0 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 0x563fa61a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 -- 172.21.15.27:0/1478852185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 msgr2=0x563fa61a5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 0x563fa61a5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 -- 172.21.15.27:0/1478852185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198f50 msgr2=0x563fa61a30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198f50 0x563fa61a30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.755+0000 7f5f52d9d640 1 -- 172.21.15.27:0/1478852185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fa5edb7d0 con 0x563fa6198630 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.756+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f3c0a2d00 0x7f5f3c0a51c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c010ca0 tx=0x7f5f4c01d000 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.756+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa6198a10 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c012e10 tx=0x7f5f4c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae29889498e83843 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.756+0000 7f5f40ff9640 1 -- 172.21.15.27:0/1478852185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c0128f0 con 0x563fa6198630 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.756+0000 7f5f40ff9640 1 -- 172.21.15.27:0/1478852185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f4c009d80 con 0x563fa6198630 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.756+0000 7f5f40ff9640 1 -- 172.21.15.27:0/1478852185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c005670 con 0x563fa6198630 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1478852185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 msgr2=0x563fa6198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa6198a10 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c012e10 tx=0x7f5f4c0107a0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1478852185 shutdown_connections 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 0x563fa61a5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198f50 0x563fa61a30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1478852185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa6198a10 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1478852185 >> 172.21.15.27:0/1478852185 conn(0x563fa6194950 msgr2=0x563fa6194d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1478852185 shutdown_connections 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1478852185 wait complete. 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 -- start start 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 Processor -- start 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 0x563fa61a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.757+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 0x563fa61a1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563fa61a6350 con 0x563fa6198f50 2024-04-25T01:37:21.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x563fa60b8f50 con 0x563fa6198630 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563fa61a64d0 con 0x563fa61a3620 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5bc70740 10 client.0 _ll_register_callbacks cb 0x563fa60f4000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34782/0 (socket says 172.21.15.27:34782) 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 -- 172.21.15.27:0/460728105 learned_addr learned my addr 172.21.15.27:0/460728105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 0x563fa61a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5259c640 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 0x563fa61a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 -- 172.21.15.27:0/460728105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 msgr2=0x563fa61a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 0x563fa61a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 -- 172.21.15.27:0/460728105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 msgr2=0x563fa61a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 0x563fa61a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 -- 172.21.15.27:0/460728105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fa5e113f0 con 0x563fa6198f50 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61a0b20 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f5f480b8410 tx=0x7f5f480bbaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e9e5f1bc4a774 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:21.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f39ffb640 1 -- 172.21.15.27:0/460728105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48049040 con 0x563fa6198f50 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f39ffb640 1 -- 172.21.15.27:0/460728105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f5f480da880 con 0x563fa6198f50 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5bc70740 1 -- 172.21.15.27:0/460728105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f4c009d80 con 0x563fa6198f50 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f39ffb640 1 -- 172.21.15.27:0/460728105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f4809f050 con 0x563fa6198f50 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5bc70740 10 client.18165 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.758+0000 7f5f5bc70740 1 -- 172.21.15.27:0/460728105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563fa5dd10f0 con 0x563fa6198f50 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f39ffb640 1 -- 172.21.15.27:0/460728105 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f4c009d80 con 0x563fa6198f50 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f39ffb640 1 -- 172.21.15.27:0/460728105 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f5f480b8620 con 0x563fa6198f50 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f5bc70740 10 client.18165 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f5bc70740 10 client.18165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f5bc70740 20 client.18165 populate_metadata read hostname 'smithi027' 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563fa61b38b0 0x563fa61b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f5bc70740 4 client.18165 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f5bc70740 1 -- 172.21.15.27:0/460728105 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fa610abf0 con 0x563fa61b38b0 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.759+0000 7f5f5259c640 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563fa61b38b0 0x563fa61b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.760+0000 7f5f5259c640 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563fa61b38b0 0x563fa61b5d30 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d87c7c6247bc9d4 server_cookie=76de4af61c9f0db in_seq=0 out_seq=0 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.760+0000 7f5f39ffb640 10 client.18165 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.760+0000 7f5f39ffb640 1 -- 172.21.15.27:0/460728105 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563fa610abf0 con 0x563fa61b38b0 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.760+0000 7f5f39ffb640 10 client.18165 handle_command_reply: tid=0 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.760+0000 7f5f5bc70740 1 client.18165 shutdown 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 -- 172.21.15.27:0/460728105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 msgr2=0x563fa61a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61a0b20 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f5f480b8410 tx=0x7f5f480bbaf0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 -- 172.21.15.27:0/460728105 shutdown_connections 2024-04-25T01:37:21.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563fa61b38b0 0x563fa61b5d30 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa61a3620 0x563fa61a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61a0b20 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f39ffb640 0 client.18165 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:37:21.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/460728105 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 0x563fa61a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 -- 172.21.15.27:0/460728105 >> 172.21.15.27:0/460728105 conn(0x563fa6194950 msgr2=0x563fa619a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:21.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 -- 172.21.15.27:0/460728105 shutdown_connections 2024-04-25T01:37:21.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 1 -- 172.21.15.27:0/460728105 wait complete. 2024-04-25T01:37:21.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.761+0000 7f5f5bc70740 20 client.18165 trim_cache size 0 max 16384 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.778+0000 7f5f5bc70740 1 Processor -- start 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.778+0000 7f5f5bc70740 1 -- start start 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6194950 0x563fa6194270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 0x563fa61af210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61bb640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f541b1640 con 0x563fa6198f50 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5f541aeee0 con 0x563fa6198630 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f541acd50 con 0x563fa6194950 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61bb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61bb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34798/0 (socket says 172.21.15.27:34798) 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 -- 172.21.15.27:0/3627529741 learned_addr learned my addr 172.21.15.27:0/3627529741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f5259c640 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6194950 0x563fa6194270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 0x563fa61af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 -- 172.21.15.27:0/3627529741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6194950 msgr2=0x563fa6194270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6194950 0x563fa6194270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 -- 172.21.15.27:0/3627529741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 msgr2=0x563fa61af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 0x563fa61af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 -- 172.21.15.27:0/3627529741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fa61b63b0 con 0x563fa6198f50 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61bb640 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00dc60 tx=0x7f5f4c0051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f030a53e52c5685 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f3bfff640 1 -- 172.21.15.27:0/3627529741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c009d80 con 0x563fa6198f50 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f3bfff640 1 -- 172.21.15.27:0/3627529741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f4c034070 con 0x563fa6198f50 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.779+0000 7f5f3bfff640 1 -- 172.21.15.27:0/3627529741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c004f90 con 0x563fa6198f50 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 -- 172.21.15.27:0/3627529741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 msgr2=0x563fa61bb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61bb640 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00dc60 tx=0x7f5f4c0051d0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 -- 172.21.15.27:0/3627529741 shutdown_connections 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198f50 0x563fa61bb640 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6198630 0x563fa61af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/3627529741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6194950 0x563fa6194270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 -- 172.21.15.27:0/3627529741 >> 172.21.15.27:0/3627529741 conn(0x563fa619f240 msgr2=0x563fa61a0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 -- 172.21.15.27:0/3627529741 shutdown_connections 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 -- 172.21.15.27:0/3627529741 wait complete. 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 -- start start 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.780+0000 7f5f5bc70740 1 Processor -- start 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6194950 0x563fa619d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa619de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6198f50 0x563fa6218eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f541ad3a0 con 0x563fa6198630 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5f541ae810 con 0x563fa6194950 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f541ada70 con 0x563fa6198f50 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 10 client.0 _ll_register_callbacks cb 0x7f5f48009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa619de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa619de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34802/0 (socket says 172.21.15.27:34802) 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 -- 172.21.15.27:0/1450685027 learned_addr learned my addr 172.21.15.27:0/1450685027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f52d9d640 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6198f50 0x563fa6218eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5259c640 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6194950 0x563fa619d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 -- 172.21.15.27:0/1450685027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6198f50 msgr2=0x563fa6218eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6198f50 0x563fa6218eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 -- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6194950 msgr2=0x563fa619d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6194950 0x563fa619d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 -- 172.21.15.27:0/1450685027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fa62193f0 con 0x563fa6198630 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5259c640 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6194950 0x563fa619d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f51d9b640 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa619de70 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f5f480c46f0 tx=0x7f5f48031ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20cf522b12fa6662 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f39ffb640 1 -- 172.21.15.27:0/1450685027 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48049070 con 0x563fa6198630 2024-04-25T01:37:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f39ffb640 1 -- 172.21.15.27:0/1450685027 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f5f480a50d0 con 0x563fa6198630 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1450685027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f4c034070 con 0x563fa6198630 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f39ffb640 1 -- 172.21.15.27:0/1450685027 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f480c4c70 con 0x563fa6198630 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.781+0000 7f5f5bc70740 10 client.18171 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1450685027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563fa621a660 con 0x563fa6198630 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f39ffb640 1 -- 172.21.15.27:0/1450685027 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f4c034070 con 0x563fa6198630 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f39ffb640 1 -- 172.21.15.27:0/1450685027 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f5f480bb400 con 0x563fa6198630 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5bc70740 10 client.18171 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5bc70740 10 client.18171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5bc70740 20 client.18171 populate_metadata read hostname 'smithi027' 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563fa61a3620 0x563fa621cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5bc70740 4 client.18171 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1450685027 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563fa6104f40 con 0x563fa61a3620 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5259c640 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563fa61a3620 0x563fa621cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f5259c640 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563fa61a3620 0x563fa621cb40 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a748ceac9cd0bc25 server_cookie=30de576ae701bb5b in_seq=0 out_seq=0 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.782+0000 7f5f39ffb640 10 client.18171 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f39ffb640 1 -- 172.21.15.27:0/1450685027 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563fa6104f40 con 0x563fa61a3620 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f39ffb640 10 client.18171 handle_command_reply: tid=0 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 client.18171 shutdown 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1450685027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 msgr2=0x563fa619de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa619de70 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f5f480c46f0 tx=0x7f5f48031ae0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1450685027 shutdown_connections 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x563fa61a3620 0x563fa621cb40 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563fa6198f50 0x563fa6218eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563fa6198630 0x563fa619de70 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 --2- 172.21.15.27:0/1450685027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x563fa6194950 0x563fa619d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1450685027 >> 172.21.15.27:0/1450685027 conn(0x563fa619f240 msgr2=0x563fa6199dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f39ffb640 0 client.18171 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:37:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1450685027 shutdown_connections 2024-04-25T01:37:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 1 -- 172.21.15.27:0/1450685027 wait complete. 2024-04-25T01:37:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f5bc70740 20 client.18171 trim_cache size 0 max 16384 2024-04-25T01:37:21.766 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:37:21.766 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T01:37:21.766 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T01:37:21.766 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:37:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f58f6a640 1 -- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f3c0a2d00 msgr2=0x7f5f3c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f3c0a2d00 0x7f5f3c0a51c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c010ca0 tx=0x7f5f4c01d000 comp rx=0 tx=0).stop 2024-04-25T01:37:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f58f6a640 1 -- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f541020d0 msgr2=0x7f5f54198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.783+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f541020d0 0x7f5f54198610 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f5f48009230 tx=0x7f5f48004270 comp rx=0 tx=0).stop 2024-04-25T01:37:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.784+0000 7f5f58f6a640 1 -- 172.21.15.27:0/4174123477 shutdown_connections 2024-04-25T01:37:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.784+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f54107720 0x7f5f5419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.784+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5f3c0a2d00 0x7f5f3c0a51c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.784+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f541020d0 0x7f5f54198610 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.784+0000 7f5f58f6a640 1 --2- 172.21.15.27:0/4174123477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f54101710 0x7f5f541980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.784+0000 7f5f58f6a640 1 -- 172.21.15.27:0/4174123477 >> 172.21.15.27:0/4174123477 conn(0x7f5f540fd180 msgr2=0x7f5f540fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.784+0000 7f5f58f6a640 1 -- 172.21.15.27:0/4174123477 shutdown_connections 2024-04-25T01:37:21.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.784+0000 7f5f58f6a640 1 -- 172.21.15.27:0/4174123477 wait complete. 2024-04-25T01:37:21.771 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:37:21.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:37:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 -- 172.21.15.27:0/2223354110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa4103340 msgr2=0x7f4aa410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/2223354110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa4103340 0x7f4aa410b820 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009900 tx=0x7f4a8c02a480 comp rx=0 tx=0).stop 2024-04-25T01:37:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 -- 172.21.15.27:0/2223354110 shutdown_connections 2024-04-25T01:37:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/2223354110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4aa410bd60 0x7f4aa410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/2223354110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa4103340 0x7f4aa410b820 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/2223354110 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4aa4102a20 0x7f4aa4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 -- 172.21.15.27:0/2223354110 >> 172.21.15.27:0/2223354110 conn(0x7f4aa40777a0 msgr2=0x7f4aa4077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 -- 172.21.15.27:0/2223354110 shutdown_connections 2024-04-25T01:37:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.899+0000 7f4aaa66b640 1 -- 172.21.15.27:0/2223354110 wait complete. 2024-04-25T01:37:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aaa66b640 1 Processor -- start 2024-04-25T01:37:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aaa66b640 1 -- start start 2024-04-25T01:37:21.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aaa66b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4aa4102a20 0x7f4aa41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aaa66b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4aa4103340 0x7f4aa4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aaa66b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa410bd60 0x7f4aa419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aaa66b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111b30 con 0x7f4aa410bd60 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aaa66b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41119b0 con 0x7f4aa4103340 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aaa66b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111cb0 con 0x7f4aa4102a20 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aa8c08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa410bd60 0x7f4aa419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aa8c08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa410bd60 0x7f4aa419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34818/0 (socket says 172.21.15.27:34818) 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.900+0000 7f4aa8c08640 1 -- 172.21.15.27:0/132557385 learned_addr learned my addr 172.21.15.27:0/132557385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa3fff640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4aa4102a20 0x7f4aa41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa8c08640 1 -- 172.21.15.27:0/132557385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4aa4102a20 msgr2=0x7f4aa41980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa8c08640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4aa4102a20 0x7f4aa41980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa8c08640 1 -- 172.21.15.27:0/132557385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4aa4103340 msgr2=0x7f4aa4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa8c08640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4aa4103340 0x7f4aa4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa8c08640 1 -- 172.21.15.27:0/132557385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa419d070 con 0x7f4aa410bd60 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa8c08640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa410bd60 0x7f4aa419c940 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00e960 tx=0x7f4a9c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa17fa640 1 -- 172.21.15.27:0/132557385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c004d20 con 0x7f4aa410bd60 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa17fa640 1 -- 172.21.15.27:0/132557385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c005020 con 0x7f4aa410bd60 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aa17fa640 1 -- 172.21.15.27:0/132557385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0116d0 con 0x7f4aa410bd60 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa419d360 con 0x7f4aa410bd60 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.901+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41a4bf0 con 0x7f4aa410bd60 2024-04-25T01:37:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.902+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa40689b0 con 0x7f4aa410bd60 2024-04-25T01:37:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.903+0000 7f4aa17fa640 1 -- 172.21.15.27:0/132557385 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4a9c005560 con 0x7f4aa410bd60 2024-04-25T01:37:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.904+0000 7f4aa17fa640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4a900a2d80 0x7f4a900a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.904+0000 7f4aa3fff640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4a900a2d80 0x7f4a900a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.904+0000 7f4aa17fa640 1 -- 172.21.15.27:0/132557385 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f4a9c0cfce0 con 0x7f4aa410bd60 2024-04-25T01:37:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.904+0000 7f4aa3fff640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4a900a2d80 0x7f4a900a5240 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4a940058c0 tx=0x7f4a94005c60 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:21.908+0000 7f4aa17fa640 1 -- 172.21.15.27:0/132557385 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a9c016030 con 0x7f4aa410bd60 2024-04-25T01:37:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.137+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4aa4106950 con 0x7f4aa410bd60 2024-04-25T01:37:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.137+0000 7f4aa17fa640 1 -- 172.21.15.27:0/132557385 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11883 (secure 0 0 0) 0x7f4a9c09c510 con 0x7f4aa410bd60 2024-04-25T01:37:22.114 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:37:22.114 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:36:52.671392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24505,24517]},"id":1}]} 2024-04-25T01:37:22.115 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T01:37:22.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4a900a2d80 msgr2=0x7f4a900a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4a900a2d80 0x7f4a900a5240 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4a940058c0 tx=0x7f4a94005c60 comp rx=0 tx=0).stop 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa410bd60 msgr2=0x7f4aa419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa410bd60 0x7f4aa419c940 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00e960 tx=0x7f4a9c00ee20 comp rx=0 tx=0).stop 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 shutdown_connections 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f4a900a2d80 0x7f4a900a5240 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aa410bd60 0x7f4aa419c940 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4aa4103340 0x7f4aa4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 --2- 172.21.15.27:0/132557385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4aa4102a20 0x7f4aa41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 >> 172.21.15.27:0/132557385 conn(0x7f4aa40777a0 msgr2=0x7f4aa40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 shutdown_connections 2024-04-25T01:37:22.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.140+0000 7f4aaa66b640 1 -- 172.21.15.27:0/132557385 wait complete. 2024-04-25T01:37:22.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:37:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- 172.21.15.27:0/2797215246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff134101710 msgr2=0x7ff134101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 --2- 172.21.15.27:0/2797215246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff134101710 0x7ff134101b90 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ff120009930 tx=0x7ff12002a420 comp rx=0 tx=0).stop 2024-04-25T01:37:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- 172.21.15.27:0/2797215246 shutdown_connections 2024-04-25T01:37:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 --2- 172.21.15.27:0/2797215246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1341020d0 0x7ff13410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 --2- 172.21.15.27:0/2797215246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff134101710 0x7ff134101b90 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 --2- 172.21.15.27:0/2797215246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff134107720 0x7ff134107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- 172.21.15.27:0/2797215246 >> 172.21.15.27:0/2797215246 conn(0x7ff1340fd180 msgr2=0x7ff1340ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- 172.21.15.27:0/2797215246 shutdown_connections 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- 172.21.15.27:0/2797215246 wait complete. 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 Processor -- start 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- start start 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff134101710 0x7ff134197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1341020d0 0x7ff134198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff134107720 0x7ff13419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff134111b30 con 0x7ff1341020d0 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341119b0 con 0x7ff134107720 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.256+0000 7ff13ab63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff134111cb0 con 0x7ff134101710 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1341020d0 0x7ff134198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1341020d0 0x7ff134198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34844/0 (socket says 172.21.15.27:34844) 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 -- 172.21.15.27:0/3717184586 learned_addr learned my addr 172.21.15.27:0/3717184586 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff1388ff640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff134101710 0x7ff134197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff139100640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff134107720 0x7ff13419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 -- 172.21.15.27:0/3717184586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff134101710 msgr2=0x7ff134197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff134101710 0x7ff134197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 -- 172.21.15.27:0/3717184586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff134107720 msgr2=0x7ff13419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff134107720 0x7ff13419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 -- 172.21.15.27:0/3717184586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13419cf40 con 0x7ff1341020d0 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff133fff640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1341020d0 0x7ff134198490 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff120009a50 tx=0x7ff120004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff131ffb640 1 -- 172.21.15.27:0/3717184586 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12003f030 con 0x7ff1341020d0 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff131ffb640 1 -- 172.21.15.27:0/3717184586 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff120003ad0 con 0x7ff1341020d0 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff131ffb640 1 -- 172.21.15.27:0/3717184586 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120035070 con 0x7ff1341020d0 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.257+0000 7ff13ab63640 1 -- 172.21.15.27:0/3717184586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13419d1d0 con 0x7ff1341020d0 2024-04-25T01:37:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.258+0000 7ff13ab63640 1 -- 172.21.15.27:0/3717184586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1341a4ac0 con 0x7ff1341020d0 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff131ffb640 1 -- 172.21.15.27:0/3717184586 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff120042030 con 0x7ff1341020d0 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff13d877740 1 Processor -- start 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff13d877740 1 -- start start 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff13d877740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 0x561fd2db3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff13d877740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3ea0 0x561fd2dbe030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff13d877740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2dc0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff13d877740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561fd2d18670 con 0x561fd2dbe570 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff13d877740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561fd2cc0970 con 0x561fd2db3ea0 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.260+0000 7ff13d877740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561fd2d14630 con 0x561fd2db3580 2024-04-25T01:37:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2dc0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2dc0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34850/0 (socket says 172.21.15.27:34850) 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 -- 172.21.15.27:0/461800833 learned_addr learned my addr 172.21.15.27:0/461800833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff131ffb640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1240a2d00 0x7ff1240a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff139100640 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 0x561fd2db3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff1388ff640 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3ea0 0x561fd2dbe030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff139100640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1240a2d00 0x7ff1240a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff131ffb640 1 -- 172.21.15.27:0/3717184586 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7ff1200f0060 con 0x7ff1341020d0 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 -- 172.21.15.27:0/461800833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 msgr2=0x561fd2db3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 0x561fd2db3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 -- 172.21.15.27:0/461800833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3ea0 msgr2=0x561fd2dbe030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3ea0 0x561fd2dbe030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 -- 172.21.15.27:0/461800833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fd2ce4400 con 0x561fd2dbe570 2024-04-25T01:37:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff133fff640 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2dc0960 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff1200b8590 tx=0x7ff120009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafaaed753f19dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff139100640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1240a2d00 0x7ff1240a51c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00afd0 tx=0x7ff12c005e90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff11effd640 1 -- 172.21.15.27:0/461800833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120049020 con 0x561fd2dbe570 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff11effd640 1 -- 172.21.15.27:0/461800833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1200bc130 con 0x561fd2dbe570 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.261+0000 7ff11effd640 1 -- 172.21.15.27:0/461800833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120003d50 con 0x561fd2dbe570 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 -- 172.21.15.27:0/461800833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 msgr2=0x561fd2dc0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2dc0960 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff1200b8590 tx=0x7ff120009680 comp rx=0 tx=0).stop 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 -- 172.21.15.27:0/461800833 shutdown_connections 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2dc0960 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3ea0 0x561fd2dbe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 --2- 172.21.15.27:0/461800833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 0x561fd2db3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 -- 172.21.15.27:0/461800833 >> 172.21.15.27:0/461800833 conn(0x561fd2daf8a0 msgr2=0x561fd2dafc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 -- 172.21.15.27:0/461800833 shutdown_connections 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 -- 172.21.15.27:0/461800833 wait complete. 2024-04-25T01:37:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.262+0000 7ff13d877740 1 -- start start 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff13d877740 1 Processor -- start 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff13d877740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3580 0x561fd2dbdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff13d877740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 0x561fd2dbb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff13d877740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 0x561fd2dbb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff13d877740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561fd2d18670 con 0x561fd2db3580 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff13d877740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x561fd2cd4050 con 0x561fd2dbe570 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff13d877740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561fd2dc1120 con 0x561fd2db3ea0 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3580 0x561fd2dbdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff13d877740 10 client.0 _ll_register_callbacks cb 0x561fd2a2bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3580 0x561fd2dbdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34862/0 (socket says 172.21.15.27:34862) 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 -- 172.21.15.27:0/1794104662 learned_addr learned my addr 172.21.15.27:0/1794104662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff139100640 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 0x561fd2dbb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 -- 172.21.15.27:0/1794104662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 msgr2=0x561fd2dbb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 0x561fd2dbb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 -- 172.21.15.27:0/1794104662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 msgr2=0x561fd2dbb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 0x561fd2dbb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 -- 172.21.15.27:0/1794104662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fd2ca5dc0 con 0x561fd2db3580 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff1388ff640 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3580 0x561fd2dbdef0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ff128010a50 tx=0x7ff128010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d925d418207b47d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.263+0000 7ff113fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff113fff640 1 -- 172.21.15.27:0/1794104662 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12801e060 con 0x561fd2db3580 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff113fff640 1 -- 172.21.15.27:0/1794104662 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7ff12800fbd0 con 0x561fd2db3580 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 1 -- 172.21.15.27:0/1794104662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1200bc130 con 0x561fd2db3580 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff113fff640 1 -- 172.21.15.27:0/1794104662 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff128025050 con 0x561fd2db3580 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 10 client.18189 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 1 -- 172.21.15.27:0/1794104662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561fd2a2c3f0 con 0x561fd2db3580 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff113fff640 1 -- 172.21.15.27:0/1794104662 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1200bc130 con 0x561fd2db3580 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff113fff640 1 -- 172.21.15.27:0/1794104662 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7ff128018400 con 0x561fd2db3580 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 10 client.18189 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 10 client.18189 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 20 client.18189 populate_metadata read hostname 'smithi027' 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561fd2dce350 0x561fd2dd07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 4 client.18189 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.264+0000 7ff13d877740 1 -- 172.21.15.27:0/1794104662 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fd2d25bb0 con 0x561fd2dce350 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.265+0000 7ff133fff640 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561fd2dce350 0x561fd2dd07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.265+0000 7ff113fff640 10 client.18189 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.265+0000 7ff133fff640 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561fd2dce350 0x561fd2dd07c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=482a84169de302a0 server_cookie=3700329d7ff5f5be in_seq=0 out_seq=0 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff113fff640 1 -- 172.21.15.27:0/1794104662 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fd2d25bb0 con 0x561fd2dce350 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff113fff640 10 client.18189 handle_command_reply: tid=0 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 client.18189 shutdown 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 -- 172.21.15.27:0/1794104662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3580 msgr2=0x561fd2dbdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3580 0x561fd2dbdef0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ff128010a50 tx=0x7ff128010f10 comp rx=0 tx=0).stop 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 -- 172.21.15.27:0/1794104662 shutdown_connections 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561fd2dce350 0x561fd2dd07c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 0x561fd2dbb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 0x561fd2dbb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff113fff640 0 client.18189 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:37:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 --2- 172.21.15.27:0/1794104662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3580 0x561fd2dbdef0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 -- 172.21.15.27:0/1794104662 >> 172.21.15.27:0/1794104662 conn(0x561fd2daf8a0 msgr2=0x561fd2db56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 -- 172.21.15.27:0/1794104662 shutdown_connections 2024-04-25T01:37:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 1 -- 172.21.15.27:0/1794104662 wait complete. 2024-04-25T01:37:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.266+0000 7ff13d877740 20 client.18189 trim_cache size 0 max 16384 2024-04-25T01:37:22.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.279+0000 7ff13d877740 1 Processor -- start 2024-04-25T01:37:22.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.279+0000 7ff13d877740 1 -- start start 2024-04-25T01:37:22.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff13d877740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 0x561fd2dc9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff13d877740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3ea0 0x561fd2dc6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff13d877740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 0x561fd2dbda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff13d877740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341b1540 con 0x561fd2db3ea0 2024-04-25T01:37:22.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff13d877740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341aede0 con 0x561fd2dbe570 2024-04-25T01:37:22.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff13d877740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341acc50 con 0x561fd2db3580 2024-04-25T01:37:22.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3ea0 0x561fd2dc6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3ea0 0x561fd2dc6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34874/0 (socket says 172.21.15.27:34874) 2024-04-25T01:37:22.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 -- 172.21.15.27:0/3588253075 learned_addr learned my addr 172.21.15.27:0/3588253075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 -- 172.21.15.27:0/3588253075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 msgr2=0x561fd2dc9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 --2- 172.21.15.27:0/3588253075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 0x561fd2dc9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 -- 172.21.15.27:0/3588253075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 msgr2=0x561fd2dbda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 --2- 172.21.15.27:0/3588253075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 0x561fd2dbda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 -- 172.21.15.27:0/3588253075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fd2dbbad0 con 0x561fd2db3ea0 2024-04-25T01:37:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff133fff640 1 --2- 172.21.15.27:0/3588253075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3ea0 0x561fd2dc6780 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7ff1200b8b00 tx=0x7ff1200b8480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebc1f3cc65fff0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff11e7fc640 1 -- 172.21.15.27:0/3588253075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120049040 con 0x561fd2db3ea0 2024-04-25T01:37:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff11e7fc640 1 -- 172.21.15.27:0/3588253075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12003b460 con 0x561fd2db3ea0 2024-04-25T01:37:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.280+0000 7ff11e7fc640 1 -- 172.21.15.27:0/3588253075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1200dc070 con 0x561fd2db3ea0 2024-04-25T01:37:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 -- 172.21.15.27:0/3588253075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3ea0 msgr2=0x561fd2dc6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 --2- 172.21.15.27:0/3588253075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3ea0 0x561fd2dc6780 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7ff1200b8b00 tx=0x7ff1200b8480 comp rx=0 tx=0).stop 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 -- 172.21.15.27:0/3588253075 shutdown_connections 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 --2- 172.21.15.27:0/3588253075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2dbe570 0x561fd2dbda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 --2- 172.21.15.27:0/3588253075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2db3ea0 0x561fd2dc6780 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 --2- 172.21.15.27:0/3588253075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3580 0x561fd2dc9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 -- 172.21.15.27:0/3588253075 >> 172.21.15.27:0/3588253075 conn(0x561fd2daf8a0 msgr2=0x561fd2db6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 -- 172.21.15.27:0/3588253075 shutdown_connections 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 -- 172.21.15.27:0/3588253075 wait complete. 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 -- start start 2024-04-25T01:37:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.281+0000 7ff13d877740 1 Processor -- start 2024-04-25T01:37:22.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff13d877740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3580 0x561fd2db7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff13d877740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 0x561fd2db7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff13d877740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2db8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff13d877740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341ad2a0 con 0x561fd2dbe570 2024-04-25T01:37:22.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff13d877740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341ae710 con 0x561fd2db3580 2024-04-25T01:37:22.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff13d877740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341ad970 con 0x561fd2db3ea0 2024-04-25T01:37:22.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff13d877740 10 client.0 _ll_register_callbacks cb 0x7ff12800a740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:22.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2db8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2db8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34890/0 (socket says 172.21.15.27:34890) 2024-04-25T01:37:22.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 -- 172.21.15.27:0/3309347865 learned_addr learned my addr 172.21.15.27:0/3309347865 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff133fff640 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 0x561fd2db7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff1388ff640 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3580 0x561fd2db7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 -- 172.21.15.27:0/3309347865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 msgr2=0x561fd2db7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 0x561fd2db7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 -- 172.21.15.27:0/3309347865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3580 msgr2=0x561fd2db7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3580 0x561fd2db7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 -- 172.21.15.27:0/3309347865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fd2af67d0 con 0x561fd2dbe570 2024-04-25T01:37:22.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff139100640 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2db8200 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00f730 tx=0x7ff12c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71b112ac79fc709 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff113fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:37:22.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff113fff640 1 -- 172.21.15.27:0/3309347865 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c01cb50 con 0x561fd2dbe570 2024-04-25T01:37:22.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff113fff640 1 -- 172.21.15.27:0/3309347865 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7ff12c01ccf0 con 0x561fd2dbe570 2024-04-25T01:37:22.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.282+0000 7ff13d877740 1 -- 172.21.15.27:0/3309347865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff12003b460 con 0x561fd2dbe570 2024-04-25T01:37:22.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff113fff640 1 -- 172.21.15.27:0/3309347865 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff12c032050 con 0x561fd2dbe570 2024-04-25T01:37:22.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff13d877740 10 client.18195 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:22.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff13d877740 1 -- 172.21.15.27:0/3309347865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561fd2e39060 con 0x561fd2dbe570 2024-04-25T01:37:22.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff113fff640 1 -- 172.21.15.27:0/3309347865 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12003b460 con 0x561fd2dbe570 2024-04-25T01:37:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff113fff640 1 -- 172.21.15.27:0/3309347865 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7ff12c005840 con 0x561fd2dbe570 2024-04-25T01:37:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff13d877740 10 client.18195 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff13d877740 10 client.18195 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T01:37:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff13d877740 20 client.18195 populate_metadata read hostname 'smithi027' 2024-04-25T01:37:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff13d877740 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561fd2e397d0 0x561fd2e3bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff13d877740 4 client.18195 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:37:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff13d877740 1 -- 172.21.15.27:0/3309347865 --> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fd2d1ff00 con 0x561fd2e397d0 2024-04-25T01:37:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.283+0000 7ff1388ff640 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561fd2e397d0 0x561fd2e3bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff1388ff640 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561fd2e397d0 0x561fd2e3bbf0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=10bf1768a0379089 server_cookie=4a09f8dd013c596d in_seq=0 out_seq=0 2024-04-25T01:37:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff113fff640 10 client.18195 ms_handle_connect on v2:172.21.15.153:6836/529371680 2024-04-25T01:37:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff113fff640 1 -- 172.21.15.27:0/3309347865 <== mds.2 v2:172.21.15.153:6836/529371680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fd2d1ff00 con 0x561fd2e397d0 2024-04-25T01:37:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff113fff640 10 client.18195 handle_command_reply: tid=0 2024-04-25T01:37:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 client.18195 shutdown 2024-04-25T01:37:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 -- 172.21.15.27:0/3309347865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 msgr2=0x561fd2db8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2db8200 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00f730 tx=0x7ff12c004a40 comp rx=0 tx=0).stop 2024-04-25T01:37:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 -- 172.21.15.27:0/3309347865 shutdown_connections 2024-04-25T01:37:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.153:6836/529371680,v1:172.21.15.153:6837/529371680] conn(0x561fd2e397d0 0x561fd2e3bbf0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fd2dbe570 0x561fd2db8200 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561fd2db3ea0 0x561fd2db7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff113fff640 0 client.18195 ms_handle_reset on v2:172.21.15.153:6836/529371680 2024-04-25T01:37:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 --2- 172.21.15.27:0/3309347865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x561fd2db3580 0x561fd2db7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 -- 172.21.15.27:0/3309347865 >> 172.21.15.27:0/3309347865 conn(0x561fd2daf8a0 msgr2=0x561fd2dae2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 -- 172.21.15.27:0/3309347865 shutdown_connections 2024-04-25T01:37:22.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 1 -- 172.21.15.27:0/3309347865 wait complete. 2024-04-25T01:37:22.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.284+0000 7ff13d877740 20 client.18195 trim_cache size 0 max 16384 2024-04-25T01:37:22.274 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:37:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 -- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1240a2d00 msgr2=0x7ff1240a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1240a2d00 0x7ff1240a51c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00afd0 tx=0x7ff12c005e90 comp rx=0 tx=0).stop 2024-04-25T01:37:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 -- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1341020d0 msgr2=0x7ff134198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1341020d0 0x7ff134198490 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff120009a50 tx=0x7ff120004270 comp rx=0 tx=0).stop 2024-04-25T01:37:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 -- 172.21.15.27:0/3717184586 shutdown_connections 2024-04-25T01:37:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff134107720 0x7ff13419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff1240a2d00 0x7ff1240a51c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1341020d0 0x7ff134198490 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 --2- 172.21.15.27:0/3717184586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff134101710 0x7ff134197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 -- 172.21.15.27:0/3717184586 >> 172.21.15.27:0/3717184586 conn(0x7ff1340fd180 msgr2=0x7ff1340fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 -- 172.21.15.27:0/3717184586 shutdown_connections 2024-04-25T01:37:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.285+0000 7ff13ab63640 1 -- 172.21.15.27:0/3717184586 wait complete. 2024-04-25T01:37:22.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:37:22.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 -- 172.21.15.27:0/3975022654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd901020d0 msgr2=0x7fbd9010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 --2- 172.21.15.27:0/3975022654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd901020d0 0x7fbd9010e5e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009cd0 tx=0x7fbd8802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 -- 172.21.15.27:0/3975022654 shutdown_connections 2024-04-25T01:37:22.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 --2- 172.21.15.27:0/3975022654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd901020d0 0x7fbd9010e5e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 --2- 172.21.15.27:0/3975022654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd90101710 0x7fbd90101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 --2- 172.21.15.27:0/3975022654 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd90107720 0x7fbd90107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 -- 172.21.15.27:0/3975022654 >> 172.21.15.27:0/3975022654 conn(0x7fbd900fd180 msgr2=0x7fbd900ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 -- 172.21.15.27:0/3975022654 shutdown_connections 2024-04-25T01:37:22.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 -- 172.21.15.27:0/3975022654 wait complete. 2024-04-25T01:37:22.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 Processor -- start 2024-04-25T01:37:22.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.404+0000 7fbd973c1640 1 -- start start 2024-04-25T01:37:22.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd973c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd90101710 0x7fbd90073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd973c1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd901020d0 0x7fbd90073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd973c1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd90107720 0x7fbd900744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd973c1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90111b30 con 0x7fbd90107720 2024-04-25T01:37:22.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd973c1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901119b0 con 0x7fbd901020d0 2024-04-25T01:37:22.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd973c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90111cb0 con 0x7fbd90101710 2024-04-25T01:37:22.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd90107720 0x7fbd900744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd90107720 0x7fbd900744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34902/0 (socket says 172.21.15.27:34902) 2024-04-25T01:37:22.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 -- 172.21.15.27:0/1925364141 learned_addr learned my addr 172.21.15.27:0/1925364141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9495c640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd901020d0 0x7fbd90073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9515d640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd90101710 0x7fbd90073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 -- 172.21.15.27:0/1925364141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd90101710 msgr2=0x7fbd90073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd90101710 0x7fbd90073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 -- 172.21.15.27:0/1925364141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd901020d0 msgr2=0x7fbd90073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd901020d0 0x7fbd90073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 -- 172.21.15.27:0/1925364141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd901a6430 con 0x7fbd90107720 2024-04-25T01:37:22.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd9595e640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd90107720 0x7fbd900744c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fbd88005cc0 tx=0x7fbd88003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd867fc640 1 -- 172.21.15.27:0/1925364141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88004040 con 0x7fbd90107720 2024-04-25T01:37:22.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd973c1640 1 -- 172.21.15.27:0/1925364141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd901a66c0 con 0x7fbd90107720 2024-04-25T01:37:22.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.405+0000 7fbd867fc640 1 -- 172.21.15.27:0/1925364141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd880046c0 con 0x7fbd90107720 2024-04-25T01:37:22.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.406+0000 7fbd867fc640 1 -- 172.21.15.27:0/1925364141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88030e70 con 0x7fbd90107720 2024-04-25T01:37:22.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.406+0000 7fbd973c1640 1 -- 172.21.15.27:0/1925364141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd901a6c10 con 0x7fbd90107720 2024-04-25T01:37:22.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd867fc640 1 -- 172.21.15.27:0/1925364141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbd880305a0 con 0x7fbd90107720 2024-04-25T01:37:22.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd9a1af740 1 Processor -- start 2024-04-25T01:37:22.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd9a1af740 1 -- start start 2024-04-25T01:37:22.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345ba2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 0x55d345bad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 0x55d345baf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d345b07670 con 0x55d345ba2580 2024-04-25T01:37:22.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d345aaf970 con 0x55d345ba2ea0 2024-04-25T01:37:22.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd867fc640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd7c0a2d00 0x7fbd7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.408+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d345b03630 con 0x55d345bad570 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345ba2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9595e640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd7c0a2d00 0x7fbd7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345ba2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34910/0 (socket says 172.21.15.27:34910) 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 -- 172.21.15.27:0/115928118 learned_addr learned my addr 172.21.15.27:0/115928118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd867fc640 1 -- 172.21.15.27:0/1925364141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fbd8802e030 con 0x7fbd90107720 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9515d640 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 0x55d345bad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 0x55d345baf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 -- 172.21.15.27:0/115928118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 msgr2=0x55d345baf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 0x55d345baf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 -- 172.21.15.27:0/115928118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 msgr2=0x55d345bad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 0x55d345bad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 -- 172.21.15.27:0/115928118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d345ad3400 con 0x55d345ba2580 2024-04-25T01:37:22.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9595e640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd7c0a2d00 0x7fbd7c0a51c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd880eddb0 tx=0x7fbd880095c0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd9495c640 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345ba2960 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000af10 tx=0x7fbd8000a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24a9643b476be15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd7b7fe640 1 -- 172.21.15.27:0/115928118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8000f650 con 0x55d345ba2580 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd7b7fe640 1 -- 172.21.15.27:0/115928118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd80017030 con 0x55d345ba2580 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.409+0000 7fbd7b7fe640 1 -- 172.21.15.27:0/115928118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8001b630 con 0x55d345ba2580 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 -- 172.21.15.27:0/115928118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 msgr2=0x55d345ba2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345ba2960 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000af10 tx=0x7fbd8000a800 comp rx=0 tx=0).stop 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 -- 172.21.15.27:0/115928118 shutdown_connections 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 0x55d345baf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 0x55d345bad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/115928118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345ba2960 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 -- 172.21.15.27:0/115928118 >> 172.21.15.27:0/115928118 conn(0x55d345b9e8a0 msgr2=0x55d345b9ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 -- 172.21.15.27:0/115928118 shutdown_connections 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 -- 172.21.15.27:0/115928118 wait complete. 2024-04-25T01:37:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 -- start start 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.410+0000 7fbd9a1af740 1 Processor -- start 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2580 0x55d345bacf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2ea0 0x55d345baa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345bad570 0x55d345baa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d345bb0360 con 0x55d345ba2ea0 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d345ac3050 con 0x55d345bad570 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d345bb04e0 con 0x55d345ba2580 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9a1af740 10 client.0 _ll_register_callbacks cb 0x55d34581abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2ea0 0x55d345baa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2ea0 0x55d345baa1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34924/0 (socket says 172.21.15.27:34924) 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 -- 172.21.15.27:0/2951457126 learned_addr learned my addr 172.21.15.27:0/2951457126 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9495c640 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2580 0x55d345bacf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9495c640 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345bad570 0x55d345baa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 -- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2580 msgr2=0x55d345bacf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2580 0x55d345bacf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 -- 172.21.15.27:0/2951457126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345bad570 msgr2=0x55d345baa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345bad570 0x55d345baa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 -- 172.21.15.27:0/2951457126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d345a94dc0 con 0x55d345ba2ea0 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9495c640 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2580 0x55d345bacf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd9515d640 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2ea0 0x55d345baa1f0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00be70 tx=0x7fbd8c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f923f017e84380a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.411+0000 7fbd78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2951457126 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c01f060 con 0x55d345ba2ea0 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2951457126 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fbd8c0108a0 con 0x55d345ba2ea0 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2951457126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd80017030 con 0x55d345ba2ea0 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2951457126 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd8c026050 con 0x55d345ba2ea0 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 10 client.18207 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2951457126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d34581b3f0 con 0x55d345ba2ea0 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2951457126 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd80017030 con 0x55d345ba2ea0 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2951457126 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fbd8c019030 con 0x55d345ba2ea0 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 10 client.18207 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 10 client.18207 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 20 client.18207 populate_metadata read hostname 'smithi027' 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55d345bbd6e0 0x55d345bbfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 4 client.18207 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.412+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2951457126 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d345b14bb0 con 0x55d345bbd6e0 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.413+0000 7fbd9495c640 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55d345bbd6e0 0x55d345bbfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.413+0000 7fbd78ff9640 10 client.18207 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.413+0000 7fbd9495c640 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55d345bbd6e0 0x55d345bbfb00 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c205b4b1012025ac server_cookie=5283d00c4d91e45a in_seq=0 out_seq=0 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2951457126 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d345b14bb0 con 0x55d345bbd6e0 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd78ff9640 10 client.18207 handle_command_reply: tid=0 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 client.18207 shutdown 2024-04-25T01:37:22.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2951457126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2ea0 msgr2=0x55d345baa1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2ea0 0x55d345baa1f0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00be70 tx=0x7fbd8c012820 comp rx=0 tx=0).stop 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2951457126 shutdown_connections 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55d345bbd6e0 0x55d345bbfb00 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345bad570 0x55d345baa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2ea0 0x55d345baa1f0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2951457126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2580 0x55d345bacf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd78ff9640 0 client.18207 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2951457126 >> 172.21.15.27:0/2951457126 conn(0x55d345b9e8a0 msgr2=0x55d345b9db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2951457126 shutdown_connections 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2951457126 wait complete. 2024-04-25T01:37:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.414+0000 7fbd9a1af740 20 client.18207 trim_cache size 0 max 16384 2024-04-25T01:37:22.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.427+0000 7fbd9a1af740 1 Processor -- start 2024-04-25T01:37:22.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9a1af740 1 -- start start 2024-04-25T01:37:22.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2580 0x55d345bb9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2ea0 0x55d345baca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345bad570 0x55d345bc5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901b3690 con 0x55d345bad570 2024-04-25T01:37:22.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901b0f30 con 0x55d345ba2580 2024-04-25T01:37:22.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901aeda0 con 0x55d345ba2ea0 2024-04-25T01:37:22.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345bad570 0x55d345bc5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345bad570 0x55d345bc5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34940/0 (socket says 172.21.15.27:34940) 2024-04-25T01:37:22.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 -- 172.21.15.27:0/696639910 learned_addr learned my addr 172.21.15.27:0/696639910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 -- 172.21.15.27:0/696639910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2ea0 msgr2=0x55d345baca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 --2- 172.21.15.27:0/696639910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2ea0 0x55d345baca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 -- 172.21.15.27:0/696639910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2580 msgr2=0x55d345bb9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 --2- 172.21.15.27:0/696639910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2580 0x55d345bb9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 -- 172.21.15.27:0/696639910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d345ba2cd0 con 0x55d345bad570 2024-04-25T01:37:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd9495c640 1 --2- 172.21.15.27:0/696639910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345bad570 0x55d345bc5390 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fbd80011c40 tx=0x7fbd80012d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c1e94e3871cba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd7affd640 1 -- 172.21.15.27:0/696639910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80032070 con 0x55d345bad570 2024-04-25T01:37:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd7affd640 1 -- 172.21.15.27:0/696639910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd80017070 con 0x55d345bad570 2024-04-25T01:37:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.428+0000 7fbd7affd640 1 -- 172.21.15.27:0/696639910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd800128a0 con 0x55d345bad570 2024-04-25T01:37:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 -- 172.21.15.27:0/696639910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345bad570 msgr2=0x55d345bc5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/696639910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345bad570 0x55d345bc5390 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fbd80011c40 tx=0x7fbd80012d10 comp rx=0 tx=0).stop 2024-04-25T01:37:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 -- 172.21.15.27:0/696639910 shutdown_connections 2024-04-25T01:37:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/696639910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345bad570 0x55d345bc5390 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/696639910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345ba2ea0 0x55d345baca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/696639910 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2580 0x55d345bb9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 -- 172.21.15.27:0/696639910 >> 172.21.15.27:0/696639910 conn(0x55d345b9e8a0 msgr2=0x55d345ba5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 -- 172.21.15.27:0/696639910 shutdown_connections 2024-04-25T01:37:22.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 -- 172.21.15.27:0/696639910 wait complete. 2024-04-25T01:37:22.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 -- start start 2024-04-25T01:37:22.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.429+0000 7fbd9a1af740 1 Processor -- start 2024-04-25T01:37:22.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345bb1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 0x55d345bb1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9a1af740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 0x55d345bb1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901af3f0 con 0x55d345ba2580 2024-04-25T01:37:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901b0860 con 0x55d345ba2ea0 2024-04-25T01:37:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9a1af740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901afac0 con 0x55d345bad570 2024-04-25T01:37:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9a1af740 10 client.0 _ll_register_callbacks cb 0x7fbd8c00b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345bb1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345bb1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34944/0 (socket says 172.21.15.27:34944) 2024-04-25T01:37:22.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 -- 172.21.15.27:0/2969224687 learned_addr learned my addr 172.21.15.27:0/2969224687 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9515d640 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 0x55d345bb1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 -- 172.21.15.27:0/2969224687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 msgr2=0x55d345bb1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 0x55d345bb1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 -- 172.21.15.27:0/2969224687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 msgr2=0x55d345bb1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 0x55d345bb1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 -- 172.21.15.27:0/2969224687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3458e57d0 con 0x55d345ba2580 2024-04-25T01:37:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd9495c640 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345bb1430 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000acf0 tx=0x7fbd8000a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e993eac439d56a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:37:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.430+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2969224687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80032070 con 0x55d345ba2580 2024-04-25T01:37:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2969224687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fbd8002c7c0 con 0x55d345ba2580 2024-04-25T01:37:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2969224687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd80017070 con 0x55d345ba2580 2024-04-25T01:37:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2969224687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd8003e050 con 0x55d345ba2580 2024-04-25T01:37:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 10 client.18213 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2969224687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d345c23f60 con 0x55d345ba2580 2024-04-25T01:37:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2969224687 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd80017070 con 0x55d345ba2580 2024-04-25T01:37:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2969224687 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fbd80004660 con 0x55d345ba2580 2024-04-25T01:37:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 10 client.18213 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 10 client.18213 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:37:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 20 client.18213 populate_metadata read hostname 'smithi027' 2024-04-25T01:37:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55d345c24470 0x55d345c26890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 4 client.18213 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:37:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2969224687 --> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d345b0ef00 con 0x55d345c24470 2024-04-25T01:37:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.431+0000 7fbd9515d640 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55d345c24470 0x55d345c26890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9515d640 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55d345c24470 0x55d345c26890 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fcaea17e67244ca server_cookie=ea3bde2ed80408fd in_seq=0 out_seq=0 2024-04-25T01:37:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd78ff9640 10 client.18213 ms_handle_connect on v2:172.21.15.153:6838/2846322026 2024-04-25T01:37:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd78ff9640 1 -- 172.21.15.27:0/2969224687 <== mds.1 v2:172.21.15.153:6838/2846322026 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d345b0ef00 con 0x55d345c24470 2024-04-25T01:37:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd78ff9640 10 client.18213 handle_command_reply: tid=0 2024-04-25T01:37:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 client.18213 shutdown 2024-04-25T01:37:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2969224687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 msgr2=0x55d345bb1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345bb1430 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000acf0 tx=0x7fbd8000a420 comp rx=0 tx=0).stop 2024-04-25T01:37:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2969224687 shutdown_connections 2024-04-25T01:37:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.153:6838/2846322026,v1:172.21.15.153:6839/2846322026] conn(0x55d345c24470 0x55d345c26890 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d345bad570 0x55d345bb1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d345ba2ea0 0x55d345bb1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd78ff9640 0 client.18213 ms_handle_reset on v2:172.21.15.153:6838/2846322026 2024-04-25T01:37:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 --2- 172.21.15.27:0/2969224687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d345ba2580 0x55d345bb1430 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2969224687 >> 172.21.15.27:0/2969224687 conn(0x55d345b9e8a0 msgr2=0x55d345b9d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2969224687 shutdown_connections 2024-04-25T01:37:22.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 1 -- 172.21.15.27:0/2969224687 wait complete. 2024-04-25T01:37:22.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.432+0000 7fbd9a1af740 20 client.18213 trim_cache size 0 max 16384 2024-04-25T01:37:22.423 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:37:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 -- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd7c0a2d00 msgr2=0x7fbd7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd7c0a2d00 0x7fbd7c0a51c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd880eddb0 tx=0x7fbd880095c0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 -- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd90107720 msgr2=0x7fbd900744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd90107720 0x7fbd900744c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fbd88005cc0 tx=0x7fbd88003970 comp rx=0 tx=0).stop 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 -- 172.21.15.27:0/1925364141 shutdown_connections 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fbd7c0a2d00 0x7fbd7c0a51c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd90107720 0x7fbd900744c0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbd901020d0 0x7fbd90073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 --2- 172.21.15.27:0/1925364141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd90101710 0x7fbd90073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 -- 172.21.15.27:0/1925364141 >> 172.21.15.27:0/1925364141 conn(0x7fbd900fd180 msgr2=0x7fbd900fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 -- 172.21.15.27:0/1925364141 shutdown_connections 2024-04-25T01:37:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.433+0000 7fbd973c1640 1 -- 172.21.15.27:0/1925364141 wait complete. 2024-04-25T01:37:22.425 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:37:22.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 -- 172.21.15.27:0/3815976522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0106630 msgr2=0x7f30e0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 --2- 172.21.15.27:0/3815976522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0106630 0x7f30e0106a10 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009930 tx=0x7f30d002a420 comp rx=0 tx=0).stop 2024-04-25T01:37:22.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 -- 172.21.15.27:0/3815976522 shutdown_connections 2024-04-25T01:37:22.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 --2- 172.21.15.27:0/3815976522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30e0100fe0 0x7f30e010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 --2- 172.21.15.27:0/3815976522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0100620 0x7f30e0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 --2- 172.21.15.27:0/3815976522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0106630 0x7f30e0106a10 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 -- 172.21.15.27:0/3815976522 >> 172.21.15.27:0/3815976522 conn(0x7f30e00776a0 msgr2=0x7f30e0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 -- 172.21.15.27:0/3815976522 shutdown_connections 2024-04-25T01:37:22.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 -- 172.21.15.27:0/3815976522 wait complete. 2024-04-25T01:37:22.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 Processor -- start 2024-04-25T01:37:22.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.554+0000 7f30e66b3640 1 -- start start 2024-04-25T01:37:22.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30e66b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0100620 0x7f30e01980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30e66b3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30e0100fe0 0x7f30e0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30e66b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0106630 0x7f30e019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30e66b3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30e0111b40 con 0x7f30e0100620 2024-04-25T01:37:22.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30e66b3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30e01119c0 con 0x7f30e0100fe0 2024-04-25T01:37:22.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30e66b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30e0111cc0 con 0x7f30e0106630 2024-04-25T01:37:22.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0100620 0x7f30e01980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0100620 0x7f30e01980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34962/0 (socket says 172.21.15.27:34962) 2024-04-25T01:37:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 -- 172.21.15.27:0/226033501 learned_addr learned my addr 172.21.15.27:0/226033501 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30df7fe640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30e0100fe0 0x7f30e0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30e4c50640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0106630 0x7f30e019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 -- 172.21.15.27:0/226033501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0106630 msgr2=0x7f30e019c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0106630 0x7f30e019c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 -- 172.21.15.27:0/226033501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30e0100fe0 msgr2=0x7f30e0198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30e0100fe0 0x7f30e0198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 -- 172.21.15.27:0/226033501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e019d060 con 0x7f30e0100620 2024-04-25T01:37:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30df7fe640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30e0100fe0 0x7f30e0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dffff640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0100620 0x7f30e01980c0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009230 tx=0x7f30d0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dd7fa640 1 -- 172.21.15.27:0/226033501 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d003f030 con 0x7f30e0100620 2024-04-25T01:37:22.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dd7fa640 1 -- 172.21.15.27:0/226033501 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d0003ad0 con 0x7f30e0100620 2024-04-25T01:37:22.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30e66b3640 1 -- 172.21.15.27:0/226033501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e019d2f0 con 0x7f30e0100620 2024-04-25T01:37:22.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.555+0000 7f30dd7fa640 1 -- 172.21.15.27:0/226033501 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0035070 con 0x7f30e0100620 2024-04-25T01:37:22.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.556+0000 7f30e66b3640 1 -- 172.21.15.27:0/226033501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e01a4be0 con 0x7f30e0100620 2024-04-25T01:37:22.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30dd7fa640 1 -- 172.21.15.27:0/226033501 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f30d0042030 con 0x7f30e0100620 2024-04-25T01:37:22.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e93c1740 1 Processor -- start 2024-04-25T01:37:22.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e93c1740 1 -- start start 2024-04-25T01:37:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a13d6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13e1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a13e3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e93c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5605a133b670 con 0x5605a13d6ea0 2024-04-25T01:37:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e93c1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5605a12e3970 con 0x5605a13e1570 2024-04-25T01:37:22.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e93c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5605a1337630 con 0x5605a13d6580 2024-04-25T01:37:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30dffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13e1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30dffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13e1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34970/0 (socket says 172.21.15.27:34970) 2024-04-25T01:37:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30dffff640 1 -- 172.21.15.27:0/571296548 learned_addr learned my addr 172.21.15.27:0/571296548 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30e4c50640 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a13d6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30dd7fa640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f30cc0a2d00 0x7f30cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.558+0000 7f30df7fe640 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a13e3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30e4c50640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f30cc0a2d00 0x7f30cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30dd7fa640 1 -- 172.21.15.27:0/226033501 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f30d00efd10 con 0x7f30e0100620 2024-04-25T01:37:22.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30dffff640 1 -- 172.21.15.27:0/571296548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 msgr2=0x5605a13d6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30dffff640 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a13d6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30dffff640 1 -- 172.21.15.27:0/571296548 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 msgr2=0x5605a13e3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30dffff640 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a13e3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30dffff640 1 -- 172.21.15.27:0/571296548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605a1307400 con 0x5605a13d6ea0 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30e4c50640 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a13d6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30df7fe640 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a13e3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30dffff640 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13e1030 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f30d00ecfe0 tx=0x7f30d0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e659b3a97ce169ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30e4c50640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f30cc0a2d00 0x7f30cc0a51c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f30d80094d0 tx=0x7f30d8004ab0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30d67fc640 1 -- 172.21.15.27:0/571296548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0049070 con 0x5605a13d6ea0 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30d67fc640 1 -- 172.21.15.27:0/571296548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d00bc540 con 0x5605a13d6ea0 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.559+0000 7f30d67fc640 1 -- 172.21.15.27:0/571296548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d00ed2f0 con 0x5605a13d6ea0 2024-04-25T01:37:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 -- 172.21.15.27:0/571296548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 msgr2=0x5605a13e1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13e1030 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f30d00ecfe0 tx=0x7f30d0009260 comp rx=0 tx=0).stop 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 -- 172.21.15.27:0/571296548 shutdown_connections 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a13e3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13e1030 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 --2- 172.21.15.27:0/571296548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a13d6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 -- 172.21.15.27:0/571296548 >> 172.21.15.27:0/571296548 conn(0x5605a13d28a0 msgr2=0x5605a13d2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 -- 172.21.15.27:0/571296548 shutdown_connections 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 -- 172.21.15.27:0/571296548 wait complete. 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 -- start start 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.560+0000 7f30e93c1740 1 Processor -- start 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13d6580 0x5605a13e0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13de210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13e1570 0x5605a13de7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30e93c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5605a133b670 con 0x5605a13d6ea0 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30e93c1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x5605a12f7050 con 0x5605a13d6580 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30e93c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5605a13e4120 con 0x5605a13e1570 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30e93c1740 10 client.0 _ll_register_callbacks cb 0x5605a104ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13de210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13de210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34974/0 (socket says 172.21.15.27:34974) 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 -- 172.21.15.27:0/3574265443 learned_addr learned my addr 172.21.15.27:0/3574265443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30e4c50640 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13e1570 0x5605a13de7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30dffff640 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13d6580 0x5605a13e0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 -- 172.21.15.27:0/3574265443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13e1570 msgr2=0x5605a13de7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13e1570 0x5605a13de7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 -- 172.21.15.27:0/3574265443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13d6580 msgr2=0x5605a13e0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13d6580 0x5605a13e0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 -- 172.21.15.27:0/3574265443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605a12c8dc0 con 0x5605a13d6ea0 2024-04-25T01:37:22.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30df7fe640 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13de210 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f30c800bfc0 tx=0x7f30c8011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a299988b9abc95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.561+0000 7f30bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30bf7fe640 1 -- 172.21.15.27:0/3574265443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8012710 con 0x5605a13d6ea0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30bf7fe640 1 -- 172.21.15.27:0/3574265443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f30c80128b0 con 0x5605a13d6ea0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 1 -- 172.21.15.27:0/3574265443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30d00bc540 con 0x5605a13d6ea0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30bf7fe640 1 -- 172.21.15.27:0/3574265443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f30c8012ab0 con 0x5605a13d6ea0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 10 client.18225 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 1 -- 172.21.15.27:0/3574265443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5605a104f3f0 con 0x5605a13d6ea0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30bf7fe640 1 -- 172.21.15.27:0/3574265443 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d00bc540 con 0x5605a13d6ea0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30bf7fe640 1 -- 172.21.15.27:0/3574265443 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f30c801a3d0 con 0x5605a13d6ea0 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 10 client.18225 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 10 client.18225 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 20 client.18225 populate_metadata read hostname 'smithi027' 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5605a13f1370 0x5605a13f3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 4 client.18225 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30e93c1740 1 -- 172.21.15.27:0/3574265443 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605a1348bb0 con 0x5605a13f1370 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.562+0000 7f30dffff640 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5605a13f1370 0x5605a13f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.563+0000 7f30dffff640 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5605a13f1370 0x5605a13f3780 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=163ce1d315289874 server_cookie=471bb2743a544111 in_seq=0 out_seq=0 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.563+0000 7f30bf7fe640 10 client.18225 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30bf7fe640 1 -- 172.21.15.27:0/3574265443 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5605a1348bb0 con 0x5605a13f1370 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30bf7fe640 10 client.18225 handle_command_reply: tid=0 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 client.18225 shutdown 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 -- 172.21.15.27:0/3574265443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 msgr2=0x5605a13de210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13de210 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f30c800bfc0 tx=0x7f30c8011870 comp rx=0 tx=0).stop 2024-04-25T01:37:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 -- 172.21.15.27:0/3574265443 shutdown_connections 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5605a13f1370 0x5605a13f3780 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13e1570 0x5605a13de7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a13de210 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30bf7fe640 0 client.18225 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 --2- 172.21.15.27:0/3574265443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13d6580 0x5605a13e0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 -- 172.21.15.27:0/3574265443 >> 172.21.15.27:0/3574265443 conn(0x5605a13d28a0 msgr2=0x5605a13d1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 -- 172.21.15.27:0/3574265443 shutdown_connections 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 1 -- 172.21.15.27:0/3574265443 wait complete. 2024-04-25T01:37:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.564+0000 7f30e93c1740 20 client.18225 trim_cache size 0 max 16384 2024-04-25T01:37:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.577+0000 7f30e93c1740 1 Processor -- start 2024-04-25T01:37:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.577+0000 7f30e93c1740 1 -- start start 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.577+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6580 0x5605a13d22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.577+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6ea0 0x5605a13ece20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.577+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a13e0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.577+0000 7f30e93c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30e01b1690 con 0x5605a13d6580 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.577+0000 7f30e93c1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30e01aef30 con 0x5605a13e1570 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.577+0000 7f30e93c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30e01acda0 con 0x5605a13d6ea0 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6580 0x5605a13d22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6580 0x5605a13d22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34984/0 (socket says 172.21.15.27:34984) 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 -- 172.21.15.27:0/241666967 learned_addr learned my addr 172.21.15.27:0/241666967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 -- 172.21.15.27:0/241666967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6ea0 msgr2=0x5605a13ece20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 --2- 172.21.15.27:0/241666967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6ea0 0x5605a13ece20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 -- 172.21.15.27:0/241666967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 msgr2=0x5605a13e0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 --2- 172.21.15.27:0/241666967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a13e0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 -- 172.21.15.27:0/241666967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605a11197d0 con 0x5605a13d6580 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30dffff640 1 --2- 172.21.15.27:0/241666967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6580 0x5605a13d22c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f30d00ed670 tx=0x7f30d0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a2b1b523fa65de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30d5ffb640 1 -- 172.21.15.27:0/241666967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d00be040 con 0x5605a13d6580 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30d5ffb640 1 -- 172.21.15.27:0/241666967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d00c4ab0 con 0x5605a13d6580 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.578+0000 7f30d5ffb640 1 -- 172.21.15.27:0/241666967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0049040 con 0x5605a13d6580 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- 172.21.15.27:0/241666967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6580 msgr2=0x5605a13d22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 --2- 172.21.15.27:0/241666967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6580 0x5605a13d22c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f30d00ed670 tx=0x7f30d0005c00 comp rx=0 tx=0).stop 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- 172.21.15.27:0/241666967 shutdown_connections 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 --2- 172.21.15.27:0/241666967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a13e0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 --2- 172.21.15.27:0/241666967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6ea0 0x5605a13ece20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 --2- 172.21.15.27:0/241666967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6580 0x5605a13d22c0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- 172.21.15.27:0/241666967 >> 172.21.15.27:0/241666967 conn(0x5605a13d28a0 msgr2=0x5605a13ec710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- 172.21.15.27:0/241666967 shutdown_connections 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- 172.21.15.27:0/241666967 wait complete. 2024-04-25T01:37:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- start start 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 Processor -- start 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a1454c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a14551d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a144f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f30e01ad3f0 con 0x5605a13d6ea0 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f30e01ae860 con 0x5605a13e1570 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.579+0000 7f30e93c1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30e01adac0 con 0x5605a13d6580 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30e93c1740 10 client.0 _ll_register_callbacks cb 0x7f30c800b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a14551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a14551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34994/0 (socket says 172.21.15.27:34994) 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 -- 172.21.15.27:0/398926107 learned_addr learned my addr 172.21.15.27:0/398926107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30dffff640 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a1454c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30e4c50640 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a144f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 -- 172.21.15.27:0/398926107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 msgr2=0x5605a1454c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a1454c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 -- 172.21.15.27:0/398926107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 msgr2=0x5605a144f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a144f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 -- 172.21.15.27:0/398926107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605a1450000 con 0x5605a13d6ea0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30df7fe640 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a14551d0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f30c801aa30 tx=0x7f30c800bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47bd03667598e1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30bf7fe640 1 -- 172.21.15.27:0/398926107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c802d070 con 0x5605a13d6ea0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30bf7fe640 1 -- 172.21.15.27:0/398926107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f30c801a3d0 con 0x5605a13d6ea0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30e93c1740 1 -- 172.21.15.27:0/398926107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30d00c4ab0 con 0x5605a13d6ea0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30bf7fe640 1 -- 172.21.15.27:0/398926107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f30c8039050 con 0x5605a13d6ea0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30e93c1740 10 client.18231 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.580+0000 7f30e93c1740 1 -- 172.21.15.27:0/398926107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5605a145c490 con 0x5605a13d6ea0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30bf7fe640 1 -- 172.21.15.27:0/398926107 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d00c4ab0 con 0x5605a13d6ea0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30bf7fe640 1 -- 172.21.15.27:0/398926107 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f30c80051d0 con 0x5605a13d6ea0 2024-04-25T01:37:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30e93c1740 10 client.18231 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30e93c1740 10 client.18231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30e93c1740 20 client.18231 populate_metadata read hostname 'smithi027' 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30e93c1740 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5605a145cc40 0x5605a145f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30e93c1740 4 client.18231 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30e93c1740 1 -- 172.21.15.27:0/398926107 --> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5605a1342f00 con 0x5605a145cc40 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30dffff640 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5605a145cc40 0x5605a145f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30dffff640 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5605a145cc40 0x5605a145f060 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a07c751d13ea6331 server_cookie=6a765d7a580e6a5c in_seq=0 out_seq=0 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.581+0000 7f30bf7fe640 10 client.18231 ms_handle_connect on v2:172.21.15.139:6838/2167485871 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30bf7fe640 1 -- 172.21.15.27:0/398926107 <== mds.0 v2:172.21.15.139:6838/2167485871 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605a1342f00 con 0x5605a145cc40 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30bf7fe640 10 client.18231 handle_command_reply: tid=0 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 client.18231 shutdown 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 -- 172.21.15.27:0/398926107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 msgr2=0x5605a14551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a14551d0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f30c801aa30 tx=0x7f30c800bc50 comp rx=0 tx=0).stop 2024-04-25T01:37:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 -- 172.21.15.27:0/398926107 shutdown_connections 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.139:6838/2167485871,v1:172.21.15.139:6839/2167485871] conn(0x5605a145cc40 0x5605a145f060 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5605a13e1570 0x5605a144f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605a13d6ea0 0x5605a14551d0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 --2- 172.21.15.27:0/398926107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5605a13d6580 0x5605a1454c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 -- 172.21.15.27:0/398926107 >> 172.21.15.27:0/398926107 conn(0x5605a13d28a0 msgr2=0x5605a13d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30bf7fe640 0 client.18231 ms_handle_reset on v2:172.21.15.139:6838/2167485871 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 -- 172.21.15.27:0/398926107 shutdown_connections 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 1 -- 172.21.15.27:0/398926107 wait complete. 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e93c1740 20 client.18231 trim_cache size 0 max 16384 2024-04-25T01:37:22.576 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e66b3640 1 -- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f30cc0a2d00 msgr2=0x7f30cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e66b3640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f30cc0a2d00 0x7f30cc0a51c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f30d80094d0 tx=0x7f30d8004ab0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e66b3640 1 -- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0100620 msgr2=0x7f30e01980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.582+0000 7f30e66b3640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0100620 0x7f30e01980c0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009230 tx=0x7f30d0004270 comp rx=0 tx=0).stop 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.583+0000 7f30e66b3640 1 -- 172.21.15.27:0/226033501 shutdown_connections 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.583+0000 7f30e66b3640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0106630 0x7f30e019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.583+0000 7f30e66b3640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30e0100fe0 0x7f30e0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.583+0000 7f30e66b3640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f30cc0a2d00 0x7f30cc0a51c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.583+0000 7f30e66b3640 1 --2- 172.21.15.27:0/226033501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30e0100620 0x7f30e01980c0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.583+0000 7f30e66b3640 1 -- 172.21.15.27:0/226033501 >> 172.21.15.27:0/226033501 conn(0x7f30e00776a0 msgr2=0x7f30e0103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.583+0000 7f30e66b3640 1 -- 172.21.15.27:0/226033501 shutdown_connections 2024-04-25T01:37:22.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:22.583+0000 7f30e66b3640 1 -- 172.21.15.27:0/226033501 wait complete. 2024-04-25T01:37:22.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:22 smithi027 ceph-mon[29197]: pgmap v373: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:22.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:22 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/132557385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:37:22.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:22 smithi153 ceph-mon[32621]: pgmap v373: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:22.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:22 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/132557385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:37:22.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:22 smithi139 ceph-mon[32760]: pgmap v373: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:22.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:22 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/132557385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:37:23.579 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-25T01:37:23.579 INFO:tasks.fwd_scrub:done joining 2024-04-25T01:37:23.580 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T01:37:23.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:37:23.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.729+0000 7f43e886b640 1 -- 172.21.15.27:0/3593794265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 msgr2=0x7f43e410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:23.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.729+0000 7f43e886b640 1 --2- 172.21.15.27:0/3593794265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 0x7f43e410b7d0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009900 tx=0x7f43d802a510 comp rx=0 tx=0).stop 2024-04-25T01:37:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.729+0000 7f43e886b640 1 -- 172.21.15.27:0/3593794265 shutdown_connections 2024-04-25T01:37:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.729+0000 7f43e886b640 1 --2- 172.21.15.27:0/3593794265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43e410bd10 0x7f43e410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.729+0000 7f43e886b640 1 --2- 172.21.15.27:0/3593794265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 0x7f43e410b7d0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.729+0000 7f43e886b640 1 --2- 172.21.15.27:0/3593794265 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43e41029d0 0x7f43e4102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.729+0000 7f43e886b640 1 -- 172.21.15.27:0/3593794265 >> 172.21.15.27:0/3593794265 conn(0x7f43e4077740 msgr2=0x7f43e4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.730+0000 7f43e886b640 1 -- 172.21.15.27:0/3593794265 shutdown_connections 2024-04-25T01:37:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.730+0000 7f43e886b640 1 -- 172.21.15.27:0/3593794265 wait complete. 2024-04-25T01:37:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.730+0000 7f43e886b640 1 Processor -- start 2024-04-25T01:37:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.730+0000 7f43e886b640 1 -- start start 2024-04-25T01:37:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e886b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43e41029d0 0x7f43e4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e886b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 0x7f43e41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e886b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43e410bd10 0x7f43e419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e886b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4111b10 con 0x7f43e41032f0 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e886b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4111990 con 0x7f43e41029d0 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e886b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4111c90 con 0x7f43e410bd10 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 0x7f43e41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 0x7f43e41985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35014/0 (socket says 172.21.15.27:35014) 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 -- 172.21.15.27:0/2569097141 learned_addr learned my addr 172.21.15.27:0/2569097141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e2d9d640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43e410bd10 0x7f43e419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e259c640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43e41029d0 0x7f43e4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 -- 172.21.15.27:0/2569097141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43e410bd10 msgr2=0x7f43e419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43e410bd10 0x7f43e419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 -- 172.21.15.27:0/2569097141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43e41029d0 msgr2=0x7f43e4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43e41029d0 0x7f43e4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 -- 172.21.15.27:0/2569097141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e419d030 con 0x7f43e41032f0 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e1d9b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 0x7f43e41985d0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f43d8006010 tx=0x7f43d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43d37fe640 1 -- 172.21.15.27:0/2569097141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d803f030 con 0x7f43e41032f0 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e419d2c0 con 0x7f43e41032f0 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43d37fe640 1 -- 172.21.15.27:0/2569097141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43d8003ce0 con 0x7f43e41032f0 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.731+0000 7f43d37fe640 1 -- 172.21.15.27:0/2569097141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8035070 con 0x7f43e41032f0 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.732+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e41a4bb0 con 0x7f43e41032f0 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.733+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e4068560 con 0x7f43e41032f0 2024-04-25T01:37:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.734+0000 7f43d37fe640 1 -- 172.21.15.27:0/2569097141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f43d8042030 con 0x7f43e41032f0 2024-04-25T01:37:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.734+0000 7f43d37fe640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43cc0a2d00 0x7f43cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.734+0000 7f43e259c640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43cc0a2d00 0x7f43cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.734+0000 7f43d37fe640 1 -- 172.21.15.27:0/2569097141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f43d80efe00 con 0x7f43e41032f0 2024-04-25T01:37:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.735+0000 7f43e259c640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43cc0a2d00 0x7f43cc0a51c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f43dc005fd0 tx=0x7f43dc005850 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.739+0000 7f43d37fe640 1 -- 172.21.15.27:0/2569097141 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f43d80bc6f0 con 0x7f43e41032f0 2024-04-25T01:37:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.925+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43e4106e90 con 0x7f43e41032f0 2024-04-25T01:37:23.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.927+0000 7f43d37fe640 1 -- 172.21.15.27:0/2569097141 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+25356 (secure 0 0 0) 0x7f43d80bbff0 con 0x7f43e41032f0 2024-04-25T01:37:23.904 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:37:23.904 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":106,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:37:15.487079+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2238094164":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/3292898694":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/4288305036":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6800/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2504114047":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43cc0a2d00 msgr2=0x7f43cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43cc0a2d00 0x7f43cc0a51c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f43dc005fd0 tx=0x7f43dc005850 comp rx=0 tx=0).stop 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 msgr2=0x7f43e41985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 0x7f43e41985d0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f43d8006010 tx=0x7f43d80042f0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 shutdown_connections 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43e410bd10 0x7f43e419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f43cc0a2d00 0x7f43cc0a51c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e41032f0 0x7f43e41985d0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 --2- 172.21.15.27:0/2569097141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f43e41029d0 0x7f43e4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 >> 172.21.15.27:0/2569097141 conn(0x7f43e4077740 msgr2=0x7f43e40fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 shutdown_connections 2024-04-25T01:37:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:23.929+0000 7f43e886b640 1 -- 172.21.15.27:0/2569097141 wait complete. 2024-04-25T01:37:23.917 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:37:23.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 -- 172.21.15.27:0/2471339171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 msgr2=0x7f1dc81000c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2471339171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 0x7f1dc81000c0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009930 tx=0x7f1db402a420 comp rx=0 tx=0).stop 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 -- 172.21.15.27:0/2471339171 shutdown_connections 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2471339171 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc8100600 0x7f1dc810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2471339171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 0x7f1dc81000c0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2471339171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc8107720 0x7f1dc8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 -- 172.21.15.27:0/2471339171 >> 172.21.15.27:0/2471339171 conn(0x7f1dc8076c00 msgr2=0x7f1dc80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 -- 172.21.15.27:0/2471339171 shutdown_connections 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.046+0000 7f1dcec12640 1 -- 172.21.15.27:0/2471339171 wait complete. 2024-04-25T01:37:24.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcec12640 1 Processor -- start 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcec12640 1 -- start start 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcec12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 0x7f1dc8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcec12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc8100600 0x7f1dc8198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcec12640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc8107720 0x7f1dc819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcec12640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc8111b30 con 0x7f1dc80ffc40 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcec12640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc81119b0 con 0x7f1dc8107720 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcec12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc8111cb0 con 0x7f1dc8100600 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 0x7f1dc8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 0x7f1dc8198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35034/0 (socket says 172.21.15.27:35034) 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 -- 172.21.15.27:0/2196938881 learned_addr learned my addr 172.21.15.27:0/2196938881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcd1af640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc8107720 0x7f1dc819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dc7fff640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc8100600 0x7f1dc8198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 -- 172.21.15.27:0/2196938881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc8100600 msgr2=0x7f1dc8198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc8100600 0x7f1dc8198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 -- 172.21.15.27:0/2196938881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc8107720 msgr2=0x7f1dc819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc8107720 0x7f1dc819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 -- 172.21.15.27:0/2196938881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc819d040 con 0x7f1dc80ffc40 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dc7fff640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc8100600 0x7f1dc8198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcd1af640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc8107720 0x7f1dc819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dcc9ae640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 0x7f1dc8198050 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00d950 tx=0x7f1dbc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.047+0000 7f1dc5ffb640 1 -- 172.21.15.27:0/2196938881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc017070 con 0x7f1dc80ffc40 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.048+0000 7f1dc5ffb640 1 -- 172.21.15.27:0/2196938881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dbc009c30 con 0x7f1dc80ffc40 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.048+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc819d330 con 0x7f1dc80ffc40 2024-04-25T01:37:24.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.048+0000 7f1dc5ffb640 1 -- 172.21.15.27:0/2196938881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc00f6b0 con 0x7f1dc80ffc40 2024-04-25T01:37:24.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.048+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc81a4bc0 con 0x7f1dc80ffc40 2024-04-25T01:37:24.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.049+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc80689b0 con 0x7f1dc80ffc40 2024-04-25T01:37:24.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.050+0000 7f1dc5ffb640 1 -- 172.21.15.27:0/2196938881 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1dbc0109d0 con 0x7f1dc80ffc40 2024-04-25T01:37:24.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.050+0000 7f1dc5ffb640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1db80a2d00 0x7f1db80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.051+0000 7f1dc7fff640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1db80a2d00 0x7f1db80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.051+0000 7f1dc5ffb640 1 -- 172.21.15.27:0/2196938881 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f1dbc0cf490 con 0x7f1dc80ffc40 2024-04-25T01:37:24.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.051+0000 7f1dc7fff640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1db80a2d00 0x7f1db80a51c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1db4000c00 tx=0x7f1db4005a20 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:24.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.055+0000 7f1dc5ffb640 1 -- 172.21.15.27:0/2196938881 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1dbc009de0 con 0x7f1dc80ffc40 2024-04-25T01:37:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.242+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1dc802d170 con 0x7f1dc80ffc40 2024-04-25T01:37:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.243+0000 7f1dc5ffb640 1 -- 172.21.15.27:0/2196938881 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f1dbc09bcc0 con 0x7f1dc80ffc40 2024-04-25T01:37:24.220 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:37:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1db80a2d00 msgr2=0x7f1db80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1db80a2d00 0x7f1db80a51c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1db4000c00 tx=0x7f1db4005a20 comp rx=0 tx=0).stop 2024-04-25T01:37:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 msgr2=0x7f1dc8198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 0x7f1dc8198050 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00d950 tx=0x7f1dbc00de10 comp rx=0 tx=0).stop 2024-04-25T01:37:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 shutdown_connections 2024-04-25T01:37:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1dc8107720 0x7f1dc819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc8100600 0x7f1dc8198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f1db80a2d00 0x7f1db80a51c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 --2- 172.21.15.27:0/2196938881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dc80ffc40 0x7f1dc8198050 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 >> 172.21.15.27:0/2196938881 conn(0x7f1dc8076c00 msgr2=0x7f1dc80fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 shutdown_connections 2024-04-25T01:37:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.245+0000 7f1dcec12640 1 -- 172.21.15.27:0/2196938881 wait complete. 2024-04-25T01:37:24.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 -- 172.21.15.27:0/2663853771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb28101710 msgr2=0x7feb28101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 --2- 172.21.15.27:0/2663853771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb28101710 0x7feb28101b90 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7feb18009930 tx=0x7feb1802a420 comp rx=0 tx=0).stop 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 -- 172.21.15.27:0/2663853771 shutdown_connections 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 --2- 172.21.15.27:0/2663853771 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb281020d0 0x7feb2810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 --2- 172.21.15.27:0/2663853771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb28101710 0x7feb28101b90 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 --2- 172.21.15.27:0/2663853771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb28107720 0x7feb28107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 -- 172.21.15.27:0/2663853771 >> 172.21.15.27:0/2663853771 conn(0x7feb280fd180 msgr2=0x7feb280ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 -- 172.21.15.27:0/2663853771 shutdown_connections 2024-04-25T01:37:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.365+0000 7feb2f876640 1 -- 172.21.15.27:0/2663853771 wait complete. 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2f876640 1 Processor -- start 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2f876640 1 -- start start 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2f876640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb28101710 0x7feb281980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2f876640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb281020d0 0x7feb28198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2f876640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb28107720 0x7feb2819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2f876640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feb28111b30 con 0x7feb281020d0 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2f876640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7feb281119b0 con 0x7feb28107720 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2f876640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb28111cb0 con 0x7feb28101710 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb281020d0 0x7feb28198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb281020d0 0x7feb28198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35042/0 (socket says 172.21.15.27:35042) 2024-04-25T01:37:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 -- 172.21.15.27:0/3495399023 learned_addr learned my addr 172.21.15.27:0/3495399023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2de13640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb28107720 0x7feb2819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2d612640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb28101710 0x7feb281980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 -- 172.21.15.27:0/3495399023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb28101710 msgr2=0x7feb281980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb28101710 0x7feb281980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 -- 172.21.15.27:0/3495399023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb28107720 msgr2=0x7feb2819c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb28107720 0x7feb2819c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 -- 172.21.15.27:0/3495399023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2819d0c0 con 0x7feb281020d0 2024-04-25T01:37:24.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.366+0000 7feb2ce11640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb281020d0 0x7feb28198610 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7feb18009230 tx=0x7feb18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:24.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.367+0000 7feb1e7fc640 1 -- 172.21.15.27:0/3495399023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1803f030 con 0x7feb281020d0 2024-04-25T01:37:24.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.367+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2819d350 con 0x7feb281020d0 2024-04-25T01:37:24.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.367+0000 7feb1e7fc640 1 -- 172.21.15.27:0/3495399023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb18003ad0 con 0x7feb281020d0 2024-04-25T01:37:24.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.367+0000 7feb1e7fc640 1 -- 172.21.15.27:0/3495399023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18035070 con 0x7feb281020d0 2024-04-25T01:37:24.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.367+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb281a4c40 con 0x7feb281020d0 2024-04-25T01:37:24.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.368+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb280689b0 con 0x7feb281020d0 2024-04-25T01:37:24.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.369+0000 7feb1e7fc640 1 -- 172.21.15.27:0/3495399023 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7feb18042030 con 0x7feb281020d0 2024-04-25T01:37:24.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.370+0000 7feb1e7fc640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb140a2d00 0x7feb140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.370+0000 7feb2d612640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb140a2d00 0x7feb140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.370+0000 7feb1e7fc640 1 -- 172.21.15.27:0/3495399023 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7feb180f01a0 con 0x7feb281020d0 2024-04-25T01:37:24.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.370+0000 7feb2d612640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb140a2d00 0x7feb140a51c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feb24005fd0 tx=0x7feb2400b510 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:24.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.373+0000 7feb1e7fc640 1 -- 172.21.15.27:0/3495399023 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb180bc9d0 con 0x7feb281020d0 2024-04-25T01:37:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.559+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feb28109c80 con 0x7feb281020d0 2024-04-25T01:37:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.559+0000 7feb1e7fc640 1 -- 172.21.15.27:0/3495399023 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7feb180bc2d0 con 0x7feb281020d0 2024-04-25T01:37:24.536 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:37:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb140a2d00 msgr2=0x7feb140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb140a2d00 0x7feb140a51c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feb24005fd0 tx=0x7feb2400b510 comp rx=0 tx=0).stop 2024-04-25T01:37:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb281020d0 msgr2=0x7feb28198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb281020d0 0x7feb28198610 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7feb18009230 tx=0x7feb18004270 comp rx=0 tx=0).stop 2024-04-25T01:37:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 shutdown_connections 2024-04-25T01:37:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb28107720 0x7feb2819c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feb140a2d00 0x7feb140a51c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb281020d0 0x7feb28198610 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 --2- 172.21.15.27:0/3495399023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb28101710 0x7feb281980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 >> 172.21.15.27:0/3495399023 conn(0x7feb280fd180 msgr2=0x7feb280fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 shutdown_connections 2024-04-25T01:37:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.561+0000 7feb2f876640 1 -- 172.21.15.27:0/3495399023 wait complete. 2024-04-25T01:37:24.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:24 smithi027 ceph-mon[29197]: pgmap v374: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:24.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:24 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2569097141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:37:24.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:24 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2196938881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:37:24.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:37:24.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 -- 172.21.15.27:0/3746350133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100fd0 msgr2=0x7f11c810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/3746350133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100fd0 0x7f11c810d4e0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009cd0 tx=0x7f11c002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 -- 172.21.15.27:0/3746350133 shutdown_connections 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/3746350133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100fd0 0x7f11c810d4e0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/3746350133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8100610 0x7f11c8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/3746350133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11c81066b0 0x7f11c8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 -- 172.21.15.27:0/3746350133 >> 172.21.15.27:0/3746350133 conn(0x7f11c8077740 msgr2=0x7f11c8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 -- 172.21.15.27:0/3746350133 shutdown_connections 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 -- 172.21.15.27:0/3746350133 wait complete. 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 Processor -- start 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.678+0000 7f11ce2bc640 1 -- start start 2024-04-25T01:37:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11ce2bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100610 0x7f11c8198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11ce2bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8100fd0 0x7f11c81992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11ce2bc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11c81066b0 0x7f11c81a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11ce2bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f11c8110a30 con 0x7f11c8100610 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11ce2bc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81108b0 con 0x7f11c81066b0 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11ce2bc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11c8110bb0 con 0x7f11c8100fd0 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100610 0x7f11c8198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100610 0x7f11c8198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35050/0 (socket says 172.21.15.27:35050) 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 -- 172.21.15.27:0/4223650980 learned_addr learned my addr 172.21.15.27:0/4223650980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c77fe640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8100fd0 0x7f11c81992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 -- 172.21.15.27:0/4223650980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8100fd0 msgr2=0x7f11c81992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8100fd0 0x7f11c81992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11cc859640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11c81066b0 0x7f11c81a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 -- 172.21.15.27:0/4223650980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11c81066b0 msgr2=0x7f11c81a3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11c81066b0 0x7f11c81a3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 -- 172.21.15.27:0/4223650980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c81a4370 con 0x7f11c8100610 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c77fe640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8100fd0 0x7f11c81992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11cc859640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11c81066b0 0x7f11c81a3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c7fff640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100610 0x7f11c8198db0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f11b800d770 tx=0x7f11b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c57fa640 1 -- 172.21.15.27:0/4223650980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8017070 con 0x7f11c8100610 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c57fa640 1 -- 172.21.15.27:0/4223650980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b8009c30 con 0x7f11c8100610 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c81a4540 con 0x7f11c8100610 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11c57fa640 1 -- 172.21.15.27:0/4223650980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8003dd0 con 0x7f11c8100610 2024-04-25T01:37:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.679+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c81a4a60 con 0x7f11c8100610 2024-04-25T01:37:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.681+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c8101d50 con 0x7f11c8100610 2024-04-25T01:37:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.682+0000 7f11c57fa640 1 -- 172.21.15.27:0/4223650980 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f11b8003f90 con 0x7f11c8100610 2024-04-25T01:37:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.682+0000 7f11c57fa640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f11b00a2d00 0x7f11b00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.682+0000 7f11c77fe640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f11b00a2d00 0x7f11b00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.682+0000 7f11c57fa640 1 -- 172.21.15.27:0/4223650980 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f11b80cf240 con 0x7f11c8100610 2024-04-25T01:37:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.683+0000 7f11c77fe640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f11b00a2d00 0x7f11b00a51c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f11c8069430 tx=0x7f11bc005d30 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:24.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.687+0000 7f11c57fa640 1 -- 172.21.15.27:0/4223650980 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f11b8009de0 con 0x7f11c8100610 2024-04-25T01:37:24.772 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:24 smithi153 ceph-mon[32621]: pgmap v374: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:24.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:24 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2569097141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:37:24.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:24 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2196938881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:37:24.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:24 smithi139 ceph-mon[32760]: pgmap v374: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:24.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:24 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2569097141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:37:24.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:24 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2196938881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:37:24.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.871+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f11c802d170 con 0x7f11c8100610 2024-04-25T01:37:24.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.871+0000 7f11c57fa640 1 -- 172.21.15.27:0/4223650980 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f11b809ba70 con 0x7f11c8100610 2024-04-25T01:37:24.848 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f11b00a2d00 msgr2=0x7f11b00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f11b00a2d00 0x7f11b00a51c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f11c8069430 tx=0x7f11bc005d30 comp rx=0 tx=0).stop 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100610 msgr2=0x7f11c8198db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100610 0x7f11c8198db0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f11b800d770 tx=0x7f11b800dc30 comp rx=0 tx=0).stop 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 shutdown_connections 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f11c81066b0 0x7f11c81a3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8100fd0 0x7f11c81992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f11b00a2d00 0x7f11b00a51c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 --2- 172.21.15.27:0/4223650980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8100610 0x7f11c8198db0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 >> 172.21.15.27:0/4223650980 conn(0x7f11c8077740 msgr2=0x7f11c80ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 shutdown_connections 2024-04-25T01:37:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.874+0000 7f11ce2bc640 1 -- 172.21.15.27:0/4223650980 wait complete. 2024-04-25T01:37:24.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:37:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.989+0000 7febc315c640 1 -- 172.21.15.27:0/3889700112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 msgr2=0x7febbc10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.989+0000 7febc315c640 1 --2- 172.21.15.27:0/3889700112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 0x7febbc10d4e0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7feba4009930 tx=0x7feba402a420 comp rx=0 tx=0).stop 2024-04-25T01:37:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 -- 172.21.15.27:0/3889700112 shutdown_connections 2024-04-25T01:37:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 --2- 172.21.15.27:0/3889700112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 0x7febbc10d4e0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 --2- 172.21.15.27:0/3889700112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc100610 0x7febbc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 --2- 172.21.15.27:0/3889700112 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febbc1066b0 0x7febbc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 -- 172.21.15.27:0/3889700112 >> 172.21.15.27:0/3889700112 conn(0x7febbc077740 msgr2=0x7febbc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 -- 172.21.15.27:0/3889700112 shutdown_connections 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 -- 172.21.15.27:0/3889700112 wait complete. 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 Processor -- start 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 -- start start 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc100610 0x7febbc198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 0x7febbc199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febbc1066b0 0x7febbc1a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7febbc110a30 con 0x7febbc100fd0 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1108b0 con 0x7febbc1066b0 2024-04-25T01:37:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febc315c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7febbc110bb0 con 0x7febbc100610 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.990+0000 7febbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 0x7febbc199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 0x7febbc199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35066/0 (socket says 172.21.15.27:35066) 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febbbfff640 1 -- 172.21.15.27:0/2368676318 learned_addr learned my addr 172.21.15.27:0/2368676318 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febc0ef8640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc100610 0x7febbc198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febc16f9640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febbc1066b0 0x7febbc1a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febbbfff640 1 -- 172.21.15.27:0/2368676318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc100610 msgr2=0x7febbc198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febbbfff640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc100610 0x7febbc198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febbbfff640 1 -- 172.21.15.27:0/2368676318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febbc1066b0 msgr2=0x7febbc1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febbbfff640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febbc1066b0 0x7febbc1a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febbbfff640 1 -- 172.21.15.27:0/2368676318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc1a43b0 con 0x7febbc100fd0 2024-04-25T01:37:24.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febbbfff640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 0x7febbc199330 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7febac00e9f0 tx=0x7febac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:24.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febb9ffb640 1 -- 172.21.15.27:0/2368676318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac00cd80 con 0x7febbc100fd0 2024-04-25T01:37:24.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febb9ffb640 1 -- 172.21.15.27:0/2368676318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febac005500 con 0x7febbc100fd0 2024-04-25T01:37:24.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febb9ffb640 1 -- 172.21.15.27:0/2368676318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac010690 con 0x7febbc100fd0 2024-04-25T01:37:24.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febbc1a4580 con 0x7febbc100fd0 2024-04-25T01:37:24.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.991+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febbc1a4aa0 con 0x7febbc100fd0 2024-04-25T01:37:24.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.992+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febbc068560 con 0x7febbc100fd0 2024-04-25T01:37:24.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.993+0000 7febb9ffb640 1 -- 172.21.15.27:0/2368676318 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7febac0046c0 con 0x7febbc100fd0 2024-04-25T01:37:24.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.994+0000 7febb9ffb640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feba80a2d80 0x7feba80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:24.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.994+0000 7febb9ffb640 1 -- 172.21.15.27:0/2368676318 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7febac0cfc70 con 0x7febbc100fd0 2024-04-25T01:37:24.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.994+0000 7febc0ef8640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feba80a2d80 0x7feba80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:24.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.994+0000 7febc0ef8640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feba80a2d80 0x7feba80a5240 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7febbc101750 tx=0x7febb4009670 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:24.998+0000 7febb9ffb640 1 -- 172.21.15.27:0/2368676318 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7febac09c4a0 con 0x7febbc100fd0 2024-04-25T01:37:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.183+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7febbc108ba0 con 0x7febbc100fd0 2024-04-25T01:37:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.184+0000 7febb9ffb640 1 -- 172.21.15.27:0/2368676318 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v106) ==== 103+0+11 (secure 0 0 0) 0x7febac09bda0 con 0x7febbc100fd0 2024-04-25T01:37:25.160 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feba80a2d80 msgr2=0x7feba80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feba80a2d80 0x7feba80a5240 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7febbc101750 tx=0x7febb4009670 comp rx=0 tx=0).stop 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 msgr2=0x7febbc199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 0x7febbc199330 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7febac00e9f0 tx=0x7febac00edf0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 shutdown_connections 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7febbc1066b0 0x7febbc1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7feba80a2d80 0x7feba80a5240 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febbc100fd0 0x7febbc199330 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 --2- 172.21.15.27:0/2368676318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febbc100610 0x7febbc198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 >> 172.21.15.27:0/2368676318 conn(0x7febbc077740 msgr2=0x7febbc0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 shutdown_connections 2024-04-25T01:37:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.186+0000 7febc315c640 1 -- 172.21.15.27:0/2368676318 wait complete. 2024-04-25T01:37:25.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T01:37:25.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 -- 172.21.15.27:0/1216584603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0102160 msgr2=0x7fa9b010e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 --2- 172.21.15.27:0/1216584603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0102160 0x7fa9b010e5a0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009ca0 tx=0x7fa9a802a950 comp rx=0 tx=0).stop 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 -- 172.21.15.27:0/1216584603 shutdown_connections 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 --2- 172.21.15.27:0/1216584603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0102160 0x7fa9b010e5a0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 --2- 172.21.15.27:0/1216584603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9b01017c0 0x7fa9b0101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 --2- 172.21.15.27:0/1216584603 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9b0107750 0x7fa9b0107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 -- 172.21.15.27:0/1216584603 >> 172.21.15.27:0/1216584603 conn(0x7fa9b00788d0 msgr2=0x7fa9b0078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 -- 172.21.15.27:0/1216584603 shutdown_connections 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.300+0000 7fa9b58da640 1 -- 172.21.15.27:0/1216584603 wait complete. 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9b58da640 1 Processor -- start 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9b58da640 1 -- start start 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9b58da640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9b01017c0 0x7fa9b0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9b58da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9b0102160 0x7fa9b01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9b58da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0107750 0x7fa9b019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9b58da640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0111b10 con 0x7fa9b0107750 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9b58da640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0111990 con 0x7fa9b01017c0 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9b58da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0111c90 con 0x7fa9b0102160 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9af7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0107750 0x7fa9b019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9af7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0107750 0x7fa9b019c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35080/0 (socket says 172.21.15.27:35080) 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9af7fe640 1 -- 172.21.15.27:0/3517780794 learned_addr learned my addr 172.21.15.27:0/3517780794 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9aeffd640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9b01017c0 0x7fa9b0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9a7fff640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9b0102160 0x7fa9b01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9af7fe640 1 -- 172.21.15.27:0/3517780794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9b0102160 msgr2=0x7fa9b01984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9af7fe640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9b0102160 0x7fa9b01984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9af7fe640 1 -- 172.21.15.27:0/3517780794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9b01017c0 msgr2=0x7fa9b0197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9af7fe640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9b01017c0 0x7fa9b0197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.301+0000 7fa9af7fe640 1 -- 172.21.15.27:0/3517780794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b019cf60 con 0x7fa9b0107750 2024-04-25T01:37:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.302+0000 7fa9af7fe640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0107750 0x7fa9b019c830 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8006010 tx=0x7fa9a802f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.302+0000 7fa9b48d8640 1 -- 172.21.15.27:0/3517780794 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8034040 con 0x7fa9b0107750 2024-04-25T01:37:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.302+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b019d1f0 con 0x7fa9b0107750 2024-04-25T01:37:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.302+0000 7fa9b48d8640 1 -- 172.21.15.27:0/3517780794 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a8003e00 con 0x7fa9b0107750 2024-04-25T01:37:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.302+0000 7fa9b48d8640 1 -- 172.21.15.27:0/3517780794 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80456e0 con 0x7fa9b0107750 2024-04-25T01:37:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.302+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b01a4ae0 con 0x7fa9b0107750 2024-04-25T01:37:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.303+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b00689b0 con 0x7fa9b0107750 2024-04-25T01:37:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.304+0000 7fa9b48d8640 1 -- 172.21.15.27:0/3517780794 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa9a8037020 con 0x7fa9b0107750 2024-04-25T01:37:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.305+0000 7fa9b48d8640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa99c0a2d00 0x7fa99c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.305+0000 7fa9aeffd640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa99c0a2d00 0x7fa99c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.305+0000 7fa9b48d8640 1 -- 172.21.15.27:0/3517780794 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fa9a80f08a0 con 0x7fa9b0107750 2024-04-25T01:37:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.305+0000 7fa9aeffd640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa99c0a2d00 0x7fa99c0a51c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000cac0 tx=0x7fa9a000c4a0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:25.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.309+0000 7fa9b48d8640 1 -- 172.21.15.27:0/3517780794 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa9a80bd1e0 con 0x7fa9b0107750 2024-04-25T01:37:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.544+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fa9b0109bf0 con 0x7fa9b0107750 2024-04-25T01:37:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.552+0000 7fa9b48d8640 1 -- 172.21.15.27:0/3517780794 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740901 (secure 0 0 0) 0x7fa9a80bcae0 con 0x7fa9b0107750 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 25, 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "active_gid": 15480, 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "active_name": "x", 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "active_addrs": { 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6800", 2024-04-25T01:37:25.540 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1825554759 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6801", 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1825554759 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "active_addr": "172.21.15.27:6801/1825554759", 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "active_change": "2024-04-25T01:25:24.976945+0000", 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "standbys": [ 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 15474, 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "y", 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_features": 4540701547738038271, 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.542 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.543 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:25.543 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.543 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.543 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.543 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.544 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.547 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.548 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.550 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T01:37:25.551 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.553 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.554 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.555 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.556 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.557 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.558 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.559 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.560 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.561 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T01:37:25.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T01:37:25.563 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.564 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.566 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.567 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.569 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.570 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T01:37:25.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.572 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:37:25.573 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.574 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:37:25.575 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.576 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T01:37:25.577 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T01:37:25.578 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T01:37:25.579 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.580 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.581 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.582 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.583 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.584 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T01:37:25.585 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T01:37:25.586 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T01:37:25.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T01:37:25.588 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.590 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.591 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.593 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.594 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:37:25.595 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.596 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.597 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.598 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.599 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T01:37:25.600 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.601 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.602 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.603 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.604 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.607 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.608 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T01:37:25.609 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T01:37:25.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T01:37:25.611 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.612 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.613 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.616 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.617 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.618 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.619 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.620 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.620 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:37:25.620 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.620 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:37:25.622 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.623 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T01:37:25.625 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T01:37:25.626 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:37:25.628 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.629 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.630 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.631 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:37:25.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.633 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.634 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.636 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T01:37:25.637 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:37:25.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.639 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T01:37:25.640 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.641 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.642 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.643 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.644 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.648 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.649 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.650 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.651 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.652 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.653 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.654 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.656 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.659 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:37:25.660 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:37:25.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:37:25.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.663 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.664 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.665 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.666 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.667 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.668 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.669 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.672 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.674 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.675 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.677 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.679 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.680 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.681 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.682 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.684 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.686 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.687 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.688 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T01:37:25.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T01:37:25.690 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T01:37:25.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T01:37:25.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.695 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.696 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.697 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.698 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.699 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.701 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T01:37:25.706 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T01:37:25.708 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.709 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.710 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.712 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.713 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.714 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.716 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.718 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:37:25.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.720 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.721 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.722 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.723 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.727 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.728 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T01:37:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T01:37:25.730 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T01:37:25.731 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.732 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.736 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.737 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.738 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.740 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:37:25.742 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.743 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.745 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.746 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.747 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.748 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.749 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.750 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.751 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.753 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:37:25.755 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:37:25.756 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.757 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.758 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.762 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.763 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.764 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.766 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.769 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.770 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:37:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.773 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.775 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.776 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.777 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.778 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.779 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.780 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.781 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.782 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.783 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.784 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.785 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.786 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.789 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.790 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.791 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.792 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.794 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.797 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.799 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.800 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.801 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.802 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.803 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.804 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.805 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.806 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.807 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.808 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.809 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.810 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:25.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.812 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T01:37:25.813 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.817 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.818 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.819 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.820 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.821 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.827 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.830 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.832 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.833 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.834 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.835 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:25.837 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T01:37:25.838 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T01:37:25.839 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.841 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.843 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.845 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.846 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.847 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.848 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.849 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.850 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.853 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.854 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.855 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.856 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.857 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.858 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.859 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.860 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.861 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.862 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.865 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.867 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.868 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.869 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.870 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.871 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.874 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.875 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.877 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.878 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.879 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.881 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.882 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.884 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T01:37:25.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T01:37:25.886 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.888 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.889 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.891 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.893 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.895 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.896 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.897 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.899 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.900 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.901 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.902 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.903 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.904 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.905 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.906 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.907 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.910 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.911 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T01:37:25.912 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T01:37:25.913 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T01:37:25.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.915 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.916 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.917 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.918 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.919 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.920 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.921 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.924 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.925 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.927 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.928 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.929 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.930 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.931 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.932 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.936 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.937 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.938 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.939 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.940 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.941 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.942 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T01:37:25.943 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.944 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.946 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.947 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.948 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:25.949 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.950 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.953 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.954 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.955 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.956 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.957 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 25135, 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "z", 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_features": 4540701547738038271, 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.958 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.959 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.960 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.962 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.963 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T01:37:25.964 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.965 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.966 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T01:37:25.968 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.969 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.970 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T01:37:25.971 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.972 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T01:37:25.973 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:25.974 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:25.975 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.976 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.977 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.978 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:37:25.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.980 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.981 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:25.982 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T01:37:25.984 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.985 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.986 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.989 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.990 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.991 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.992 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:37:25.993 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.994 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.995 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:37:25.996 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T01:37:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T01:37:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T01:37:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T01:37:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T01:37:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:37:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T01:37:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:37:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:37:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T01:37:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T01:37:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T01:37:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T01:37:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T01:37:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T01:37:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:37:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T01:37:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T01:37:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T01:37:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:37:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T01:37:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:37:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T01:37:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T01:37:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T01:37:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:37:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:37:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.072 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.074 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.075 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.076 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.077 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.078 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.079 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.080 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.081 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.082 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.087 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.088 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.089 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.090 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:37:26.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.092 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.093 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.094 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.095 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.096 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.097 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.098 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.099 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.100 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.101 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.102 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.103 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.105 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.106 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T01:37:26.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.108 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.109 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T01:37:26.110 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T01:37:26.111 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.112 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.113 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.114 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.115 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.116 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.117 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.118 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.119 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.120 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.121 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.122 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T01:37:26.123 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T01:37:26.124 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.125 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.126 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.127 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.128 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.129 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.130 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.131 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.132 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.133 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:37:26.134 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:37:26.135 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:37:26.136 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.137 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:37:26.138 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.139 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.140 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.142 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.143 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.145 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.146 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T01:37:26.147 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T01:37:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T01:37:26.149 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.150 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.151 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.152 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.153 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.154 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.155 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.156 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.157 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.158 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:37:26.159 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.160 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.161 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.162 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.166 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.167 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.168 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.169 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.170 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.171 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.172 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.175 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.176 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.177 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.181 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.182 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.183 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.184 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.186 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.187 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T01:37:26.188 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T01:37:26.189 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.192 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.193 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.194 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.195 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.196 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.197 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.198 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.199 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.200 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.201 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.202 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.203 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.207 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.208 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.209 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.210 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.211 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.212 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T01:37:26.213 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.215 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.216 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.217 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.218 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.219 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.221 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.224 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.225 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.226 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.228 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.229 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.230 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.231 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T01:37:26.231 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:37:26.234 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.235 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.237 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.238 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.239 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.240 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.241 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.244 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.245 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.246 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.247 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.248 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.249 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.251 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T01:37:26.252 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:26.253 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:26.254 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.255 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T01:37:26.256 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.258 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.259 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.260 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.262 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.263 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.264 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.265 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.266 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.267 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.268 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.269 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.270 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.271 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.272 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.273 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.274 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.275 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.276 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.278 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.279 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.280 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.281 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.282 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.283 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.284 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.285 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.286 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.287 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.288 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.289 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.290 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.291 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.292 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.293 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.294 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.295 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.296 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.297 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.298 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.299 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.314 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.315 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.316 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.317 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.319 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.320 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.321 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.322 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.323 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.324 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.325 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.326 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.327 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.328 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.329 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.330 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:37:26.331 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.332 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.333 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.334 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.336 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.338 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T01:37:26.339 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T01:37:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.341 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.342 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.343 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.344 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.345 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.346 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.347 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.348 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.349 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.350 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.352 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.353 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.354 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.355 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.356 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.358 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T01:37:26.359 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.360 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.361 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T01:37:26.363 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T01:37:26.364 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.365 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.366 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.367 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.368 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.369 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.370 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.371 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T01:37:26.372 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.373 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: "dashboard", 2024-04-25T01:37:26.374 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "restful", 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_schedule" 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.375 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.376 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.377 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.379 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.380 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.382 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.383 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T01:37:26.384 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T01:37:26.385 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.386 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.387 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.388 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.389 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.391 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:37:26.392 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:37:26.393 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:37:26.394 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:37:26.395 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.397 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.399 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.400 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T01:37:26.401 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T01:37:26.403 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.404 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.405 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.406 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.407 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.408 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.409 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.410 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.411 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.412 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.413 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T01:37:26.414 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:37:26.416 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:37:26.417 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.418 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:37:26.419 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.420 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.421 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.422 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.423 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.426 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T01:37:26.428 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T01:37:26.429 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T01:37:26.430 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.433 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.434 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.435 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.436 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T01:37:26.437 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.438 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.439 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.440 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:37:26.441 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:37:26.442 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.443 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.444 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.445 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.446 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.447 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.448 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:37:26.449 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:37:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T01:37:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.453 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.454 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T01:37:26.455 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:37:26.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T01:37:26.457 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T01:37:26.458 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.459 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.460 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.461 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.462 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.463 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.464 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.466 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.467 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.468 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.469 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.470 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.471 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.472 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.473 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.474 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T01:37:26.475 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.476 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:37:26.477 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:37:26.478 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:37:26.479 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:37:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.482 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.485 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.486 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.487 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.488 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.489 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.490 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.491 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.492 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.493 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.494 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.495 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.496 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:37:26.497 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:37:26.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.499 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.500 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.501 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.502 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.503 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.504 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.505 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.506 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.507 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.508 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.509 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.510 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.512 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.514 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.517 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T01:37:26.518 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.521 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.523 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.524 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.525 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.526 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.527 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.528 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.529 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.530 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.531 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.532 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.533 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.534 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.535 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.536 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.537 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.538 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.540 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.541 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.542 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.544 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T01:37:26.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.546 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.547 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.548 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.549 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.551 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.553 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.554 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.555 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.557 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.558 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.559 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T01:37:26.560 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.562 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.563 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.564 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.565 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.566 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.568 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.569 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.570 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.571 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.572 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.573 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.575 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.576 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.577 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.578 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.580 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.581 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.582 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.583 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.585 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.587 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.588 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.589 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:37:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.592 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.593 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.595 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.596 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.597 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T01:37:26.599 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.600 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.601 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.602 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.603 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.605 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.606 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.607 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.608 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.609 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.610 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.611 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.612 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.613 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.614 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:37:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.619 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.620 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T01:37:26.621 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.622 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.623 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.625 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.626 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.628 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.629 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.630 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.631 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.634 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.635 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.636 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.637 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.638 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.639 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.640 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.641 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.642 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.643 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.644 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.645 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.646 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.647 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.648 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.649 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.651 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.652 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.653 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.654 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.655 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.656 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.657 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.658 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.659 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.660 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T01:37:26.662 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.665 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.667 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.668 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.669 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.670 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.671 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.672 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.674 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.676 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.678 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.689 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.690 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T01:37:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.692 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.693 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.694 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.695 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.696 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.697 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.698 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.699 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.701 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.706 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.708 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.709 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.710 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.711 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.712 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.713 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.714 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.715 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.717 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.718 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.719 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.722 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.723 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.724 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.726 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.728 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.729 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.730 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.732 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.733 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.736 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.737 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.738 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.740 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.743 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.744 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.748 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.750 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.753 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:37:26.754 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.756 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.757 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.760 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.761 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.762 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.763 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.764 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.766 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.767 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T01:37:26.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T01:37:26.769 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.770 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T01:37:26.771 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T01:37:26.772 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.773 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.774 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T01:37:26.775 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.777 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T01:37:26.778 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:37:26.779 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "services": { 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "dashboard": "https://172.21.15.27:8443/" 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.780 INFO:teuthology.orchestra.run.smithi027.stdout: "always_on_modules": { 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "octopus": [ 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "pacific": [ 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T01:37:26.781 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "quincy": [ 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T01:37:26.782 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "reef": [ 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T01:37:26.783 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "squid": [ 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T01:37:26.784 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: "last_failure_osd_epoch": 94, 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: "active_clients": [ 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T01:37:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3926435016 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "libcephsqlite", 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T01:37:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 204652553 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2781537907 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2869729766 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T01:37:26.788 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 758629768 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4054413824 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T01:37:26.789 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:37:26.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:25 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3495399023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:25 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/4223650980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:25 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2368676318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:26 smithi139 ceph-mon[32760]: pgmap v375: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:26.792 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:26 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3517780794' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:37:26.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:25 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3495399023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:25 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/4223650980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:25 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2368676318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:26 smithi153 ceph-mon[32621]: pgmap v375: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:26.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:26 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3517780794' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:37:26.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa99c0a2d00 msgr2=0x7fa99c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa99c0a2d00 0x7fa99c0a51c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000cac0 tx=0x7fa9a000c4a0 comp rx=0 tx=0).stop 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0107750 msgr2=0x7fa9b019c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0107750 0x7fa9b019c830 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8006010 tx=0x7fa9a802f710 comp rx=0 tx=0).stop 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 shutdown_connections 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa99c0a2d00 0x7fa99c0a51c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b0107750 0x7fa9b019c830 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9b0102160 0x7fa9b01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 --2- 172.21.15.27:0/3517780794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9b01017c0 0x7fa9b0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 >> 172.21.15.27:0/3517780794 conn(0x7fa9b00788d0 msgr2=0x7fa9b0100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 shutdown_connections 2024-04-25T01:37:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:37:25.557+0000 7fa9b58da640 1 -- 172.21.15.27:0/3517780794 wait complete. 2024-04-25T01:37:26.799 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T01:37:26.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T01:37:26.799 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:25 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3495399023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:25 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/4223650980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:25 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2368676318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:37:26.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:26 smithi027 ceph-mon[29197]: pgmap v375: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-25T01:37:26.825 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:26 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3517780794' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:37:27.059 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:37:28.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:28 smithi027 ceph-mon[29197]: pgmap v376: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2024-04-25T01:37:28.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:28 smithi153 ceph-mon[32621]: pgmap v376: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2024-04-25T01:37:28.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:28 smithi139 ceph-mon[32760]: pgmap v376: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 87 B/s wr, 7 op/s 2024-04-25T01:37:29.062 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 0, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 2, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 0, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 0, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.001358954, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.000181249, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:29.063 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 244, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 234, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1449200, 2024-04-25T01:37:29.064 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 144544, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 2, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.046289568, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.015994768, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.032136903, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.004924191, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 233, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.011502754, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000049368 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1449200, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 144544 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 6, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 182, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 3344, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 187536, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.016149567, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.011212574, 2024-04-25T01:37:29.066 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.002343436, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 180, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.020400245, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000113334 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 187536 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:29.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:29.069 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-25T01:37:29.070 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 148995, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 31, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:29.071 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 928, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 2, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T01:37:29.072 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 246, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 1414796, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 246, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 246, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1414796, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:29.073 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:29.074 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:37:29.733 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-25T01:37:29.733 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-25T01:37:29.733 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-25T01:37:29.733 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-25T01:37:29.733 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-25T01:37:29.733 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T01:37:29.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T01:37:30.027 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:37:30.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:30 smithi027 ceph-mon[29197]: pgmap v377: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:30.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:30 smithi153 ceph-mon[32621]: pgmap v377: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:30.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:30 smithi139 ceph-mon[32760]: pgmap v377: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:31.905 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:37:31.905 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:31.905 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 0, 2024-04-25T01:37:31.905 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 2, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 0, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 0, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.001261114, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.000205650, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.906 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 252, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 242, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1452064, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 148880, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 2, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.049593070, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.015891365, 2024-04-25T01:37:31.907 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.036175532, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.004889835, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 241, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.011482703, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000047646 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1452064, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 148880 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 6, 2024-04-25T01:37:31.908 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 187, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 3344, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 188416, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.016251184, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.011171046, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.002239311, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 185, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.015282227, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000082606 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:31.909 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 188416 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 0, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:31.910 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:31.911 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 149017, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 31, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:31.912 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 928, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 2, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:31.913 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 254, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 1416537, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 254, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 254, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1416537, 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T01:37:31.914 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:31.915 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:31.915 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:31.915 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:31.915 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:31.915 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:37:32.550 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T01:37:32.550 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T01:37:32.550 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-25T01:37:32.550 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-25T01:37:32.550 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-25T01:37:32.550 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T01:37:32.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T01:37:32.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:32 smithi153 ceph-mon[32621]: pgmap v378: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:32.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:32 smithi139 ceph-mon[32760]: pgmap v378: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:37:32.924 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:32 smithi027 ceph-mon[29197]: pgmap v378: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:34.627 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:34 smithi027 ceph-mon[29197]: pgmap v379: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:34.660 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 0, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 2, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 0, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 0, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.001523471, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.000309236, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:34.661 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 2, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 182, 2024-04-25T01:37:34.662 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 320, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 187440, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.016524927, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.011373044, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.002286296, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 179, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.013192846, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000073703 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:34.663 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 187440 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 252, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 240, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1453376, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 147072, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 2, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.045699666, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.015591873, 2024-04-25T01:37:34.664 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.032430118, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.004968204, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 240, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.013122748, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000054678 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1453376, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 147072 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 0, 2024-04-25T01:37:34.665 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:34.666 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:34.667 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 148977, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 30, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-25T01:37:34.668 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 928, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 2, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 250, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 1415380, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 250, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 250, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1415380, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:34.670 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:34.671 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:34.671 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:37:34.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:34 smithi153 ceph-mon[32621]: pgmap v379: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:34.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:34 smithi139 ceph-mon[32760]: pgmap v379: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:35.445 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-25T01:37:35.445 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-25T01:37:35.446 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-25T01:37:35.446 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-25T01:37:35.446 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-25T01:37:35.446 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T01:37:35.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T01:37:35.695 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:37:36.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:36 smithi153 ceph-mon[32621]: pgmap v380: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:36.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:36 smithi139 ceph-mon[32760]: pgmap v380: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:36.833 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:36 smithi027 ceph-mon[29197]: pgmap v380: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:37.433 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T01:37:37.433 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:37.433 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 0, 2024-04-25T01:37:37.433 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 2, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 0, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 4, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 0, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.001296400, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.000325515, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 2, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 187, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 320, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 188320, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.015742625, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.011225799, 2024-04-25T01:37:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.001824057, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 184, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.017457082, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000094875 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 188320 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 258, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 246, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 1455632, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 150608, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 2, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 1, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.049847773, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.016409696, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.035356726, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.005158675, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 246, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.013404378, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000054489 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1455632, 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 150608 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 0, 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 149034, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 32, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 928, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 2, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 256, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 1416797, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 256, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 256, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 1416797, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-25T01:37:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-25T01:37:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T01:37:37.445 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T01:37:38.109 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T01:37:38.109 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T01:37:38.109 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-25T01:37:38.109 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-25T01:37:38.109 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-25T01:37:38.109 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T01:37:38.110 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T01:37:38.359 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:37:38.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:38 smithi153 ceph-mon[32621]: pgmap v381: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:38.789 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:38 smithi139 ceph-mon[32760]: pgmap v381: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:38.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:38 smithi027 ceph-mon[29197]: pgmap v381: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:40.162 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T01:37:40.162 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1143, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1115, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 275120, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 3203390, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 26, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 4, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.132384814, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.042960247, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.043000747, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.027264649, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1117, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.308055119, 2024-04-25T01:37:40.163 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000275787 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1019, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000238981, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000234 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 355248 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1683, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1516, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1955376, 2024-04-25T01:37:40.164 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 2331149, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 164, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 6, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.221301245, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.058339282, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.131630002, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.036720845, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1518, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.196051843, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000129151 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1773, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000355707, 2024-04-25T01:37:40.165 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000200 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 1573088, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 189504 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 2200, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 729, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 911156, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 549663, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 25, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 8, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.291574518, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.061338218, 2024-04-25T01:37:40.166 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.110235458, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.115165167, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 720, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.278803997, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000387227 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2183, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000418391, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000191 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 819904, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 312128 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.167 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 631, 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.186681546, 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000295850 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T01:37:40.168 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000007012, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000003506 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 203, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 202, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 202, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.201117768, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000995632 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 1, 2024-04-25T01:37:40.169 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 12, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 42, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 2, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 40, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 0, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 39, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 1, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 3, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 3, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 2, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 283, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 271, 2024-04-25T01:37:40.170 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 1, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 0, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 1, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 1, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 1, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 11, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 281, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-04-25T01:37:40.171 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 4, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 234, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 6, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 281, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 193, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 22, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 171, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 88, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 2, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 13, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 12, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 1, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:37:40.172 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 21, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 5, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 2, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 22 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 0, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 6, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 2, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:37:40.173 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 2, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 22, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 1, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:37:40.174 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 218, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 218, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 2, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 2, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-04-25T01:37:40.175 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 6552564, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 0, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 187, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.390388762, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.002087640 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 0 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 35, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 58, 2024-04-25T01:37:40.176 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 23, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 20, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 20, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 0, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 42, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 43, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 1, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 3, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 23, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 20, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 55852, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 231804 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 265, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 3, 2024-04-25T01:37:40.177 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 203, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 38, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 8, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.004816983, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000963396 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 136, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.030627619, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000225203 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.178 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 4, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000795442, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000198860 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.179 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000154527, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000154527 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.180 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 12, 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.002212636, 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000184386 2024-04-25T01:37:40.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.182 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-04-25T01:37:40.182 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 8, 2024-04-25T01:37:40.182 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.014702569, 2024-04-25T01:37:40.182 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001837821 2024-04-25T01:37:40.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.182 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-04-25T01:37:40.182 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 12, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.063483172, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.005290264 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000271573, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000271573 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-04-25T01:37:40.183 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000780857, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000780857 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.002033757, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.001016878 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-04-25T01:37:40.184 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.021954368, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.010977184 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 6, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.028134462, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.004689077 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-04-25T01:37:40.185 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.009201436, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.003067145 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-04-25T01:37:40.186 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 9, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.021948367, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.002438707 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-04-25T01:37:40.187 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 2, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 6, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 4, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 2, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 30, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 15, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 715, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:40.189 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:40.190 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:40.191 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 365995, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 146, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 528, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 6, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:40.192 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 9608, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 184, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 190683, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 1486, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-04-25T01:37:40.193 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 1108, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 2377879, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 37, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 1108, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1108, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 20.045588423, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.018091686 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1108, 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1152 2024-04-25T01:37:40.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 1108, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 325, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 783, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 0, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 11, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 22, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 187, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 47, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-04-25T01:37:40.195 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 506, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 303, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 22, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-04-25T01:37:40.196 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 1, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-04-25T01:37:40.197 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 106, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 22, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 12, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 12, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 45, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 0, 2024-04-25T01:37:40.198 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 8 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 1, 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 0 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.199 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 0, 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 0, 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 0, 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:40.200 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 5022, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 2579092, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 5022, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 5022, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2579092, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 1108, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 2377879, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1108, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2377879, 2024-04-25T01:37:40.202 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 1108, 2024-04-25T01:37:40.203 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 1108, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 1108, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 1108, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:40.204 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x55c6df6b2500": { 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 218, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 2358260, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 218, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 187, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2358260, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:40.206 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T01:37:40.693 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:40 smithi027 ceph-mon[29197]: pgmap v382: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:40.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:40 smithi153 ceph-mon[32621]: pgmap v382: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:40.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:40 smithi139 ceph-mon[32760]: pgmap v382: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:40.869 INFO:tasks.check_counter:Daemon mds.b mds.exported=1 2024-04-25T01:37:40.870 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-04-25T01:37:40.870 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-25T01:37:40.870 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=12 2024-04-25T01:37:40.870 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=6 2024-04-25T01:37:40.870 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T01:37:40.870 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T01:37:41.116 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:37:42.646 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:42 smithi139 ceph-mon[32760]: pgmap v383: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:42.646 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:42 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:42.773 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:42 smithi153 ceph-mon[32621]: pgmap v383: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:42.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:42 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1469, 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1471, 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 2719915, 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 467206, 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 2, 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.122132853, 2024-04-25T01:37:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.048514994, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.030013125, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.032168362, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1469, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.043488926, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000029604 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:37:42.898 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1018, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1004, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1679122, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 388673, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 4, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 3, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.119945657, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.039631050, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.050630202, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.030959509, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1003, 2024-04-25T01:37:42.899 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.044194807, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000044062 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 1497552, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 153984 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 864, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1047, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 291632, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 756508, 2024-04-25T01:37:42.900 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 4, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.123246210, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.043095510, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.030057881, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.036909626, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1045, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.062927329, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000060217 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 483664 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 5298, 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.142250993, 2024-04-25T01:37:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000026849 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 0, 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 0, 2024-04-25T01:37:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 2, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 2, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 0, 2024-04-25T01:37:42.904 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 2, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 1, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 0, 2024-04-25T01:37:42.905 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:37:42.906 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 0 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-04-25T01:37:42.907 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-04-25T01:37:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 2001, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-04-25T01:37:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 0, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 3, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 6538430, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 6537831, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 2001 2024-04-25T01:37:42.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 4, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 23, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 19, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 2, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 11, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 9, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 2, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 21, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 19, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 0, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 0, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 0, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 51152, 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 215420 2024-04-25T01:37:42.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 0, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 0, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 0, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 0, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 0, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-04-25T01:37:42.913 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-04-25T01:37:42.914 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.915 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.916 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.916 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-04-25T01:37:42.916 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.916 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.916 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.916 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.919 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-04-25T01:37:42.920 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.921 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.922 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 1, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 2, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 1, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 2, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 0, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 0, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 3178, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:42.923 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:42.924 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:42.925 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 189393, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 51, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 9608, 2024-04-25T01:37:42.926 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 184, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 15072, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 40, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 3065, 2024-04-25T01:37:42.927 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 21, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 3, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 3065, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3065, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1.390971299, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000453824 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3065, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 3066 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 3065, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 3065, 2024-04-25T01:37:42.928 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 1594, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 1469, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-04-25T01:37:42.929 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 1, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-04-25T01:37:42.930 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-04-25T01:37:42.931 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 106, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 17, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 5, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 5, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 0, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 2, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 0 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 0, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-04-25T01:37:42.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 0 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 0, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 0, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 0, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:42.933 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 3347, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 4372114, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 3347, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 3347, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 4372114, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.934 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 3065, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 2344131, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 3065, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2344131, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.935 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 3065, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 3065, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 3065, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 3065, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.936 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T01:37:42.937 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x5627f0bd66e0": { 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:42.938 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:42.939 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T01:37:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:42 smithi027 ceph-mon[29197]: pgmap v383: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:42.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:42 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:43.651 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:43 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:37:43.652 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:43 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:37:43.652 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:43 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:37:43.679 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T01:37:43.679 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T01:37:43.679 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-25T01:37:43.679 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:37:43.679 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:37:43.679 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T01:37:43.679 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T01:37:43.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:43 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:37:43.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:43 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:37:43.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:43 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:37:43.919 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:37:43.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:43 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:37:43.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:43 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:37:43.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:43 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' 2024-04-25T01:37:44.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:44 smithi139 ceph-mon[32760]: pgmap v384: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:44.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:44 smithi027 ceph-mon[29197]: pgmap v384: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:45.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:44 smithi153 ceph-mon[32621]: pgmap v384: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1841, 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1842, 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 2903075, 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 585130, 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 6, 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 2, 2024-04-25T01:37:45.585 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.166052711, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.066113611, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.038943394, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.044999134, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1840, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.068360769, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000037152 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.586 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 10, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 191, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 5000, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 484650, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 7, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 5, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.011540875, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.007174474, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.002342169, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.000090660, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 189, 2024-04-25T01:37:45.587 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.018721364, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000099054 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 483312 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1141, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 1127, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1709317, 2024-04-25T01:37:45.588 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 427303, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 3, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 2, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.124907678, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.038177665, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.051722726, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.035536460, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 1126, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.039709520, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000035266 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.589 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 1497440, 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 153504 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 1, 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-MDSRank": { 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 4536, 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.108556337, 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000023932 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "finisher-PurgeQueue": { 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "queue_len": 0, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "complete_latency": { 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "mds": { 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "request": 0, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "reply": 0, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "reply_latency": { 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.591 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_reply": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "forward": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_complete": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_fetch_keys": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_commit": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_split": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_merge": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes": 30, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_top": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_bottom": 30, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pin_tail": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_pinned": 30, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_expired": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "inodes_with_caps": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "caps": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "subtrees": 2, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_hit": 0, 2024-04-25T01:37:45.592 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_forward": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_discover": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_remote_ino": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "traverse_lock": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "load_cent": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "q": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "exported": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "exported_inodes": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "imported": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "imported_inodes": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_dir_fetch": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "openino_peer_discover": 0, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rfiles": 4, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rbytes": 234, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "root_rsnaps": 6, 2024-04-25T01:37:45.593 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_set_tag": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "scrub_file_inodes": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:37:45.594 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_cap_release": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "process_request_cap_release": 0 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_cache": { 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_update_receipt": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_try_discover": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_send_discover": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "dir_handle_discover": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_delayed": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_created": 0, 2024-04-25T01:37:45.595 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_enqueued": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_reintegrated": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "strays_migrated": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_processing": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_started": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "recovery_completed": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_exportdir": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_flush": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_fragstats": 0, 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: "ireq_inodestats": 0 2024-04-25T01:37:45.596 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_log": { 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "evlrg": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "evadd": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "evex": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "evtrm": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "ev": 218, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "evexg": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "evexd": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "segadd": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "segex": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "segtrm": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "seg": 2, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "segmjr": 2, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "segexg": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "segexd": 0, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "expos": 4194304, 2024-04-25T01:37:45.597 INFO:teuthology.orchestra.run.smithi139.stdout: "wrpos": 6552564, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "rdpos": 6552428, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "jlat": { 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "replayed": 218 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_mem": { 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "ino": 26, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "ino+": 27, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "ino-": 1, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "dir": 11, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "dir+": 11, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "dir-": 0, 2024-04-25T01:37:45.598 INFO:teuthology.orchestra.run.smithi139.stdout: "dn": 30, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "dn+": 31, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "dn-": 1, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "cap": 0, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "cap+": 0, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "cap-": 0, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "rss": 51268, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "heap": 215420 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_server": { 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_client_request": 0, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "dispatch_server_request": 0, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_request": 0, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_client_session": 0, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "handle_peer_request": 0, 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "req_create_latency": { 2024-04-25T01:37:45.599 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getattr_latency": { 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getfilelock_latency": { 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "req_link_latency": { 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.600 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookup_latency": { 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookuphash_latency": { 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupino_latency": { 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.601 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupname_latency": { 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupparent_latency": { 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lookupsnap_latency": { 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.602 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: "req_lssnap_latency": { 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mkdir_latency": { 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.603 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mknod_latency": { 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "req_mksnap_latency": { 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "req_open_latency": { 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.604 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "req_readdir_latency": { 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rename_latency": { 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "req_renamesnap_latency": { 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.605 INFO:teuthology.orchestra.run.smithi139.stdout: "req_snapdiff_latency": { 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmdir_latency": { 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmsnap_latency": { 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.606 INFO:teuthology.orchestra.run.smithi139.stdout: "req_rmxattr_latency": { 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setattr_latency": { 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setdirlayout_latency": { 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setfilelock_latency": { 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.607 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setlayout_latency": { 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "req_setxattr_latency": { 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "req_symlink_latency": { 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.608 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "req_unlink_latency": { 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "req_getvxattr_latency": { 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.609 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_sessions": { 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "session_count": 2, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "session_add": 6, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "session_remove": 4, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_open": 4, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "sessions_stale": 0, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "total_load": 0, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "average_load": 0, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "avg_session_uptime": 3182, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "mdthresh_evicted": 0 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:45.610 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:45.611 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:45.612 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 190583, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 83, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 9608, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 184, 2024-04-25T01:37:45.613 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 151965, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 1207, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "objecter": { 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "op_active": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "op_laggy": 0, 2024-04-25T01:37:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send": 2705, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "op_send_bytes": 26, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "op_resend": 4, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "op_reply": 2705, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "op_latency": { 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2705, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 1.535968349, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000567825 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "op_inflight": 0, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "oplen_avg": { 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 2705, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 2706 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "op": 2705, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "op_r": 2705, 2024-04-25T01:37:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "op_w": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "op_rmw": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "op_pg": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_stat": 1720, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_create": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_read": 982, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_write": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writefull": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_writesame": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_append": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_zero": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_truncate": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_delete": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_mapext": 0, 2024-04-25T01:37:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_sparse_read": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_clonerange": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_getxattr": 2, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_setxattr": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_rmxattr": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_call": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_watch": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_notify": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "osdop_other": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_active": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_send": 0, 2024-04-25T01:37:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_resend": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "linger_ping": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_active": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_send": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "poolop_resend": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_active": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_send": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "poolstat_resend": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_active": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_send": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "statfs_resend": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "command_active": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "command_send": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "command_resend": 0, 2024-04-25T01:37:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "map_epoch": 106, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "map_full": 0, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "map_inc": 17, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_sessions": 5, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_open": 5, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_session_close": 0, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_laggy": 0, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_wr": 0, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_rd": 2, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_del": 0 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "oft": { 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_objs": 0, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_updates": 0, 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "omap_total_removes": 0 2024-04-25T01:37:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "osd-slow-ops": { 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "slow_ops_count": 0 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "purge_queue": { 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops": 0, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing": 0, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executing_high_water": 0, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed_ops": 0, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_executed": 0, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pq_item_in_journal": 0 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 2988, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 4331153, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 2988, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2988, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 4331153, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_bytes": { 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 2705, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 2358270, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2705, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2358270, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-objecter_ops": { 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 1024, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 2705, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 2705, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 2705, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 2705, 2024-04-25T01:37:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T01:37:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-write_buf_throttle-0x560d835da6e0": { 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 3758096384, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 0, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 0, 2024-04-25T01:37:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 0, 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 0, 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:45.626 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T01:37:46.281 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-25T01:37:46.281 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-25T01:37:46.281 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-25T01:37:46.281 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:37:46.281 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:37:46.281 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T01:37:46.281 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T01:37:46.521 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:37:46.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:46 smithi139 ceph-mon[32760]: pgmap v385: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:46.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:46 smithi027 ceph-mon[29197]: pgmap v385: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:47.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:46 smithi153 ceph-mon[32621]: pgmap v385: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:48.173 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-04-25T01:37:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 258, 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 248, 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 1453696, 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 152720, 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 4, 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 0, 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.051447311, 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.016041709, 2024-04-25T01:37:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.035456499, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.006465264, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 247, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.013635168, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000055203 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 1453696, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 152720 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 2, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 189, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 320, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 188672, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 5, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 3, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.013128263, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.008862485, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.001605956, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 186, 2024-04-25T01:37:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.016273901, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000087494 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 188672 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 4, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages": 4, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_bytes": 3024, 2024-04-25T01:37:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_bytes": 448, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_created_connections": 2, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_active_connections": 1, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_total_time": 0.000703265, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_send_time": 0.000042246, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_recv_time": 0.000645845, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 3, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000742074, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000247358 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "cct": { 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "total_workers": 0, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "unhealthy_workers": 0 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "mempool": { 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_items": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_bytes": 149745, 2024-04-25T01:37:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_anon_items": 31, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_bytes": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_items": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_bytes": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_items": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, 2024-04-25T01:37:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 928, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 2, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_1_items": 0, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "unittest_2_items": 0 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 0, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "max": 104857600, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "get_started": 0, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "get": 260, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "get_sum": 1417317, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "get_or_fail_success": 260, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "take": 0, 2024-04-25T01:37:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "take_sum": 0, 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "put": 260, 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "put_sum": 1417317, 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "wait": { 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "avgcount": 0, 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "sum": 0.000000000, 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "avgtime": 0.000000000 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-04-25T01:37:48.184 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-04-25T01:37:48.778 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:48 smithi139 ceph-mon[32760]: pgmap v386: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:48.806 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-25T01:37:48.806 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-25T01:37:48.806 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-25T01:37:48.806 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-25T01:37:48.806 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-25T01:37:48.806 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T01:37:48.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T01:37:48.833 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:48 smithi153 ceph-mon[32621]: pgmap v386: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:48.943 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:48 smithi027 ceph-mon[29197]: pgmap v386: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:37:50.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:50 smithi153 ceph-mon[32621]: pgmap v387: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:50.790 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:50 smithi139 ceph-mon[32760]: pgmap v387: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:50.833 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T01:37:50.833 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1978, 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2127, 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 488554, 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 3379811, 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 13, 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.186636729, 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.062501064, 2024-04-25T01:37:50.834 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.046760562, 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.058249666, 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2126, 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.088670636, 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000041707 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4, 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000337, 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000084 2024-04-25T01:37:50.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 355808 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 2059, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 4526, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 665981, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 2319501, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 48, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 6, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.279099768, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.126512994, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.149296298, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.015794349, 2024-04-25T01:37:50.836 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4523, 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.116349345, 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000025723 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4022, 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000924005, 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000229 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 496992, 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 1632128 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.837 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1103, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 1154, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1734795, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 543988, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 11, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 6, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.136528329, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.052689499, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.082467118, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.011813990, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1153, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.054897232, 2024-04-25T01:37:50.838 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000047612 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1536, 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000392331, 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000255 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1513600, 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 174576 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.839 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4524, 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.934463418, 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000206556 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T01:37:50.840 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 16, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000481912, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000030119 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 606, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 605, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 605, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2.514666519, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.004156473 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-25T01:37:50.841 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 1, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 14, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 2, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 12, 2024-04-25T01:37:50.842 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 12, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 17, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 1, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 1, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 3, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 614, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 606, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 3, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:37:50.843 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 1, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 11, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 1, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 1, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-25T01:37:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 18, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 1, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 5, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 18, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:37:50.845 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:37:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:37:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:37:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 18, 2024-04-25T01:37:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:37:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 604, 2024-04-25T01:37:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 601, 2024-04-25T01:37:50.846 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 598, 2024-04-25T01:37:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:37:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:37:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 717, 2024-04-25T01:37:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 710, 2024-04-25T01:37:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 2, 2024-04-25T01:37:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 601 2024-04-25T01:37:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-25T01:37:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-04-25T01:37:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-25T01:37:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-25T01:37:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 4, 2024-04-25T01:37:50.848 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-04-25T01:37:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-25T01:37:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-25T01:37:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:37:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 1, 2024-04-25T01:37:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 1, 2024-04-25T01:37:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-04-25T01:37:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-25T01:37:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-25T01:37:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:37:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:37:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-25T01:37:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-25T01:37:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:37:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:37:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:37:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 1, 2024-04-25T01:37:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-25T01:37:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:37:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-25T01:37:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-25T01:37:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-25T01:37:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-25T01:37:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 2001, 2024-04-25T01:37:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-25T01:37:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-25T01:37:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 2001, 2024-04-25T01:37:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-25T01:37:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-25T01:37:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 3, 2024-04-25T01:37:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-25T01:37:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-25T01:37:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 3, 2024-04-25T01:37:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-04-25T01:37:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-25T01:37:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-25T01:37:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-25T01:37:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 6538430, 2024-04-25T01:37:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-04-25T01:37:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-25T01:37:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1910, 2024-04-25T01:37:50.854 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 3.848659293, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002015004 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 18, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 32, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 14, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 17, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 21, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 4, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 14, 2024-04-25T01:37:50.855 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 32, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 18, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 1, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 5, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 4, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 54840, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 1808, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 1, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 606, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 14, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 5, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T01:37:50.856 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000464802, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000464802 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000722639, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000722639 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.857 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001542860, 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000771430 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-25T01:37:50.858 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000129831, 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000129831 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.859 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-25T01:37:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.861 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-25T01:37:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.863 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-25T01:37:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-25T01:37:50.864 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000419935, 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000419935 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.865 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.867 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-25T01:37:50.867 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.867 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.867 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.867 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-25T01:37:50.867 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.867 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-25T01:37:50.868 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 598, 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2.508476734, 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.004194777 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.869 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-25T01:37:50.870 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.870 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.870 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.870 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-25T01:37:50.870 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.870 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.870 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.871 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.872 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.872 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-25T01:37:50.872 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T01:37:50.872 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.002909718, 2024-04-25T01:37:50.872 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002909718 2024-04-25T01:37:50.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.872 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 2, 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 1, 2024-04-25T01:37:50.873 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 1, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 87, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 87, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 732, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:50.874 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:50.875 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:50.876 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 283303, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 110, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9608, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 184, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:50.877 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 76952, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 288, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2620, 2024-04-25T01:37:50.878 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 2359587, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 33, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2620, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2620, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 5.318904330, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002030116 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2620, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2648 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2620, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 19, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 2601, 2024-04-25T01:37:50.879 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 1, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 1911, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 49, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 14, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 18, 2024-04-25T01:37:50.880 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 598, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-25T01:37:50.881 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-25T01:37:50.882 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 106, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 17, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 9, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 9, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 24, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 3 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-04-25T01:37:50.883 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 2, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 1, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 2, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 1, 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-25T01:37:50.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 5136, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 2343801, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 5136, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 5136, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2343801, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.885 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2620, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2359744, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2620, 2024-04-25T01:37:50.886 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2359744, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2620, 2024-04-25T01:37:50.887 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2620, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2620, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2620, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 1, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 157, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 1, 2024-04-25T01:37:50.888 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 157, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x55928709a500": { 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2001, 2024-04-25T01:37:50.889 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 2344126, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2001, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1910, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2344126, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:50.890 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T01:37:50.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:50 smithi027 ceph-mon[29197]: pgmap v387: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:51.525 INFO:tasks.check_counter:Daemon mds.c mds.exported=1 2024-04-25T01:37:51.525 INFO:tasks.check_counter:Daemon mds.c mds.imported=1 2024-04-25T01:37:51.525 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-25T01:37:51.525 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:37:51.525 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:37:51.525 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T01:37:51.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T01:37:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:37:52.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:52 smithi139 ceph-mon[32760]: pgmap v388: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:52.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:52 smithi027 ceph-mon[29197]: pgmap v388: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:52.971 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:52 smithi153 ceph-mon[32621]: pgmap v388: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 2532, 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2534, 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 717261, 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 805183, 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-04-25T01:37:53.570 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.225618061, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.075554690, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.055054536, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.077396447, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2532, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.099202265, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000039179 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.571 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 284, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 270, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1498652, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 155962, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.052260463, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.016880018, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.036522520, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.006635357, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 269, 2024-04-25T01:37:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.014594559, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000054254 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1498096, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 155264 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 7, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 191, 2024-04-25T01:37:53.573 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 3612, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 487390, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.012571181, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.007582007, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.002986894, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000020930, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 189, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.013165991, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000069661 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 487072 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4276, 2024-04-25T01:37:53.575 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.120460072, 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000028171 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-04-25T01:37:53.576 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-04-25T01:37:53.577 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 1, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-04-25T01:37:53.578 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-04-25T01:37:53.579 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-04-25T01:37:53.580 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-25T01:37:53.581 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-25T01:37:53.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 6, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-25T01:37:53.583 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4195956, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 4195815, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 6 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 3, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 3, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-04-25T01:37:53.584 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 1, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 1, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 50556, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-04-25T01:37:53.585 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.586 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.587 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-25T01:37:53.589 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.590 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.590 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.590 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-25T01:37:53.590 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.590 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.590 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.592 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.593 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.594 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.595 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-25T01:37:53.596 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 0, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 2, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 2, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 1, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 1300, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:53.597 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:53.598 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:53.599 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 181347, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 49, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9608, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 184, 2024-04-25T01:37:53.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 8464, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 13, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2535, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 21, 2024-04-25T01:37:53.601 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 3, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2535, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2535, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 1.311976426, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000517544 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2535, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2536 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2535, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 2535, 2024-04-25T01:37:53.602 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 1656, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 877, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-25T01:37:53.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 1, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-25T01:37:53.604 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 106, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 17, 2024-04-25T01:37:53.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 4, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 4, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T01:37:53.607 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2819, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1948386, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2819, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2819, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1948386, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-25T01:37:53.608 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2535, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 1657, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2535, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1657, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.609 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2535, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2535, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2535, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2535, 2024-04-25T01:37:53.610 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:53.611 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x561007c826e0": { 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:53.612 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:53.613 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T01:37:54.385 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T01:37:54.386 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T01:37:54.386 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-25T01:37:54.386 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:37:54.386 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:37:54.386 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T01:37:54.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T01:37:54.627 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:37:54.750 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:54 smithi153 ceph-mon[32621]: pgmap v389: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:54.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:54 smithi027 ceph-mon[29197]: pgmap v389: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:55.040 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:54 smithi139 ceph-mon[32760]: pgmap v389: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 453, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 630, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 129499, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 137903, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 8, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.070030795, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.025956225, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.031957397, 2024-04-25T01:37:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.006393668, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 629, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.044263717, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000070371 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1013, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000293783, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000290 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 67, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 253, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 12332, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 1012085, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 55, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 7, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.029233429, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.008987706, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.016895684, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000215609, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 250, 2024-04-25T01:37:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.021015431, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000084061 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 65, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000007206, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000110 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 490624 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 322, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 304, 2024-04-25T01:37:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1513829, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 214573, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 9, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.054369710, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.017234843, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.037936125, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.006647768, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 303, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.026260345, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000086667 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4, 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000398, 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000099 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1511024, 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 158624 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T01:37:56.345 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 47, 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001753097, 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000037299 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000009493, 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000004746 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-25T01:37:56.346 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 3, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 1, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000203297, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000203297 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 2, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-25T01:37:56.347 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 11, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 11, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 10, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 2, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 5, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 2, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 1, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-25T01:37:56.348 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 5, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-25T01:37:56.349 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-04-25T01:37:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 2 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 2, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-04-25T01:37:56.351 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.352 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 6, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 6, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 2, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-25T01:37:56.353 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4195956, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 5, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.008544533, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001708906 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 14, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 14, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 12, 2024-04-25T01:37:56.354 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 13, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 1, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 11, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 11, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 1, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 1, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 54724, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 4, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 3, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 8, 2024-04-25T01:37:56.355 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000203297, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000203297 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.357 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.358 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-25T01:37:56.359 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.360 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.360 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.360 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.361 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.363 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-25T01:37:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 0, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 2, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 2, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 0, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 1, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 1, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 88, 2024-04-25T01:37:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:56.368 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:56.369 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 255191, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 88, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:56.370 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 9608, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 184, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 54856, 2024-04-25T01:37:56.371 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 213, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 42, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 12044, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 28, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 42, 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-25T01:37:56.372 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 42, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.220996061, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.005261810 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 42, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 64 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 42, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 0, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 42, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-25T01:37:56.373 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 5, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 8, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 6, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 0, 2024-04-25T01:37:56.374 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-25T01:37:56.375 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 106, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-25T01:37:56.376 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 17, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 9, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 9, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 14, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 1 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.377 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:56.378 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 838, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1561051, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 838, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 838, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1561051, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 42, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 12044, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 42, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 12044, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 42, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 42, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 42, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 42, 2024-04-25T01:37:56.381 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:56.382 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x560ec2d28500": { 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 6, 2024-04-25T01:37:56.383 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1652, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 6, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 5, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1652, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:56.384 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T01:37:56.774 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:56 smithi153 ceph-mon[32621]: pgmap v390: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:56.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:56 smithi027 ceph-mon[29197]: pgmap v390: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:57.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:56 smithi139 ceph-mon[32760]: pgmap v390: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:37:57.182 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-25T01:37:57.182 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-25T01:37:57.182 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-25T01:37:57.182 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:37:57.182 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:37:57.182 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T01:37:57.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T01:37:57.425 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:37:58.875 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:37:58 smithi153 ceph-mon[32621]: pgmap v391: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:58.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:37:58 smithi027 ceph-mon[29197]: pgmap v391: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:59.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:37:58 smithi139 ceph-mon[32760]: pgmap v391: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:37:59.127 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-25T01:37:59.127 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:37:59.127 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 0, 2024-04-25T01:37:59.127 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2, 2024-04-25T01:37:59.127 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:37:59.127 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 0, 2024-04-25T01:37:59.127 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-04-25T01:37:59.127 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 0, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.000985858, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.000185428, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.128 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 6, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 194, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 3344, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 189648, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 5, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.013030304, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.008573770, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.001860634, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:37:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 192, 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.022745531, 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000118466 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-25T01:37:59.130 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 189648 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 261, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 251, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1454512, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 154640, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 2, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 1, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.049473348, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.016301786, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.033363842, 2024-04-25T01:37:59.131 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.006600634, 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 250, 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.014715758, 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000058863 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.132 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1454512, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 154640 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:37:59.133 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:37:59.134 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:37:59.135 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 148714, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 30, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 0, 2024-04-25T01:37:59.136 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 0, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 928, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 2, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:37:59.137 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 263, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1417707, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 263, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 263, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1417707, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-25T01:37:59.138 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-25T01:37:59.139 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-25T01:37:59.139 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:59.139 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-25T01:37:59.139 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-25T01:37:59.844 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-25T01:37:59.844 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-25T01:37:59.844 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-25T01:37:59.844 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-25T01:37:59.844 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-25T01:37:59.845 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T01:37:59.856 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T01:37:59.856 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:37:59.856 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:37:59.877 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T01:37:59.877 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:37:59.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.112+0000 7f2473ac3640 1 -- 172.21.15.27:0/1793217725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103340 msgr2=0x7f246c10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.112+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1793217725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103340 0x7f246c10b820 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f245c009900 tx=0x7f245c02a480 comp rx=0 tx=0).stop 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 -- 172.21.15.27:0/1793217725 shutdown_connections 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1793217725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f246c10bd60 0x7f246c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1793217725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103340 0x7f246c10b820 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1793217725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f246c102a20 0x7f246c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 -- 172.21.15.27:0/1793217725 >> 172.21.15.27:0/1793217725 conn(0x7f246c0777a0 msgr2=0x7f246c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 -- 172.21.15.27:0/1793217725 shutdown_connections 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 -- 172.21.15.27:0/1793217725 wait complete. 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 Processor -- start 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 -- start start 2024-04-25T01:38:00.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c102a20 0x7f246c100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f246c103340 0x7f246c100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f246c10bd60 0x7f246c1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f246c111b80 con 0x7f246c102a20 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f246c111a00 con 0x7f246c10bd60 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.113+0000 7f2473ac3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f246c111d00 con 0x7f246c103340 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c102a20 0x7f246c100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c102a20 0x7f246c100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57270/0 (socket says 172.21.15.27:57270) 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247105e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f246c103340 0x7f246c100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 -- 172.21.15.27:0/1235301822 learned_addr learned my addr 172.21.15.27:0/1235301822 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f2472060640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f246c10bd60 0x7f246c1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 -- 172.21.15.27:0/1235301822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f246c103340 msgr2=0x7f246c100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f246c103340 0x7f246c100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 -- 172.21.15.27:0/1235301822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f246c10bd60 msgr2=0x7f246c1010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f246c10bd60 0x7f246c1010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 -- 172.21.15.27:0/1235301822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246c1a64e0 con 0x7f246c102a20 2024-04-25T01:38:00.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f247185f640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c102a20 0x7f246c100670 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f246800c890 tx=0x7f246800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f2462ffd640 1 -- 172.21.15.27:0/1235301822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468004220 con 0x7f246c102a20 2024-04-25T01:38:00.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f2462ffd640 1 -- 172.21.15.27:0/1235301822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24680043c0 con 0x7f246c102a20 2024-04-25T01:38:00.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246c1a67d0 con 0x7f246c102a20 2024-04-25T01:38:00.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f2462ffd640 1 -- 172.21.15.27:0/1235301822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680049e0 con 0x7f246c102a20 2024-04-25T01:38:00.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.114+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f246c1a6cf0 con 0x7f246c102a20 2024-04-25T01:38:00.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.115+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f246c0689b0 con 0x7f246c102a20 2024-04-25T01:38:00.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.116+0000 7f2462ffd640 1 -- 172.21.15.27:0/1235301822 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f24680079b0 con 0x7f246c102a20 2024-04-25T01:38:00.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.117+0000 7f2462ffd640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f24580a2d30 0x7f24580a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.117+0000 7f247105e640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f24580a2d30 0x7f24580a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.117+0000 7f2462ffd640 1 -- 172.21.15.27:0/1235301822 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f24680cf890 con 0x7f246c102a20 2024-04-25T01:38:00.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.118+0000 7f247105e640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f24580a2d30 0x7f24580a51f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f245c000c00 tx=0x7f245c005b50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.121+0000 7f2462ffd640 1 -- 172.21.15.27:0/1235301822 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2468014030 con 0x7f246c102a20 2024-04-25T01:38:00.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.306+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f246c101b50 con 0x7f246c102a20 2024-04-25T01:38:00.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.308+0000 7f2462ffd640 1 -- 172.21.15.27:0/1235301822 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+25356 (secure 0 0 0) 0x7f246809c0c0 con 0x7f246c102a20 2024-04-25T01:38:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:38:00.287 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":106,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:37:15.487079+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2238094164":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/3292898694":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/4288305036":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6800/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2504114047":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:38:00.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f24580a2d30 msgr2=0x7f24580a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f24580a2d30 0x7f24580a51f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f245c000c00 tx=0x7f245c005b50 comp rx=0 tx=0).stop 2024-04-25T01:38:00.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c102a20 msgr2=0x7f246c100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c102a20 0x7f246c100670 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f246800c890 tx=0x7f246800cd50 comp rx=0 tx=0).stop 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 shutdown_connections 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f246c10bd60 0x7f246c1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f246c103340 0x7f246c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f24580a2d30 0x7f24580a51f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 --2- 172.21.15.27:0/1235301822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c102a20 0x7f246c100670 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 >> 172.21.15.27:0/1235301822 conn(0x7f246c0777a0 msgr2=0x7f246c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 shutdown_connections 2024-04-25T01:38:00.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.310+0000 7f2473ac3640 1 -- 172.21.15.27:0/1235301822 wait complete. 2024-04-25T01:38:00.300 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:18:11.811188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:22:55.765659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:22:56.604736+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:23:00.289710+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:38:00.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:38:00.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 -- 172.21.15.27:0/1484109215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990100620 msgr2=0x7f5990100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 --2- 172.21.15.27:0/1484109215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990100620 0x7f5990100aa0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f5980009930 tx=0x7f598002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 -- 172.21.15.27:0/1484109215 shutdown_connections 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 --2- 172.21.15.27:0/1484109215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5990100fe0 0x7f599010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 --2- 172.21.15.27:0/1484109215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990100620 0x7f5990100aa0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 --2- 172.21.15.27:0/1484109215 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5990106630 0x7f5990106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 -- 172.21.15.27:0/1484109215 >> 172.21.15.27:0/1484109215 conn(0x7f59900776a0 msgr2=0x7f5990077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 -- 172.21.15.27:0/1484109215 shutdown_connections 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.427+0000 7f59981a2640 1 -- 172.21.15.27:0/1484109215 wait complete. 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f59981a2640 1 Processor -- start 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f59981a2640 1 -- start start 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f59981a2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5990100620 0x7f5990198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f59981a2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5990100fe0 0x7f5990198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f59981a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990106630 0x7f599019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f59981a2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5990110a40 con 0x7f5990106630 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f59981a2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f59901108c0 con 0x7f5990100fe0 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f59981a2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5990110bc0 con 0x7f5990100620 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f599673f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990106630 0x7f599019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f599673f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990106630 0x7f599019c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57282/0 (socket says 172.21.15.27:57282) 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f599673f640 1 -- 172.21.15.27:0/2756199890 learned_addr learned my addr 172.21.15.27:0/2756199890 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.428+0000 7f5995f3e640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5990100620 0x7f5990198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f599573d640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5990100fe0 0x7f5990198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f599673f640 1 -- 172.21.15.27:0/2756199890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5990100620 msgr2=0x7f5990198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f599673f640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5990100620 0x7f5990198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f599673f640 1 -- 172.21.15.27:0/2756199890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5990100fe0 msgr2=0x7f5990198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f599673f640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5990100fe0 0x7f5990198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f599673f640 1 -- 172.21.15.27:0/2756199890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599019cfc0 con 0x7f5990106630 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f599573d640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5990100fe0 0x7f5990198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f599673f640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990106630 0x7f599019c890 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f598800eaa0 tx=0x7f598800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f5986ffd640 1 -- 172.21.15.27:0/2756199890 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598800cdb0 con 0x7f5990106630 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599019d2b0 con 0x7f5990106630 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f5986ffd640 1 -- 172.21.15.27:0/2756199890 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5988005020 con 0x7f5990106630 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f5986ffd640 1 -- 172.21.15.27:0/2756199890 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59880116b0 con 0x7f5990106630 2024-04-25T01:38:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.429+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59901a4b40 con 0x7f5990106630 2024-04-25T01:38:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.430+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59900689b0 con 0x7f5990106630 2024-04-25T01:38:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.431+0000 7f5986ffd640 1 -- 172.21.15.27:0/2756199890 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5988005560 con 0x7f5990106630 2024-04-25T01:38:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.432+0000 7f5986ffd640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f597c0a2d00 0x7f597c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.432+0000 7f5995f3e640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f597c0a2d00 0x7f597c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.432+0000 7f5986ffd640 1 -- 172.21.15.27:0/2756199890 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f59880cfe30 con 0x7f5990106630 2024-04-25T01:38:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.432+0000 7f5995f3e640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f597c0a2d00 0x7f597c0a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f598c00bb60 tx=0x7f598c00b390 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.435+0000 7f5986ffd640 1 -- 172.21.15.27:0/2756199890 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5988016030 con 0x7f5990106630 2024-04-25T01:38:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.620+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59901a4de0 con 0x7f5990106630 2024-04-25T01:38:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.620+0000 7f5986ffd640 1 -- 172.21.15.27:0/2756199890 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f598809c660 con 0x7f5990106630 2024-04-25T01:38:00.598 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T01:38:00.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f597c0a2d00 msgr2=0x7f597c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f597c0a2d00 0x7f597c0a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f598c00bb60 tx=0x7f598c00b390 comp rx=0 tx=0).stop 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990106630 msgr2=0x7f599019c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990106630 0x7f599019c890 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f598800eaa0 tx=0x7f598800ef60 comp rx=0 tx=0).stop 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 shutdown_connections 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f597c0a2d00 0x7f597c0a51c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5990106630 0x7f599019c890 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5990100fe0 0x7f5990198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 --2- 172.21.15.27:0/2756199890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5990100620 0x7f5990198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 >> 172.21.15.27:0/2756199890 conn(0x7f59900776a0 msgr2=0x7f59900ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 shutdown_connections 2024-04-25T01:38:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.622+0000 7f59981a2640 1 -- 172.21.15.27:0/2756199890 wait complete. 2024-04-25T01:38:00.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 -- 172.21.15.27:0/3972989136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c069dc0 msgr2=0x7fcb0c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/3972989136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c069dc0 0x7fcb0c06a240 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009930 tx=0x7fcb0002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 -- 172.21.15.27:0/3972989136 shutdown_connections 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/3972989136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c06a780 0x7fcb0c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/3972989136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c069dc0 0x7fcb0c06a240 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/3972989136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb0c107730 0x7fcb0c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 -- 172.21.15.27:0/3972989136 >> 172.21.15.27:0/3972989136 conn(0x7fcb0c0776a0 msgr2=0x7fcb0c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 -- 172.21.15.27:0/3972989136 shutdown_connections 2024-04-25T01:38:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.738+0000 7fcb10ddb640 1 -- 172.21.15.27:0/3972989136 wait complete. 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb10ddb640 1 Processor -- start 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb10ddb640 1 -- start start 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb10ddb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c069dc0 0x7fcb0c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb10ddb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c06a780 0x7fcb0c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb10ddb640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb0c107730 0x7fcb0c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb10ddb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c111b40 con 0x7fcb0c06a780 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb10ddb640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c1119c0 con 0x7fcb0c107730 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb10ddb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c111cc0 con 0x7fcb0c069dc0 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.739+0000 7fcb09d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c06a780 0x7fcb0c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb09d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c06a780 0x7fcb0c1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57288/0 (socket says 172.21.15.27:57288) 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb09d9b640 1 -- 172.21.15.27:0/1410852424 learned_addr learned my addr 172.21.15.27:0/1410852424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb0a59c640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c069dc0 0x7fcb0c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb0ad9d640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb0c107730 0x7fcb0c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb09d9b640 1 -- 172.21.15.27:0/1410852424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c069dc0 msgr2=0x7fcb0c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb09d9b640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c069dc0 0x7fcb0c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb09d9b640 1 -- 172.21.15.27:0/1410852424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb0c107730 msgr2=0x7fcb0c19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb09d9b640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb0c107730 0x7fcb0c19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb09d9b640 1 -- 172.21.15.27:0/1410852424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c19d000 con 0x7fcb0c06a780 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb09d9b640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c06a780 0x7fcb0c1985a0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009230 tx=0x7fcb00004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcafb7fe640 1 -- 172.21.15.27:0/1410852424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0003f030 con 0x7fcb0c06a780 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcafb7fe640 1 -- 172.21.15.27:0/1410852424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb00003ad0 con 0x7fcb0c06a780 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0c19d290 con 0x7fcb0c06a780 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcafb7fe640 1 -- 172.21.15.27:0/1410852424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00035070 con 0x7fcb0c06a780 2024-04-25T01:38:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.740+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0c1a4b80 con 0x7fcb0c06a780 2024-04-25T01:38:00.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.741+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb0c0689b0 con 0x7fcb0c06a780 2024-04-25T01:38:00.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.742+0000 7fcafb7fe640 1 -- 172.21.15.27:0/1410852424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcb00042030 con 0x7fcb0c06a780 2024-04-25T01:38:00.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.743+0000 7fcafb7fe640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcaf40a2d80 0x7fcaf40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.743+0000 7fcb0a59c640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcaf40a2d80 0x7fcaf40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.743+0000 7fcafb7fe640 1 -- 172.21.15.27:0/1410852424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fcb000efbe0 con 0x7fcb0c06a780 2024-04-25T01:38:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.743+0000 7fcb0a59c640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcaf40a2d80 0x7fcaf40a5240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcafc005d00 tx=0x7fcafc00b410 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.747+0000 7fcafb7fe640 1 -- 172.21.15.27:0/1410852424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcb000bbdc0 con 0x7fcb0c06a780 2024-04-25T01:38:00.842 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:00 smithi139 ceph-mon[32760]: pgmap v392: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:38:00.843 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:00 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1235301822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:00.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.948+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcb0c109cb0 con 0x7fcb0c06a780 2024-04-25T01:38:00.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.948+0000 7fcafb7fe640 1 -- 172.21.15.27:0/1410852424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7fcb000c00c0 con 0x7fcb0c06a780 2024-04-25T01:38:00.926 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:38:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcaf40a2d80 msgr2=0x7fcaf40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcaf40a2d80 0x7fcaf40a5240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcafc005d00 tx=0x7fcafc00b410 comp rx=0 tx=0).stop 2024-04-25T01:38:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c06a780 msgr2=0x7fcb0c1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c06a780 0x7fcb0c1985a0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009230 tx=0x7fcb00004270 comp rx=0 tx=0).stop 2024-04-25T01:38:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 shutdown_connections 2024-04-25T01:38:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb0c107730 0x7fcb0c19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fcaf40a2d80 0x7fcaf40a5240 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb0c06a780 0x7fcb0c1985a0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 --2- 172.21.15.27:0/1410852424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb0c069dc0 0x7fcb0c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 >> 172.21.15.27:0/1410852424 conn(0x7fcb0c0776a0 msgr2=0x7fcb0c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 shutdown_connections 2024-04-25T01:38:00.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:00.950+0000 7fcb10ddb640 1 -- 172.21.15.27:0/1410852424 wait complete. 2024-04-25T01:38:00.939 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:00 smithi027 ceph-mon[29197]: pgmap v392: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:38:00.939 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:00 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1235301822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:00.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:38:01.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:00 smithi153 ceph-mon[32621]: pgmap v392: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:38:01.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:00 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1235301822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- 172.21.15.27:0/2019467185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4100540 msgr2=0x7fa7e4100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 --2- 172.21.15.27:0/2019467185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4100540 0x7fa7e4100920 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009900 tx=0x7fa7d402a410 comp rx=0 tx=0).stop 2024-04-25T01:38:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- 172.21.15.27:0/2019467185 shutdown_connections 2024-04-25T01:38:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 --2- 172.21.15.27:0/2019467185 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7e410e0f0 0x7fa7e41104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 --2- 172.21.15.27:0/2019467185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e4100e60 0x7fa7e410dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 --2- 172.21.15.27:0/2019467185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4100540 0x7fa7e4100920 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- 172.21.15.27:0/2019467185 >> 172.21.15.27:0/2019467185 conn(0x7fa7e4077740 msgr2=0x7fa7e4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- 172.21.15.27:0/2019467185 shutdown_connections 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- 172.21.15.27:0/2019467185 wait complete. 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 Processor -- start 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- start start 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e4100540 0x7fa7e419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7e4100e60 0x7fa7e419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e410e0f0 0x7fa7e419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4113d20 con 0x7fa7e410e0f0 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4113ba0 con 0x7fa7e4100e60 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.067+0000 7fa7e9815640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4113ea0 con 0x7fa7e4100540 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e410e0f0 0x7fa7e419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e410e0f0 0x7fa7e419eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57300/0 (socket says 172.21.15.27:57300) 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 -- 172.21.15.27:0/967634325 learned_addr learned my addr 172.21.15.27:0/967634325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e2ffd640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e4100540 0x7fa7e419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7dbfff640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7e4100e60 0x7fa7e419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 -- 172.21.15.27:0/967634325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e4100540 msgr2=0x7fa7e419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e4100540 0x7fa7e419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 -- 172.21.15.27:0/967634325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7e4100e60 msgr2=0x7fa7e419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7e4100e60 0x7fa7e419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 -- 172.21.15.27:0/967634325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e419f1f0 con 0x7fa7e410e0f0 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e37fe640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e410e0f0 0x7fa7e419eac0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00eb40 tx=0x7fa7dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e8813640 1 -- 172.21.15.27:0/967634325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc004b20 con 0x7fa7e410e0f0 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e8813640 1 -- 172.21.15.27:0/967634325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7dc004cc0 con 0x7fa7e410e0f0 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e419f4e0 con 0x7fa7e410e0f0 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e8813640 1 -- 172.21.15.27:0/967634325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc010850 con 0x7fa7e410e0f0 2024-04-25T01:38:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.068+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41a6d70 con 0x7fa7e410e0f0 2024-04-25T01:38:01.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.069+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e4068560 con 0x7fa7e410e0f0 2024-04-25T01:38:01.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.071+0000 7fa7e8813640 1 -- 172.21.15.27:0/967634325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa7dc005000 con 0x7fa7e410e0f0 2024-04-25T01:38:01.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.071+0000 7fa7e8813640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa7d00a2d80 0x7fa7d00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.071+0000 7fa7e2ffd640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa7d00a2d80 0x7fa7d00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.071+0000 7fa7e8813640 1 -- 172.21.15.27:0/967634325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fa7dc0cfe70 con 0x7fa7e410e0f0 2024-04-25T01:38:01.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.072+0000 7fa7e2ffd640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa7d00a2d80 0x7fa7d00a5240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009230 tx=0x7fa7d40058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:01.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.075+0000 7fa7e8813640 1 -- 172.21.15.27:0/967634325 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa7dc09c6a0 con 0x7fa7e410e0f0 2024-04-25T01:38:01.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.261+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa7e402d170 con 0x7fa7e410e0f0 2024-04-25T01:38:01.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.262+0000 7fa7e8813640 1 -- 172.21.15.27:0/967634325 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7fa7dc09bfa0 con 0x7fa7e410e0f0 2024-04-25T01:38:01.240 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T01:38:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa7d00a2d80 msgr2=0x7fa7d00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa7d00a2d80 0x7fa7d00a5240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009230 tx=0x7fa7d40058d0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e410e0f0 msgr2=0x7fa7e419eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e410e0f0 0x7fa7e419eac0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00eb40 tx=0x7fa7dc00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 shutdown_connections 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa7d00a2d80 0x7fa7d00a5240 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e410e0f0 0x7fa7e419eac0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7e4100e60 0x7fa7e419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 --2- 172.21.15.27:0/967634325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7e4100540 0x7fa7e419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 >> 172.21.15.27:0/967634325 conn(0x7fa7e4077740 msgr2=0x7fa7e410f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 shutdown_connections 2024-04-25T01:38:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.264+0000 7fa7e9815640 1 -- 172.21.15.27:0/967634325 wait complete. 2024-04-25T01:38:01.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 -- 172.21.15.27:0/677219745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c1020d0 msgr2=0x7fa66c10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 --2- 172.21.15.27:0/677219745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c1020d0 0x7fa66c10e5e0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fa664009cd0 tx=0x7fa66402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 -- 172.21.15.27:0/677219745 shutdown_connections 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 --2- 172.21.15.27:0/677219745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c1020d0 0x7fa66c10e5e0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 --2- 172.21.15.27:0/677219745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66c101710 0x7fa66c101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 --2- 172.21.15.27:0/677219745 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa66c107720 0x7fa66c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 -- 172.21.15.27:0/677219745 >> 172.21.15.27:0/677219745 conn(0x7fa66c0fd180 msgr2=0x7fa66c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 -- 172.21.15.27:0/677219745 shutdown_connections 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 -- 172.21.15.27:0/677219745 wait complete. 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 Processor -- start 2024-04-25T01:38:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.380+0000 7fa67251d640 1 -- start start 2024-04-25T01:38:01.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa67251d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c101710 0x7fa66c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa67251d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66c1020d0 0x7fa66c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa67251d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa66c107720 0x7fa66c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa67251d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c111b30 con 0x7fa66c101710 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa67251d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1119b0 con 0x7fa66c107720 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa67251d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c111cb0 con 0x7fa66c1020d0 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c101710 0x7fa66c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c101710 0x7fa66c19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57312/0 (socket says 172.21.15.27:57312) 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 -- 172.21.15.27:0/83942417 learned_addr learned my addr 172.21.15.27:0/83942417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa670aba640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa66c107720 0x7fa66c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66b7fe640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66c1020d0 0x7fa66c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 -- 172.21.15.27:0/83942417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66c1020d0 msgr2=0x7fa66c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66c1020d0 0x7fa66c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 -- 172.21.15.27:0/83942417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa66c107720 msgr2=0x7fa66c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa66c107720 0x7fa66c19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 -- 172.21.15.27:0/83942417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c19f250 con 0x7fa66c101710 2024-04-25T01:38:01.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.381+0000 7fa66bfff640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c101710 0x7fa66c19a260 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00d950 tx=0x7fa65c00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:01.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.382+0000 7fa6697fa640 1 -- 172.21.15.27:0/83942417 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c017030 con 0x7fa66c101710 2024-04-25T01:38:01.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.382+0000 7fa6697fa640 1 -- 172.21.15.27:0/83942417 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65c004020 con 0x7fa66c101710 2024-04-25T01:38:01.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.382+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66c19f540 con 0x7fa66c101710 2024-04-25T01:38:01.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.382+0000 7fa6697fa640 1 -- 172.21.15.27:0/83942417 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c005650 con 0x7fa66c101710 2024-04-25T01:38:01.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.382+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66c1a6dd0 con 0x7fa66c101710 2024-04-25T01:38:01.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.383+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa66c0689b0 con 0x7fa66c101710 2024-04-25T01:38:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.384+0000 7fa6697fa640 1 -- 172.21.15.27:0/83942417 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa65c0041d0 con 0x7fa66c101710 2024-04-25T01:38:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.385+0000 7fa6697fa640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6540a2d80 0x7fa6540a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.385+0000 7fa66b7fe640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6540a2d80 0x7fa6540a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.385+0000 7fa6697fa640 1 -- 172.21.15.27:0/83942417 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fa65c0cf0f0 con 0x7fa66c101710 2024-04-25T01:38:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.385+0000 7fa66b7fe640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6540a2d80 0x7fa6540a5240 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa66c1070b0 tx=0x7fa660009110 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:01.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.389+0000 7fa6697fa640 1 -- 172.21.15.27:0/83942417 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa65c09b920 con 0x7fa66c101710 2024-04-25T01:38:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.574+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa66c1a7090 con 0x7fa66c101710 2024-04-25T01:38:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.575+0000 7fa6697fa640 1 -- 172.21.15.27:0/83942417 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v106) ==== 103+0+11 (secure 0 0 0) 0x7fa65c09b220 con 0x7fa66c101710 2024-04-25T01:38:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T01:38:01.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6540a2d80 msgr2=0x7fa6540a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6540a2d80 0x7fa6540a5240 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa66c1070b0 tx=0x7fa660009110 comp rx=0 tx=0).stop 2024-04-25T01:38:01.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c101710 msgr2=0x7fa66c19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c101710 0x7fa66c19a260 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00d950 tx=0x7fa65c00dd50 comp rx=0 tx=0).stop 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 shutdown_connections 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa66c107720 0x7fa66c19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66c1020d0 0x7fa66c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fa6540a2d80 0x7fa6540a5240 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 --2- 172.21.15.27:0/83942417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c101710 0x7fa66c19a260 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 >> 172.21.15.27:0/83942417 conn(0x7fa66c0fd180 msgr2=0x7fa66c10cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 shutdown_connections 2024-04-25T01:38:01.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.577+0000 7fa67251d640 1 -- 172.21.15.27:0/83942417 wait complete. 2024-04-25T01:38:01.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:38:01.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2840769177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 msgr2=0x7fdc98107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2840769177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 0x7fdc98107b10 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009930 tx=0x7fdc8802a420 comp rx=0 tx=0).stop 2024-04-25T01:38:01.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2840769177 shutdown_connections 2024-04-25T01:38:01.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2840769177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc9806a780 0x7fdc9810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2840769177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc98069dc0 0x7fdc9806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2840769177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 0x7fdc98107b10 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2840769177 >> 172.21.15.27:0/2840769177 conn(0x7fdc980776a0 msgr2=0x7fdc98077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:01.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2840769177 shutdown_connections 2024-04-25T01:38:01.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.692+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2840769177 wait complete. 2024-04-25T01:38:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc9c8e7640 1 Processor -- start 2024-04-25T01:38:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc9c8e7640 1 -- start start 2024-04-25T01:38:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc9c8e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc98069dc0 0x7fdc98198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc9c8e7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc9806a780 0x7fdc981985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc9c8e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 0x7fdc9819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc9c8e7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdc98111b00 con 0x7fdc98107730 2024-04-25T01:38:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc9c8e7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fdc98111980 con 0x7fdc9806a780 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc9c8e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc98111c80 con 0x7fdc98069dc0 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc96d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 0x7fdc9819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc96d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 0x7fdc9819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57330/0 (socket says 172.21.15.27:57330) 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.693+0000 7fdc96d9d640 1 -- 172.21.15.27:0/2676587305 learned_addr learned my addr 172.21.15.27:0/2676587305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc9659c640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc98069dc0 0x7fdc98198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc8ffff640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc9806a780 0x7fdc981985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc96d9d640 1 -- 172.21.15.27:0/2676587305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc98069dc0 msgr2=0x7fdc98198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc96d9d640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc98069dc0 0x7fdc98198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc96d9d640 1 -- 172.21.15.27:0/2676587305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc9806a780 msgr2=0x7fdc981985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc96d9d640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc9806a780 0x7fdc981985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc96d9d640 1 -- 172.21.15.27:0/2676587305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9819d050 con 0x7fdc98107730 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc8ffff640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc9806a780 0x7fdc981985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:01.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc96d9d640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 0x7fdc9819c920 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000cad0 tx=0x7fdc9000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:01.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc8f7fe640 1 -- 172.21.15.27:0/2676587305 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9000ae20 con 0x7fdc98107730 2024-04-25T01:38:01.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9819d340 con 0x7fdc98107730 2024-04-25T01:38:01.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc8f7fe640 1 -- 172.21.15.27:0/2676587305 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc90005020 con 0x7fdc98107730 2024-04-25T01:38:01.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc8f7fe640 1 -- 172.21.15.27:0/2676587305 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9000f6f0 con 0x7fdc98107730 2024-04-25T01:38:01.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.694+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc981a4bd0 con 0x7fdc98107730 2024-04-25T01:38:01.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.695+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc980689b0 con 0x7fdc98107730 2024-04-25T01:38:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.696+0000 7fdc8f7fe640 1 -- 172.21.15.27:0/2676587305 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdc900051d0 con 0x7fdc98107730 2024-04-25T01:38:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.697+0000 7fdc8f7fe640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc840a2d00 0x7fdc840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.697+0000 7fdc9659c640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc840a2d00 0x7fdc840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.697+0000 7fdc8f7fe640 1 -- 172.21.15.27:0/2676587305 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fdc900cfcb0 con 0x7fdc98107730 2024-04-25T01:38:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.697+0000 7fdc9659c640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc840a2d00 0x7fdc840a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdc88000c00 tx=0x7fdc88005a20 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.700+0000 7fdc8f7fe640 1 -- 172.21.15.27:0/2676587305 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdc90014030 con 0x7fdc98107730 2024-04-25T01:38:01.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.903+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdc9802d170 con 0x7fdc98107730 2024-04-25T01:38:01.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.903+0000 7fdc8f7fe640 1 -- 172.21.15.27:0/2676587305 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v106) ==== 81+0+940 (secure 0 0 0) 0x7fdc9009c4e0 con 0x7fdc98107730 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2238094164 2024-04-26T01:25:24.976712+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/4052240654 2024-04-26T01:25:24.976712+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3292898694 2024-04-26T01:25:24.976712+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/12403185 2024-04-26T01:15:45.288446+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4288305036 2024-04-26T01:25:24.976712+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/354957039 2024-04-26T01:15:45.288446+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/4052240654 2024-04-26T01:25:24.976712+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/3353153293 2024-04-26T01:15:45.288446+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/234792808 2024-04-26T01:15:45.288446+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3645132749 2024-04-26T01:15:18.102339+0000 2024-04-25T01:38:01.882 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/219526886 2024-04-26T01:15:18.102339+0000 2024-04-25T01:38:01.883 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/219526886 2024-04-26T01:15:18.102339+0000 2024-04-25T01:38:01.883 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2409557466 2024-04-26T01:15:18.102339+0000 2024-04-25T01:38:01.883 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4191124222 2024-04-26T01:15:18.102339+0000 2024-04-25T01:38:01.883 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2504114047 2024-04-26T01:25:24.976712+0000 2024-04-25T01:38:01.883 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/3353153293 2024-04-26T01:15:45.288446+0000 2024-04-25T01:38:01.883 INFO:teuthology.orchestra.run.smithi027.stderr:listed 16 entries 2024-04-25T01:38:01.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.905+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc840a2d00 msgr2=0x7fdc840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.905+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc840a2d00 0x7fdc840a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdc88000c00 tx=0x7fdc88005a20 comp rx=0 tx=0).stop 2024-04-25T01:38:01.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 msgr2=0x7fdc9819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:01.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 0x7fdc9819c920 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000cad0 tx=0x7fdc9000cf90 comp rx=0 tx=0).stop 2024-04-25T01:38:01.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 shutdown_connections 2024-04-25T01:38:01.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fdc840a2d00 0x7fdc840a51c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc98107730 0x7fdc9819c920 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdc9806a780 0x7fdc981985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 --2- 172.21.15.27:0/2676587305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc98069dc0 0x7fdc98198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:01.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 >> 172.21.15.27:0/2676587305 conn(0x7fdc980776a0 msgr2=0x7fdc980fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:01.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 shutdown_connections 2024-04-25T01:38:01.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:01.906+0000 7fdc9c8e7640 1 -- 172.21.15.27:0/2676587305 wait complete. 2024-04-25T01:38:01.894 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:01 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2756199890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:38:01.894 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:01 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1410852424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:38:01.894 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:01 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/967634325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:38:01.894 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:01 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/83942417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:38:01.895 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:38:01.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:38:01.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T01:38:01.936 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:38:01.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:38:01.936 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:38:01.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:38:01.992 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T01:38:02.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:01 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2756199890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:38:02.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:01 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1410852424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:38:02.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:01 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/967634325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:38:02.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:01 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/83942417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:38:02.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:01 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2756199890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:38:02.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:01 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1410852424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:38:02.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:01 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/967634325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:38:02.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:01 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/83942417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:38:02.048 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:38:02.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:38:02.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:38:02.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:38:02.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-04-25T01:38:02.215 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T01:38:02.226 INFO:tasks.cephadm:Teardown begin 2024-04-25T01:38:02.226 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:38:02.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.351+0000 7fe4c6f63640 1 -- 172.21.15.27:0/2669556044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0106650 msgr2=0x7fe4c0106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.351+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/2669556044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0106650 0x7fe4c0106a30 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009930 tx=0x7fe4b002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 -- 172.21.15.27:0/2669556044 shutdown_connections 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/2669556044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4c0101050 0x7fe4c010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/2669556044 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4c0100690 0x7fe4c0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/2669556044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0106650 0x7fe4c0106a30 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 -- 172.21.15.27:0/2669556044 >> 172.21.15.27:0/2669556044 conn(0x7fe4c00777a0 msgr2=0x7fe4c0077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 -- 172.21.15.27:0/2669556044 shutdown_connections 2024-04-25T01:38:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 -- 172.21.15.27:0/2669556044 wait complete. 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 Processor -- start 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 -- start start 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0100690 0x7fe4c0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4c0101050 0x7fe4c01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4c0106650 0x7fe4c019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c0110a70 con 0x7fe4c0100690 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c01108f0 con 0x7fe4c0106650 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.352+0000 7fe4c6f63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c0110bf0 con 0x7fe4c0101050 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0100690 0x7fe4c0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0100690 0x7fe4c0198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57344/0 (socket says 172.21.15.27:57344) 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 -- 172.21.15.27:0/1025428462 learned_addr learned my addr 172.21.15.27:0/1025428462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4bffff640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4c0101050 0x7fe4c01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c5500640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4c0106650 0x7fe4c019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 -- 172.21.15.27:0/1025428462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4c0101050 msgr2=0x7fe4c01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4c0101050 0x7fe4c01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 -- 172.21.15.27:0/1025428462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4c0106650 msgr2=0x7fe4c019c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4c0106650 0x7fe4c019c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 -- 172.21.15.27:0/1025428462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c019d050 con 0x7fe4c0100690 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c4cff640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0100690 0x7fe4c0198060 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe4b002a940 tx=0x7fe4b0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4bdffb640 1 -- 172.21.15.27:0/1025428462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b003f070 con 0x7fe4c0100690 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4bdffb640 1 -- 172.21.15.27:0/1025428462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4b0031c90 con 0x7fe4c0100690 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c019d2e0 con 0x7fe4c0100690 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4bdffb640 1 -- 172.21.15.27:0/1025428462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0035030 con 0x7fe4c0100690 2024-04-25T01:38:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.353+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c01a4bd0 con 0x7fe4c0100690 2024-04-25T01:38:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.354+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c00689b0 con 0x7fe4c0100690 2024-04-25T01:38:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.356+0000 7fe4bdffb640 1 -- 172.21.15.27:0/1025428462 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe4b0042070 con 0x7fe4c0100690 2024-04-25T01:38:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.356+0000 7fe4bdffb640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe4a80a2d80 0x7fe4a80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.356+0000 7fe4bffff640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe4a80a2d80 0x7fe4a80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.356+0000 7fe4bdffb640 1 -- 172.21.15.27:0/1025428462 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fe4b00f02c0 con 0x7fe4c0100690 2024-04-25T01:38:02.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.357+0000 7fe4bffff640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe4a80a2d80 0x7fe4a80a5240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0100540 tx=0x7fe4b400b390 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:02.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.359+0000 7fe4bdffb640 1 -- 172.21.15.27:0/1025428462 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe4b00bc5b0 con 0x7fe4c0100690 2024-04-25T01:38:02.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.584+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4c0109bc0 con 0x7fe4c0100690 2024-04-25T01:38:02.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.589+0000 7fe4bdffb640 1 -- 172.21.15.27:0/1025428462 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11883 (secure 0 0 0) 0x7fe4b00c08b0 con 0x7fe4c0100690 2024-04-25T01:38:02.568 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:38:02.568 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:36:52.671392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24505,24517]},"id":1}]} 2024-04-25T01:38:02.568 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T01:38:02.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe4a80a2d80 msgr2=0x7fe4a80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe4a80a2d80 0x7fe4a80a5240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0100540 tx=0x7fe4b400b390 comp rx=0 tx=0).stop 2024-04-25T01:38:02.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0100690 msgr2=0x7fe4c0198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0100690 0x7fe4c0198060 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe4b002a940 tx=0x7fe4b0003b90 comp rx=0 tx=0).stop 2024-04-25T01:38:02.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 shutdown_connections 2024-04-25T01:38:02.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe4c0106650 0x7fe4c019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4c0101050 0x7fe4c01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe4a80a2d80 0x7fe4a80a5240 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 --2- 172.21.15.27:0/1025428462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c0100690 0x7fe4c0198060 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 >> 172.21.15.27:0/1025428462 conn(0x7fe4c00777a0 msgr2=0x7fe4c006a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:02.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 shutdown_connections 2024-04-25T01:38:02.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.592+0000 7fe4c6f63640 1 -- 172.21.15.27:0/1025428462 wait complete. 2024-04-25T01:38:02.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:38:02.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 -- 172.21.15.27:0/202978916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 msgr2=0x7fc434106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/202978916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 0x7fc434106a30 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc430009930 tx=0x7fc43002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:02.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 -- 172.21.15.27:0/202978916 shutdown_connections 2024-04-25T01:38:02.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/202978916 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc434101050 0x7fc43410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/202978916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc434100690 0x7fc434100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/202978916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 0x7fc434106a30 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 -- 172.21.15.27:0/202978916 >> 172.21.15.27:0/202978916 conn(0x7fc4340777a0 msgr2=0x7fc434077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:02.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 -- 172.21.15.27:0/202978916 shutdown_connections 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.705+0000 7fc43c7dc640 1 -- 172.21.15.27:0/202978916 wait complete. 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43c7dc640 1 Processor -- start 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43c7dc640 1 -- start start 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43c7dc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc434100690 0x7fc434196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43c7dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc434101050 0x7fc4341974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43c7dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 0x7fc43419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43c7dc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc434110a60 con 0x7fc434106650 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43c7dc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341108e0 con 0x7fc434100690 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43c7dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc434110be0 con 0x7fc434101050 2024-04-25T01:38:02.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43ad79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 0x7fc43419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43ad79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 0x7fc43419b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57368/0 (socket says 172.21.15.27:57368) 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc43ad79640 1 -- 172.21.15.27:0/1871522186 learned_addr learned my addr 172.21.15.27:0/1871522186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.706+0000 7fc439d77640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc434101050 0x7fc4341974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43a578640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc434100690 0x7fc434196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43ad79640 1 -- 172.21.15.27:0/1871522186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc434101050 msgr2=0x7fc4341974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43ad79640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc434101050 0x7fc4341974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43ad79640 1 -- 172.21.15.27:0/1871522186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc434100690 msgr2=0x7fc434196f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43ad79640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc434100690 0x7fc434196f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43ad79640 1 -- 172.21.15.27:0/1871522186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43419bf70 con 0x7fc434106650 2024-04-25T01:38:02.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc439d77640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc434101050 0x7fc4341974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:02.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43a578640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc434100690 0x7fc434196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:02.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43ad79640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 0x7fc43419b840 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00e960 tx=0x7fc42c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:02.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc42b7fe640 1 -- 172.21.15.27:0/1871522186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c004d20 con 0x7fc434106650 2024-04-25T01:38:02.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43419c260 con 0x7fc434106650 2024-04-25T01:38:02.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc42b7fe640 1 -- 172.21.15.27:0/1871522186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc42c005020 con 0x7fc434106650 2024-04-25T01:38:02.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc42b7fe640 1 -- 172.21.15.27:0/1871522186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c0116d0 con 0x7fc434106650 2024-04-25T01:38:02.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.707+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc434069da0 con 0x7fc434106650 2024-04-25T01:38:02.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.708+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4340689b0 con 0x7fc434106650 2024-04-25T01:38:02.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.709+0000 7fc42b7fe640 1 -- 172.21.15.27:0/1871522186 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc42c005560 con 0x7fc434106650 2024-04-25T01:38:02.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.710+0000 7fc42b7fe640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4200a2d00 0x7fc4200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:02.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.710+0000 7fc43a578640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4200a2d00 0x7fc4200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:02.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.710+0000 7fc42b7fe640 1 -- 172.21.15.27:0/1871522186 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9613+0+0 (secure 0 0 0) 0x7fc42c0cfa80 con 0x7fc434106650 2024-04-25T01:38:02.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.710+0000 7fc43a578640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4200a2d00 0x7fc4200a51c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc430000c00 tx=0x7fc430032c70 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:02.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.714+0000 7fc42b7fe640 1 -- 172.21.15.27:0/1871522186 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc42c016030 con 0x7fc434106650 2024-04-25T01:38:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.955+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc43406a880 con 0x7fc434106650 2024-04-25T01:38:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.956+0000 7fc42b7fe640 1 -- 172.21.15.27:0/1871522186 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11883 (secure 0 0 0) 0x7fc42c09c260 con 0x7fc434106650 2024-04-25T01:38:02.934 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:38:02.934 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2965551549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2965551549},{"type":"v1","addr":"172.21.15.27:6835","nonce":2965551549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3543255516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3543255516},{"type":"v1","addr":"172.21.15.27:6837","nonce":3543255516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/1204805980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":1204805980},{"type":"v1","addr":"172.21.15.27:6839","nonce":1204805980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1912988733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1912988733},{"type":"v1","addr":"172.21.15.27:6841","nonce":1912988733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/4036036788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":4036036788},{"type":"v1","addr":"172.21.15.139:6833","nonce":4036036788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3706725137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3706725137},{"type":"v1","addr":"172.21.15.153:6833","nonce":3706725137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:22:57.180706+0000","modified":"2024-04-25T01:36:52.671392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24517,"mds_2":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/418855472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":418855472},{"type":"v1","addr":"172.21.15.139:6835","nonce":418855472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/1981209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1981209233},{"type":"v1","addr":"172.21.15.153:6835","nonce":1981209233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/1472553553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":1472553553},{"type":"v1","addr":"172.21.15.139:6837","nonce":1472553553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/529371680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":529371680},{"type":"v1","addr":"172.21.15.153:6837","nonce":529371680}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.153:6839/2846322026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2846322026},{"type":"v1","addr":"172.21.15.153:6839","nonce":2846322026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.139:6839/2167485871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":2167485871},{"type":"v1","addr":"172.21.15.139:6839","nonce":2167485871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24505,24517]},"id":1}]} 2024-04-25T01:38:02.935 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4200a2d00 msgr2=0x7fc4200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4200a2d00 0x7fc4200a51c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc430000c00 tx=0x7fc430032c70 comp rx=0 tx=0).stop 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 msgr2=0x7fc43419b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 0x7fc43419b840 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00e960 tx=0x7fc42c00ee20 comp rx=0 tx=0).stop 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 shutdown_connections 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fc4200a2d00 0x7fc4200a51c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc434106650 0x7fc43419b840 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc434101050 0x7fc4341974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 --2- 172.21.15.27:0/1871522186 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc434100690 0x7fc434196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 >> 172.21.15.27:0/1871522186 conn(0x7fc4340777a0 msgr2=0x7fc434103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:02.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 shutdown_connections 2024-04-25T01:38:02.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:02.958+0000 7fc43c7dc640 1 -- 172.21.15.27:0/1871522186 wait complete. 2024-04-25T01:38:02.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:02 smithi027 ceph-mon[29197]: pgmap v393: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-25T01:38:02.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:02 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/2676587305' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:38:02.948 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:38:03.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:02 smithi153 ceph-mon[32621]: pgmap v393: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-25T01:38:03.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:02 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/2676587305' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:38:03.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:02 smithi139 ceph-mon[32760]: pgmap v393: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-25T01:38:03.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:02 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/2676587305' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 -- 172.21.15.27:0/4284457913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c1066b0 msgr2=0x7f6b3c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 --2- 172.21.15.27:0/4284457913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c1066b0 0x7f6b3c106a90 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009930 tx=0x7f6b2c02a420 comp rx=0 tx=0).stop 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 -- 172.21.15.27:0/4284457913 shutdown_connections 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 --2- 172.21.15.27:0/4284457913 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6b3c100fd0 0x7f6b3c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 --2- 172.21.15.27:0/4284457913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c100610 0x7f6b3c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 --2- 172.21.15.27:0/4284457913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c1066b0 0x7f6b3c106a90 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 -- 172.21.15.27:0/4284457913 >> 172.21.15.27:0/4284457913 conn(0x7f6b3c0769c0 msgr2=0x7f6b3c076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 -- 172.21.15.27:0/4284457913 shutdown_connections 2024-04-25T01:38:03.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.071+0000 7f6b40e15640 1 -- 172.21.15.27:0/4284457913 wait complete. 2024-04-25T01:38:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b40e15640 1 Processor -- start 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b40e15640 1 -- start start 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b40e15640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c100610 0x7f6b3c198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b40e15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c100fd0 0x7f6b3c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b40e15640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6b3c1066b0 0x7f6b3c1a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b40e15640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c110a30 con 0x7f6b3c100fd0 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b40e15640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1108b0 con 0x7f6b3c1066b0 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b40e15640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c110bb0 con 0x7f6b3c100610 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.072+0000 7f6b31d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c100fd0 0x7f6b3c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b31d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c100fd0 0x7f6b3c199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57392/0 (socket says 172.21.15.27:57392) 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b31d9b640 1 -- 172.21.15.27:0/3160593634 learned_addr learned my addr 172.21.15.27:0/3160593634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b3a59c640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c100610 0x7f6b3c198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b3ad9d640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6b3c1066b0 0x7f6b3c1a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b31d9b640 1 -- 172.21.15.27:0/3160593634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c100610 msgr2=0x7f6b3c198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b31d9b640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c100610 0x7f6b3c198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b31d9b640 1 -- 172.21.15.27:0/3160593634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6b3c1066b0 msgr2=0x7f6b3c1a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b31d9b640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6b3c1066b0 0x7f6b3c1a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b31d9b640 1 -- 172.21.15.27:0/3160593634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c1a4490 con 0x7f6b3c100fd0 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b3a59c640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c100610 0x7f6b3c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b3ad9d640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6b3c1066b0 0x7f6b3c1a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b31d9b640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c100fd0 0x7f6b3c199410 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400b4a0 tx=0x7f6b2400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b33fff640 1 -- 172.21.15.27:0/3160593634 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24014070 con 0x7f6b3c100fd0 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b3c1a4660 con 0x7f6b3c100fd0 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b33fff640 1 -- 172.21.15.27:0/3160593634 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b24010af0 con 0x7f6b3c100fd0 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b33fff640 1 -- 172.21.15.27:0/3160593634 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24017d10 con 0x7f6b3c100fd0 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.073+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3c1a4ba0 con 0x7f6b3c100fd0 2024-04-25T01:38:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.074+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b3c0689b0 con 0x7f6b3c100fd0 2024-04-25T01:38:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.075+0000 7f6b33fff640 1 -- 172.21.15.27:0/3160593634 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6b24017070 con 0x7f6b3c100fd0 2024-04-25T01:38:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.076+0000 7f6b33fff640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6b280a2d00 0x7f6b280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.076+0000 7f6b3a59c640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6b280a2d00 0x7f6b280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.076+0000 7f6b33fff640 1 -- 172.21.15.27:0/3160593634 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9613+0+0 (secure 0 0 0) 0x7f6b240cef40 con 0x7f6b3c100fd0 2024-04-25T01:38:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.077+0000 7f6b3a59c640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6b280a2d00 0x7f6b280a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006010 tx=0x7f6b2c005a20 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.080+0000 7f6b33fff640 1 -- 172.21.15.27:0/3160593634 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6b2409b830 con 0x7f6b3c100fd0 2024-04-25T01:38:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.266+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b3c108b80 con 0x7f6b3c100fd0 2024-04-25T01:38:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.268+0000 7f6b33fff640 1 -- 172.21.15.27:0/3160593634 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+25491 (secure 0 0 0) 0x7f6b2409b130 con 0x7f6b3c100fd0 2024-04-25T01:38:03.246 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T01:38:03.247 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":107,"fsid":"1469ec2e-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:14:45.961368+0000","modified":"2024-04-25T01:38:02.581641+0000","last_up_change":"2024-04-25T01:22:04.088343+0000","last_in_change":"2024-04-25T01:21:43.291567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:18:11.811188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:22:55.765659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:22:56.604736+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:23:00.289710+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"88cb2314-6b03-44ba-b2ec-e38198da56b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6803","nonce":4134559555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6805","nonce":4134559555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6809","nonce":4134559555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4134559555},{"type":"v1","addr":"172.21.15.27:6807","nonce":4134559555}]},"public_addr":"172.21.15.27:6803/4134559555","cluster_addr":"172.21.15.27:6805/4134559555","heartbeat_back_addr":"172.21.15.27:6809/4134559555","heartbeat_front_addr":"172.21.15.27:6807/4134559555","state":["exists","up"]},{"osd":1,"uuid":"817954a3-7495-44a0-9983-511339d330f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6801","nonce":4036947883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6803","nonce":4036947883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6807","nonce":4036947883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":4036947883},{"type":"v1","addr":"172.21.15.139:6805","nonce":4036947883}]},"public_addr":"172.21.15.139:6801/4036947883","cluster_addr":"172.21.15.139:6803/4036947883","heartbeat_back_addr":"172.21.15.139:6807/4036947883","heartbeat_front_addr":"172.21.15.139:6805/4036947883","state":["exists","up"]},{"osd":2,"uuid":"cbaef244-9159-4e52-912a-0c7d4857f6fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6801","nonce":3400401807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6803","nonce":3400401807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6807","nonce":3400401807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3400401807},{"type":"v1","addr":"172.21.15.153:6805","nonce":3400401807}]},"public_addr":"172.21.15.153:6801/3400401807","cluster_addr":"172.21.15.153:6803/3400401807","heartbeat_back_addr":"172.21.15.153:6807/3400401807","heartbeat_front_addr":"172.21.15.153:6805/3400401807","state":["exists","up"]},{"osd":3,"uuid":"205e94f9-7782-426b-b5bd-1b56c93d55c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6811","nonce":2907204232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6813","nonce":2907204232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6817","nonce":2907204232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2907204232},{"type":"v1","addr":"172.21.15.27:6815","nonce":2907204232}]},"public_addr":"172.21.15.27:6811/2907204232","cluster_addr":"172.21.15.27:6813/2907204232","heartbeat_back_addr":"172.21.15.27:6817/2907204232","heartbeat_front_addr":"172.21.15.27:6815/2907204232","state":["exists","up"]},{"osd":4,"uuid":"57cc1b06-8233-435d-af98-2de145cfcc8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6809","nonce":1457327101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6811","nonce":1457327101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6815","nonce":1457327101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1457327101},{"type":"v1","addr":"172.21.15.139:6813","nonce":1457327101}]},"public_addr":"172.21.15.139:6809/1457327101","cluster_addr":"172.21.15.139:6811/1457327101","heartbeat_back_addr":"172.21.15.139:6815/1457327101","heartbeat_front_addr":"172.21.15.139:6813/1457327101","state":["exists","up"]},{"osd":5,"uuid":"087587ee-f7ae-4546-a39a-2a6a774c54ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6809","nonce":4140079274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6811","nonce":4140079274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6815","nonce":4140079274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4140079274},{"type":"v1","addr":"172.21.15.153:6813","nonce":4140079274}]},"public_addr":"172.21.15.153:6809/4140079274","cluster_addr":"172.21.15.153:6811/4140079274","heartbeat_back_addr":"172.21.15.153:6815/4140079274","heartbeat_front_addr":"172.21.15.153:6813/4140079274","state":["exists","up"]},{"osd":6,"uuid":"6834f4e7-9977-44db-8075-4e7596c95a0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6819","nonce":1607133562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6821","nonce":1607133562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6825","nonce":1607133562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1607133562},{"type":"v1","addr":"172.21.15.27:6823","nonce":1607133562}]},"public_addr":"172.21.15.27:6819/1607133562","cluster_addr":"172.21.15.27:6821/1607133562","heartbeat_back_addr":"172.21.15.27:6825/1607133562","heartbeat_front_addr":"172.21.15.27:6823/1607133562","state":["exists","up"]},{"osd":7,"uuid":"c6f1241c-9012-47d6-80e3-ef957fca03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6817","nonce":3705167644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6819","nonce":3705167644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6823","nonce":3705167644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3705167644},{"type":"v1","addr":"172.21.15.139:6821","nonce":3705167644}]},"public_addr":"172.21.15.139:6817/3705167644","cluster_addr":"172.21.15.139:6819/3705167644","heartbeat_back_addr":"172.21.15.139:6823/3705167644","heartbeat_front_addr":"172.21.15.139:6821/3705167644","state":["exists","up"]},{"osd":8,"uuid":"7ceef8a0-8a21-4256-ab43-ad6e4c1cda16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6817","nonce":4050878685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6819","nonce":4050878685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6823","nonce":4050878685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4050878685},{"type":"v1","addr":"172.21.15.153:6821","nonce":4050878685}]},"public_addr":"172.21.15.153:6817/4050878685","cluster_addr":"172.21.15.153:6819/4050878685","heartbeat_back_addr":"172.21.15.153:6823/4050878685","heartbeat_front_addr":"172.21.15.153:6821/4050878685","state":["exists","up"]},{"osd":9,"uuid":"132f821e-9337-40f8-b79b-02322b1a7962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6827","nonce":91302017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6829","nonce":91302017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6833","nonce":91302017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":91302017},{"type":"v1","addr":"172.21.15.27:6831","nonce":91302017}]},"public_addr":"172.21.15.27:6827/91302017","cluster_addr":"172.21.15.27:6829/91302017","heartbeat_back_addr":"172.21.15.27:6833/91302017","heartbeat_front_addr":"172.21.15.27:6831/91302017","state":["exists","up"]},{"osd":10,"uuid":"a6ee88f9-6df0-4fbe-af82-401e2c864c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6825","nonce":155195264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6827","nonce":155195264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6831","nonce":155195264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":155195264},{"type":"v1","addr":"172.21.15.139:6829","nonce":155195264}]},"public_addr":"172.21.15.139:6825/155195264","cluster_addr":"172.21.15.139:6827/155195264","heartbeat_back_addr":"172.21.15.139:6831/155195264","heartbeat_front_addr":"172.21.15.139:6829/155195264","state":["exists","up"]},{"osd":11,"uuid":"79072415-ed78-47ee-b950-e02e8c9c30c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6825","nonce":3171978672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6827","nonce":3171978672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6831","nonce":3171978672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3171978672},{"type":"v1","addr":"172.21.15.153:6829","nonce":3171978672}]},"public_addr":"172.21.15.153:6825/3171978672","cluster_addr":"172.21.15.153:6827/3171978672","heartbeat_back_addr":"172.21.15.153:6831/3171978672","heartbeat_front_addr":"172.21.15.153:6829/3171978672","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:21.100382+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:17:44.743570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:08.876017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:34.937483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:18:59.781044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:24.782594+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:51.397374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:16.864502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:42.339643+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:09.396226+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:35.418519+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:01.765462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/2238094164":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/3292898694":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/12403185":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/4288305036":"2024-04-26T01:25:24.976712+0000","172.21.15.27:0/354957039":"2024-04-26T01:15:45.288446+0000","172.21.15.27:6800/4052240654":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6801/3353153293":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/234792808":"2024-04-26T01:15:45.288446+0000","172.21.15.27:0/3645132749":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6801/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:6800/219526886":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2409557466":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/4191124222":"2024-04-26T01:15:18.102339+0000","172.21.15.27:0/2504114047":"2024-04-26T01:25:24.976712+0000","172.21.15.27:6800/3353153293":"2024-04-26T01:15:45.288446+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:38:03.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6b280a2d00 msgr2=0x7f6b280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6b280a2d00 0x7f6b280a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006010 tx=0x7f6b2c005a20 comp rx=0 tx=0).stop 2024-04-25T01:38:03.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c100fd0 msgr2=0x7f6b3c199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c100fd0 0x7f6b3c199410 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400b4a0 tx=0x7f6b2400b960 comp rx=0 tx=0).stop 2024-04-25T01:38:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 shutdown_connections 2024-04-25T01:38:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6b3c1066b0 0x7f6b3c1a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f6b280a2d00 0x7f6b280a51c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b3c100fd0 0x7f6b3c199410 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 --2- 172.21.15.27:0/3160593634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c100610 0x7f6b3c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 >> 172.21.15.27:0/3160593634 conn(0x7f6b3c0769c0 msgr2=0x7f6b3c0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 shutdown_connections 2024-04-25T01:38:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.270+0000 7f6b40e15640 1 -- 172.21.15.27:0/3160593634 wait complete. 2024-04-25T01:38:03.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.384+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/3933152050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 msgr2=0x7f5d24107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.384+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/3933152050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 0x7f5d24107b00 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009930 tx=0x7f5d1402a420 comp rx=0 tx=0).stop 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.384+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/3933152050 shutdown_connections 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.384+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/3933152050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d241020d0 0x7f5d2410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.384+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/3933152050 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d24101710 0x7f5d24101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.384+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/3933152050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 0x7f5d24107b00 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.384+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/3933152050 >> 172.21.15.27:0/3933152050 conn(0x7f5d24078b90 msgr2=0x7f5d24078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/3933152050 shutdown_connections 2024-04-25T01:38:03.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/3933152050 wait complete. 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 Processor -- start 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 -- start start 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d24101710 0x7f5d24198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d241020d0 0x7f5d241985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 0x7f5d2419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24111b30 con 0x7f5d24107720 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5d241119b0 con 0x7f5d24101710 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2bdb7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24111cb0 con 0x7f5d241020d0 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2a354640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 0x7f5d2419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d29b53640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d24101710 0x7f5d24198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2a354640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 0x7f5d2419c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57410/0 (socket says 172.21.15.27:57410) 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.385+0000 7f5d2a354640 1 -- 172.21.15.27:0/1916887473 learned_addr learned my addr 172.21.15.27:0/1916887473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d29352640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d241020d0 0x7f5d241985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d2a354640 1 -- 172.21.15.27:0/1916887473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d241020d0 msgr2=0x7f5d241985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d2a354640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d241020d0 0x7f5d241985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d2a354640 1 -- 172.21.15.27:0/1916887473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d24101710 msgr2=0x7f5d24198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d2a354640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d24101710 0x7f5d24198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d2a354640 1 -- 172.21.15.27:0/1916887473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2419d060 con 0x7f5d24107720 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d29b53640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d24101710 0x7f5d24198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d2a354640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 0x7f5d2419c930 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00cad0 tx=0x7f5d1c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d1affd640 1 -- 172.21.15.27:0/1916887473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00ae20 con 0x7f5d24107720 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d1affd640 1 -- 172.21.15.27:0/1916887473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d1c005020 con 0x7f5d24107720 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d1affd640 1 -- 172.21.15.27:0/1916887473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00f6f0 con 0x7f5d24107720 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2419d350 con 0x7f5d24107720 2024-04-25T01:38:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.386+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d241a4be0 con 0x7f5d24107720 2024-04-25T01:38:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.387+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d240689b0 con 0x7f5d24107720 2024-04-25T01:38:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.388+0000 7f5d1affd640 1 -- 172.21.15.27:0/1916887473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5d1c0051d0 con 0x7f5d24107720 2024-04-25T01:38:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.389+0000 7f5d1affd640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5d100a2d80 0x7f5d100a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.389+0000 7f5d29b53640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5d100a2d80 0x7f5d100a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.389+0000 7f5d1affd640 1 -- 172.21.15.27:0/1916887473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9613+0+0 (secure 0 0 0) 0x7f5d1c0cfa40 con 0x7f5d24107720 2024-04-25T01:38:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.389+0000 7f5d29b53640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5d100a2d80 0x7f5d100a5240 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009900 tx=0x7f5d140058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.394+0000 7f5d1affd640 1 -- 172.21.15.27:0/1916887473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5d1c014030 con 0x7f5d24107720 2024-04-25T01:38:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.617+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5d241096f0 con 0x7f5d24107720 2024-04-25T01:38:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.619+0000 7f5d1affd640 1 -- 172.21.15.27:0/1916887473 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7f5d1c09bbd0 con 0x7f5d24107720 2024-04-25T01:38:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:38:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5d100a2d80 msgr2=0x7f5d100a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5d100a2d80 0x7f5d100a5240 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009900 tx=0x7f5d140058d0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 msgr2=0x7f5d2419c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 0x7f5d2419c930 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00cad0 tx=0x7f5d1c00cf90 comp rx=0 tx=0).stop 2024-04-25T01:38:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 shutdown_connections 2024-04-25T01:38:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f5d100a2d80 0x7f5d100a5240 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d24107720 0x7f5d2419c930 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d241020d0 0x7f5d241985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 --2- 172.21.15.27:0/1916887473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5d24101710 0x7f5d24198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.622+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 >> 172.21.15.27:0/1916887473 conn(0x7f5d24078b90 msgr2=0x7f5d240ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.623+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 shutdown_connections 2024-04-25T01:38:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.623+0000 7f5d2bdb7640 1 -- 172.21.15.27:0/1916887473 wait complete. 2024-04-25T01:38:03.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 -- 172.21.15.27:0/4190428366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 msgr2=0x7fe94010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 --2- 172.21.15.27:0/4190428366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 0x7fe94010d4f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fe938009cd0 tx=0x7fe93802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 -- 172.21.15.27:0/4190428366 shutdown_connections 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 --2- 172.21.15.27:0/4190428366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 0x7fe94010d4f0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 --2- 172.21.15.27:0/4190428366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe940100620 0x7fe940100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 --2- 172.21.15.27:0/4190428366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe940106630 0x7fe940106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 -- 172.21.15.27:0/4190428366 >> 172.21.15.27:0/4190428366 conn(0x7fe9400776a0 msgr2=0x7fe940077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 -- 172.21.15.27:0/4190428366 shutdown_connections 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 -- 172.21.15.27:0/4190428366 wait complete. 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 Processor -- start 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.741+0000 7fe9464d4640 1 -- start start 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe9464d4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe940100620 0x7fe9400fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe9464d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 0x7fe9400fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe9464d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe940106630 0x7fe9400fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe9464d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe940110a90 con 0x7fe940100fe0 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe9464d4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe940110910 con 0x7fe940100620 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe9464d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe940110c10 con 0x7fe940106630 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 0x7fe9400fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 0x7fe9400fdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57420/0 (socket says 172.21.15.27:57420) 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 -- 172.21.15.27:0/772708982 learned_addr learned my addr 172.21.15.27:0/772708982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe944a71640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe940106630 0x7fe9400fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93ffff640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe940100620 0x7fe9400fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 -- 172.21.15.27:0/772708982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe940106630 msgr2=0x7fe9400fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe940106630 0x7fe9400fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 -- 172.21.15.27:0/772708982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe940100620 msgr2=0x7fe9400fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe940100620 0x7fe9400fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 -- 172.21.15.27:0/772708982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9401a65a0 con 0x7fe940100fe0 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.742+0000 7fe93f7fe640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 0x7fe9400fdb90 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fe93400e9f0 tx=0x7fe93400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.743+0000 7fe93d7fa640 1 -- 172.21.15.27:0/772708982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93400cd80 con 0x7fe940100fe0 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.743+0000 7fe93d7fa640 1 -- 172.21.15.27:0/772708982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe934010910 con 0x7fe940100fe0 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.743+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9401a6890 con 0x7fe940100fe0 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.743+0000 7fe93d7fa640 1 -- 172.21.15.27:0/772708982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe934018990 con 0x7fe940100fe0 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.743+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9401a6de0 con 0x7fe940100fe0 2024-04-25T01:38:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.744+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9400689b0 con 0x7fe940100fe0 2024-04-25T01:38:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.745+0000 7fe93d7fa640 1 -- 172.21.15.27:0/772708982 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe934010ac0 con 0x7fe940100fe0 2024-04-25T01:38:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.745+0000 7fe93d7fa640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9280a2d00 0x7fe9280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.746+0000 7fe93ffff640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9280a2d00 0x7fe9280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.746+0000 7fe93d7fa640 1 -- 172.21.15.27:0/772708982 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9783+0+0 (secure 0 0 0) 0x7fe9340d0140 con 0x7fe940100fe0 2024-04-25T01:38:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.746+0000 7fe93ffff640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9280a2d00 0x7fe9280a51c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe930009ae0 tx=0x7fe930005f90 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.749+0000 7fe93d7fa640 1 -- 172.21.15.27:0/772708982 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe93409c870 con 0x7fe940100fe0 2024-04-25T01:38:03.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:03 smithi027 ceph-mon[29197]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T01:38:03.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1025428462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:03.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1871522186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:03.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:03 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3160593634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:03.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a[29193]: 2024-04-25T01:38:03.618+0000 7fa2deb74700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:38:03.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:03.974+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe940109bc0 con 0x7fe940100fe0 2024-04-25T01:38:04.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-mon[32621]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T01:38:04.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1025428462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:04.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1871522186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:04.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3160593634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:04.025 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-c[63707]: starting mds.c at 2024-04-25T01:38:04.025 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f[63398]: ignoring --setuser ceph since I am not root 2024-04-25T01:38:04.025 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f[63398]: ignoring --setgroup ceph since I am not root 2024-04-25T01:38:04.025 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-mds[63402]: -- 172.21.15.153:0/2272983244 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561fd84fba40 con 0x561fd84cec00 2024-04-25T01:38:04.025 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f[63398]: starting mds.f at 2024-04-25T01:38:04.026 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i[64015]: ignoring --setuser ceph since I am not root 2024-04-25T01:38:04.026 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i[64015]: ignoring --setgroup ceph since I am not root 2024-04-25T01:38:04.026 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-mds[64019]: -- 172.21.15.153:0/501055241 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562333493a40 con 0x562333466c00 2024-04-25T01:38:04.026 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:38:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i[64015]: starting mds.i at 2024-04-25T01:38:04.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-mon[32760]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T01:38:04.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1025428462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:04.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1871522186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:04.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3160593634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:04.041 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b[63104]: ignoring --setuser ceph since I am not root 2024-04-25T01:38:04.041 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b[63104]: ignoring --setgroup ceph since I am not root 2024-04-25T01:38:04.042 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-mds[63108]: -- 172.21.15.139:0/2453564233 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562fdb177a40 con 0x562fdb14ac00 2024-04-25T01:38:04.042 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b[63104]: starting mds.b at 2024-04-25T01:38:04.042 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e[62486]: ignoring --setuser ceph since I am not root 2024-04-25T01:38:04.042 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e[62486]: ignoring --setgroup ceph since I am not root 2024-04-25T01:38:04.042 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-mds[62490]: -- 172.21.15.139:0/5937883 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5620f6a27a40 con 0x5620f69fac00 2024-04-25T01:38:04.042 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e[62486]: starting mds.e at 2024-04-25T01:38:04.042 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h[62797]: ignoring --setuser ceph since I am not root 2024-04-25T01:38:04.042 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h[62797]: ignoring --setgroup ceph since I am not root 2024-04-25T01:38:04.043 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-mds[62801]: -- 172.21.15.139:0/2855812753 <== mon.2 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ecef67a40 con 0x562ecef3b000 2024-04-25T01:38:04.043 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:38:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h[62797]: starting mds.h at 2024-04-25T01:38:04.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.621+0000 7fe93d7fa640 1 -- 172.21.15.27:0/772708982 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7fe93409c170 con 0x7fe940100fe0 2024-04-25T01:38:04.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9280a2d00 msgr2=0x7fe9280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9280a2d00 0x7fe9280a51c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe930009ae0 tx=0x7fe930005f90 comp rx=0 tx=0).stop 2024-04-25T01:38:04.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 msgr2=0x7fe9400fdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 0x7fe9400fdb90 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fe93400e9f0 tx=0x7fe93400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 shutdown_connections 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe940106630 0x7fe9400fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7fe9280a2d00 0x7fe9280a51c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe940100fe0 0x7fe9400fdb90 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 --2- 172.21.15.27:0/772708982 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe940100620 0x7fe9400fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 >> 172.21.15.27:0/772708982 conn(0x7fe9400776a0 msgr2=0x7fe94006a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 shutdown_connections 2024-04-25T01:38:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.624+0000 7fe9464d4640 1 -- 172.21.15.27:0/772708982 wait complete. 2024-04-25T01:38:04.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:38:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 -- 172.21.15.27:0/3397726763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd81066b0 msgr2=0x7f8cd8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/3397726763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd81066b0 0x7f8cd8106a90 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009930 tx=0x7f8cc802a420 comp rx=0 tx=0).stop 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 -- 172.21.15.27:0/3397726763 shutdown_connections 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/3397726763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cd8100fd0 0x7f8cd810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/3397726763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd8100610 0x7f8cd8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/3397726763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd81066b0 0x7f8cd8106a90 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 -- 172.21.15.27:0/3397726763 >> 172.21.15.27:0/3397726763 conn(0x7f8cd8077740 msgr2=0x7f8cd8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 -- 172.21.15.27:0/3397726763 shutdown_connections 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.752+0000 7f8ce00d2640 1 -- 172.21.15.27:0/3397726763 wait complete. 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8ce00d2640 1 Processor -- start 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8ce00d2640 1 -- start start 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8ce00d2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cd8100610 0x7f8cd81022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8ce00d2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd8100fd0 0x7f8cd81027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8ce00d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd81066b0 0x7f8cd8102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8ce00d2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd8110a30 con 0x7f8cd8100fd0 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8ce00d2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81108b0 con 0x7f8cd8100610 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8ce00d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd8110bb0 con 0x7f8cd81066b0 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdd66d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd8100fd0 0x7f8cd81027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdd66d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd8100fd0 0x7f8cd81027f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57438/0 (socket says 172.21.15.27:57438) 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdd66d640 1 -- 172.21.15.27:0/329042760 learned_addr learned my addr 172.21.15.27:0/329042760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdde6e640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cd8100610 0x7f8cd81022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cde66f640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd81066b0 0x7f8cd8102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdd66d640 1 -- 172.21.15.27:0/329042760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd81066b0 msgr2=0x7f8cd8102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdd66d640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd81066b0 0x7f8cd8102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdd66d640 1 -- 172.21.15.27:0/329042760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cd8100610 msgr2=0x7f8cd81022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdd66d640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cd8100610 0x7f8cd81022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.753+0000 7f8cdd66d640 1 -- 172.21.15.27:0/329042760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd81a64b0 con 0x7f8cd8100fd0 2024-04-25T01:38:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.754+0000 7f8cdd66d640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd8100fd0 0x7f8cd81027f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400c890 tx=0x7f8cd400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.754+0000 7f8cceffd640 1 -- 172.21.15.27:0/329042760 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4004220 con 0x7f8cd8100fd0 2024-04-25T01:38:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.754+0000 7f8cceffd640 1 -- 172.21.15.27:0/329042760 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cd40043c0 con 0x7f8cd8100fd0 2024-04-25T01:38:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.754+0000 7f8cceffd640 1 -- 172.21.15.27:0/329042760 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd40049e0 con 0x7f8cd8100fd0 2024-04-25T01:38:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.754+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd81a67a0 con 0x7f8cd8100fd0 2024-04-25T01:38:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.754+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd81a6cf0 con 0x7f8cd8100fd0 2024-04-25T01:38:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.755+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cd8068560 con 0x7f8cd8100fd0 2024-04-25T01:38:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.756+0000 7f8cceffd640 1 -- 172.21.15.27:0/329042760 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8cd40079b0 con 0x7f8cd8100fd0 2024-04-25T01:38:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.757+0000 7f8cceffd640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8cc40a2d00 0x7f8cc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.757+0000 7f8cdde6e640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8cc40a2d00 0x7f8cc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.757+0000 7f8cceffd640 1 -- 172.21.15.27:0/329042760 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9783+0+0 (secure 0 0 0) 0x7f8cd40cf920 con 0x7f8cd8100fd0 2024-04-25T01:38:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.757+0000 7f8cdde6e640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8cc40a2d00 0x7f8cc40a51c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8031550 tx=0x7f8cc80058d0 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.765+0000 7f8cceffd640 1 -- 172.21.15.27:0/329042760 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8cd4014030 con 0x7f8cd8100fd0 2024-04-25T01:38:04.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:04 smithi027 ceph-mon[29197]: pgmap v395: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-25T01:38:04.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1916887473' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:38:04.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:04 smithi027 ceph-mon[29197]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:38:04.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:04 smithi027 ceph-mon[29197]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:38:04.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/1916887473' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:38:04.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:04 smithi027 ceph-mon[29197]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T01:38:04.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:04 smithi027 ceph-mon[29197]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T01:38:04.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:04 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/772708982' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:38:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:04.950+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8cd81037e0 con 0x7f8cd8100fd0 2024-04-25T01:38:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:04 smithi153 ceph-mon[32621]: pgmap v395: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-25T01:38:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:04 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1916887473' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:38:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:04 smithi153 ceph-mon[32621]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:38:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:04 smithi153 ceph-mon[32621]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:38:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:04 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/1916887473' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:38:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:04 smithi153 ceph-mon[32621]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T01:38:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:04 smithi153 ceph-mon[32621]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T01:38:05.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:04 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/772708982' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:38:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:04 smithi139 ceph-mon[32760]: pgmap v395: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-25T01:38:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:04 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1916887473' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:38:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:04 smithi139 ceph-mon[32760]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:38:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:04 smithi139 ceph-mon[32760]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:38:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:04 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/1916887473' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:38:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:04 smithi139 ceph-mon[32760]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T01:38:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:04 smithi139 ceph-mon[32760]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T01:38:05.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:04 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/772708982' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:38:05.606 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-04-25T01:38:05.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.627+0000 7f8cceffd640 1 -- 172.21.15.27:0/329042760 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v109) ==== 181+0+0 (secure 0 0 0) 0x7f8cd81037e0 con 0x7f8cd8100fd0 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8cc40a2d00 msgr2=0x7f8cc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8cc40a2d00 0x7f8cc40a51c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8031550 tx=0x7f8cc80058d0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd8100fd0 msgr2=0x7f8cd81027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd8100fd0 0x7f8cd81027f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400c890 tx=0x7f8cd400cd50 comp rx=0 tx=0).stop 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 shutdown_connections 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd81066b0 0x7f8cd8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f8cc40a2d00 0x7f8cc40a51c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cd8100fd0 0x7f8cd81027f0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 --2- 172.21.15.27:0/329042760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8cd8100610 0x7f8cd81022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 >> 172.21.15.27:0/329042760 conn(0x7f8cd8077740 msgr2=0x7f8cd80ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 shutdown_connections 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.631+0000 7f8ce00d2640 1 -- 172.21.15.27:0/329042760 wait complete. 2024-04-25T01:38:05.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:38:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.750+0000 7ff05b49e640 1 -- 172.21.15.27:0/2243561608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 msgr2=0x7ff05410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.750+0000 7ff05b49e640 1 --2- 172.21.15.27:0/2243561608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 0x7ff05410e1a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff04c009ca0 tx=0x7ff04c02a9e0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.750+0000 7ff05b49e640 1 -- 172.21.15.27:0/2243561608 shutdown_connections 2024-04-25T01:38:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.750+0000 7ff05b49e640 1 --2- 172.21.15.27:0/2243561608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 0x7ff05410e1a0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.750+0000 7ff05b49e640 1 --2- 172.21.15.27:0/2243561608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0541032e0 0x7ff05410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.750+0000 7ff05b49e640 1 --2- 172.21.15.27:0/2243561608 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0541029c0 0x7ff054102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.750+0000 7ff05b49e640 1 -- 172.21.15.27:0/2243561608 >> 172.21.15.27:0/2243561608 conn(0x7ff054077740 msgr2=0x7ff054077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 -- 172.21.15.27:0/2243561608 shutdown_connections 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 -- 172.21.15.27:0/2243561608 wait complete. 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 Processor -- start 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 -- start start 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0541029c0 0x7ff054100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0541032e0 0x7ff054100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 0x7ff054101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff054111b00 con 0x7ff05410bd00 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff054111980 con 0x7ff0541032e0 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff05b49e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff054111c80 con 0x7ff0541029c0 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.751+0000 7ff059a3b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 0x7ff054101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff059a3b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 0x7ff054101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57468/0 (socket says 172.21.15.27:57468) 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff059a3b640 1 -- 172.21.15.27:0/3028877231 learned_addr learned my addr 172.21.15.27:0/3028877231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff05923a640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0541029c0 0x7ff054100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff058a39640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0541032e0 0x7ff054100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff059a3b640 1 -- 172.21.15.27:0/3028877231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0541029c0 msgr2=0x7ff054100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff059a3b640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0541029c0 0x7ff054100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff059a3b640 1 -- 172.21.15.27:0/3028877231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0541032e0 msgr2=0x7ff054100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff059a3b640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0541032e0 0x7ff054100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff059a3b640 1 -- 172.21.15.27:0/3028877231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0541a6480 con 0x7ff05410bd00 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff05923a640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0541029c0 0x7ff054100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff058a39640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0541032e0 0x7ff054100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff059a3b640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 0x7ff054101090 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff04c006010 tx=0x7ff04c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff04a7fc640 1 -- 172.21.15.27:0/3028877231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c047070 con 0x7ff05410bd00 2024-04-25T01:38:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff04a7fc640 1 -- 172.21.15.27:0/3028877231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04c003e50 con 0x7ff05410bd00 2024-04-25T01:38:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff04a7fc640 1 -- 172.21.15.27:0/3028877231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c033050 con 0x7ff05410bd00 2024-04-25T01:38:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0541a6710 con 0x7ff05410bd00 2024-04-25T01:38:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.752+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0541a6c00 con 0x7ff05410bd00 2024-04-25T01:38:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.753+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff054068560 con 0x7ff05410bd00 2024-04-25T01:38:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.755+0000 7ff04a7fc640 1 -- 172.21.15.27:0/3028877231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff04c04a070 con 0x7ff05410bd00 2024-04-25T01:38:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.755+0000 7ff04a7fc640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff0400a2d30 0x7ff0400a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.755+0000 7ff05923a640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff0400a2d30 0x7ff0400a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.755+0000 7ff04a7fc640 1 -- 172.21.15.27:0/3028877231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9229+0+0 (secure 0 0 0) 0x7ff04c0f08f0 con 0x7ff05410bd00 2024-04-25T01:38:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.756+0000 7ff05923a640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff0400a2d30 0x7ff0400a51f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff050009b40 tx=0x7ff050005e50 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.763+0000 7ff04a7fc640 1 -- 172.21.15.27:0/3028877231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff04c051080 con 0x7ff05410bd00 2024-04-25T01:38:05.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:38:05.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:38:05.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: Cluster is now healthy 2024-04-25T01:38:05.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/772708982' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:38:05.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6834/2271414287,v1:172.21.15.153:6835/2271414287] up:boot 2024-04-25T01:38:05.890 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.139:6834/2724174590,v1:172.21.15.139:6835/2724174590] up:boot 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.139:6836/102201551,v1:172.21.15.139:6838/102201551] up:boot 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.139:6837/947723603,v1:172.21.15.139:6839/947723603] up:boot 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6836/2394172155,v1:172.21.15.153:6838/2394172155] up:boot 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: mds.? [v2:172.21.15.153:6837/8260919,v1:172.21.15.153:6839/8260919] up:boot 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: fsmap 12 up:standby 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:38:05.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:05 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/329042760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:05.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:05.953+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff05402d170 con 0x7ff05410bd00 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: Cluster is now healthy 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/772708982' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6834/2271414287,v1:172.21.15.153:6835/2271414287] up:boot 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.139:6834/2724174590,v1:172.21.15.139:6835/2724174590] up:boot 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.139:6836/102201551,v1:172.21.15.139:6838/102201551] up:boot 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.139:6837/947723603,v1:172.21.15.139:6839/947723603] up:boot 2024-04-25T01:38:06.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6836/2394172155,v1:172.21.15.153:6838/2394172155] up:boot 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: mds.? [v2:172.21.15.153:6837/8260919,v1:172.21.15.153:6839/8260919] up:boot 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: fsmap 12 up:standby 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:38:06.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:05 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/329042760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:06.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:38:06.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:38:06.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: Cluster is now healthy 2024-04-25T01:38:06.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/772708982' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:38:06.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6834/2271414287,v1:172.21.15.153:6835/2271414287] up:boot 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.139:6834/2724174590,v1:172.21.15.139:6835/2724174590] up:boot 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.139:6836/102201551,v1:172.21.15.139:6838/102201551] up:boot 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.139:6837/947723603,v1:172.21.15.139:6839/947723603] up:boot 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6836/2394172155,v1:172.21.15.153:6838/2394172155] up:boot 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: mds.? [v2:172.21.15.153:6837/8260919,v1:172.21.15.153:6839/8260919] up:boot 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: fsmap 12 up:standby 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: from='mgr.15480 172.21.15.27:0/905485838' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:38:06.042 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:05 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/329042760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.633+0000 7ff04a7fc640 1 -- 172.21.15.27:0/3028877231 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v110) ==== 169+0+0 (secure 0 0 0) 0x7ff04c0bd250 con 0x7ff05410bd00 2024-04-25T01:38:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-04-25T01:38:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff0400a2d30 msgr2=0x7ff0400a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff0400a2d30 0x7ff0400a51f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff050009b40 tx=0x7ff050005e50 comp rx=0 tx=0).stop 2024-04-25T01:38:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 msgr2=0x7ff054101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 0x7ff054101090 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff04c006010 tx=0x7ff04c003ce0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 shutdown_connections 2024-04-25T01:38:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7ff0400a2d30 0x7ff0400a51f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05410bd00 0x7ff054101090 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0541032e0 0x7ff054100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 --2- 172.21.15.27:0/3028877231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0541029c0 0x7ff054100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.637+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 >> 172.21.15.27:0/3028877231 conn(0x7ff054077740 msgr2=0x7ff0540fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.638+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 shutdown_connections 2024-04-25T01:38:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.638+0000 7ff05b49e640 1 -- 172.21.15.27:0/3028877231 wait complete. 2024-04-25T01:38:06.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T01:38:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.763+0000 7f76b22df640 1 -- 172.21.15.27:0/2234777210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100fd0 msgr2=0x7f76ac10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.763+0000 7f76b22df640 1 --2- 172.21.15.27:0/2234777210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100fd0 0x7f76ac10d4e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009cd0 tx=0x7f76a402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 -- 172.21.15.27:0/2234777210 shutdown_connections 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 --2- 172.21.15.27:0/2234777210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100fd0 0x7f76ac10d4e0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 --2- 172.21.15.27:0/2234777210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac100610 0x7f76ac100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 --2- 172.21.15.27:0/2234777210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76ac1066b0 0x7f76ac106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 -- 172.21.15.27:0/2234777210 >> 172.21.15.27:0/2234777210 conn(0x7f76ac077740 msgr2=0x7f76ac077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 -- 172.21.15.27:0/2234777210 shutdown_connections 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 -- 172.21.15.27:0/2234777210 wait complete. 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 Processor -- start 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.764+0000 7f76b22df640 1 -- start start 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b22df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100610 0x7f76ac197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b22df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac100fd0 0x7f76ac198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b22df640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76ac1066b0 0x7f76ac19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b22df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f76ac110a30 con 0x7f76ac100610 2024-04-25T01:38:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b22df640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f76ac1108b0 con 0x7f76ac1066b0 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b22df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76ac110bb0 con 0x7f76ac100fd0 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100610 0x7f76ac197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100610 0x7f76ac197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57486/0 (socket says 172.21.15.27:57486) 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 -- 172.21.15.27:0/467491997 learned_addr learned my addr 172.21.15.27:0/467491997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76ab7fe640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac100fd0 0x7f76ac198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b087c640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76ac1066b0 0x7f76ac19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 -- 172.21.15.27:0/467491997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac100fd0 msgr2=0x7f76ac198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac100fd0 0x7f76ac198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 -- 172.21.15.27:0/467491997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76ac1066b0 msgr2=0x7f76ac19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76ac1066b0 0x7f76ac19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 -- 172.21.15.27:0/467491997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ac19cee0 con 0x7f76ac100610 2024-04-25T01:38:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76abfff640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100610 0x7f76ac197f40 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f769c00c970 tx=0x7f769c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76a97fa640 1 -- 172.21.15.27:0/467491997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c007c10 con 0x7f76ac100610 2024-04-25T01:38:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76a97fa640 1 -- 172.21.15.27:0/467491997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f769c007db0 con 0x7f76ac100610 2024-04-25T01:38:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76a97fa640 1 -- 172.21.15.27:0/467491997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c00f940 con 0x7f76ac100610 2024-04-25T01:38:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76ac19d1d0 con 0x7f76ac100610 2024-04-25T01:38:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.765+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ac1a4a60 con 0x7f76ac100610 2024-04-25T01:38:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.766+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76ac068560 con 0x7f76ac100610 2024-04-25T01:38:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.768+0000 7f76a97fa640 1 -- 172.21.15.27:0/467491997 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f769c00fae0 con 0x7f76ac100610 2024-04-25T01:38:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.768+0000 7f76a97fa640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f76940a2d80 0x7f76940a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.768+0000 7f76ab7fe640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f76940a2d80 0x7f76940a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.768+0000 7f76a97fa640 1 -- 172.21.15.27:0/467491997 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8670+0+0 (secure 0 0 0) 0x7f769c0cfa30 con 0x7f76ac100610 2024-04-25T01:38:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.769+0000 7f76ab7fe640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f76940a2d80 0x7f76940a5240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f76ac105d20 tx=0x7f76a000a250 comp rx=0 tx=0).ready entity=mgr.15480 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.772+0000 7f76a97fa640 1 -- 172.21.15.27:0/467491997 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f769c09c6e0 con 0x7f76ac100610 2024-04-25T01:38:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:06.959+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f76ac199240 con 0x7f76ac100610 2024-04-25T01:38:06.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:06 smithi027 ceph-mon[29197]: pgmap v397: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2024-04-25T01:38:06.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:06 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/329042760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:38:06.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:06 smithi027 ceph-mon[29197]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T01:38:06.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:06 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3028877231' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:07.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:06 smithi153 ceph-mon[32621]: pgmap v397: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2024-04-25T01:38:07.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:06 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/329042760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:38:07.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:06 smithi153 ceph-mon[32621]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T01:38:07.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:06 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3028877231' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:07.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:06 smithi139 ceph-mon[32760]: pgmap v397: 129 pgs: 129 active+clean; 5.7 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 67 KiB/s wr, 29 op/s 2024-04-25T01:38:07.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:06 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/329042760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:38:07.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:06 smithi139 ceph-mon[32760]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T01:38:07.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:06 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3028877231' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:07.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.639+0000 7f76a97fa640 1 -- 172.21.15.27:0/467491997 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v111) ==== 178+0+0 (secure 0 0 0) 0x7f769c09bfe0 con 0x7f76ac100610 2024-04-25T01:38:07.618 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data_ec' removed 2024-04-25T01:38:07.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f76940a2d80 msgr2=0x7f76940a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:07.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f76940a2d80 0x7f76940a5240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f76ac105d20 tx=0x7f76a000a250 comp rx=0 tx=0).stop 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100610 msgr2=0x7f76ac197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100610 0x7f76ac197f40 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f769c00c970 tx=0x7f769c00ce30 comp rx=0 tx=0).stop 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 shutdown_connections 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f76ac1066b0 0x7f76ac19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac100fd0 0x7f76ac198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:6800/1825554759,v1:172.21.15.27:6801/1825554759] conn(0x7f76940a2d80 0x7f76940a5240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 --2- 172.21.15.27:0/467491997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ac100610 0x7f76ac197f40 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 >> 172.21.15.27:0/467491997 conn(0x7f76ac077740 msgr2=0x7f76ac0ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 shutdown_connections 2024-04-25T01:38:07.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T01:38:07.643+0000 7f76b22df640 1 -- 172.21.15.27:0/467491997 wait complete. 2024-04-25T01:38:07.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:38:07.662 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:38:07.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:38:07.728 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T01:38:07.728 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:38:07.745 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:38:07.763 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:38:07.785 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T01:38:07.786 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T01:38:07.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a 2024-04-25T01:38:07.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:07 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/3028877231' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:38:07.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:07 smithi027 ceph-mon[29197]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T01:38:07.944 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:07 smithi027 ceph-mon[29197]: from='client.? 172.21.15.27:0/467491997' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:07.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:07 smithi027 systemd[1]: Stopping Ceph mon.a for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:08.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:07 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/3028877231' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:38:08.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:07 smithi153 ceph-mon[32621]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T01:38:08.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:07 smithi153 ceph-mon[32621]: from='client.? 172.21.15.27:0/467491997' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:08.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:07 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/3028877231' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:38:08.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:07 smithi139 ceph-mon[32760]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T01:38:08.041 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:07 smithi139 ceph-mon[32760]: from='client.? 172.21.15.27:0/467491997' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:38:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a[29193]: 2024-04-25T01:38:08.138+0000 7fa2e437f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:08.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a[29193]: 2024-04-25T01:38:08.138+0000 7fa2e437f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T01:38:08.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:08 smithi027 podman[99582]: 2024-04-25 01:38:08.302671651 +0000 UTC m=+0.303605899 container died be8d93f92e99d480d18cc736384ac0575dfbca7d47fc4efe4bd66caeb6398132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T01:38:08.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:08 smithi027 podman[99582]: 2024-04-25 01:38:08.722639348 +0000 UTC m=+0.723573593 container cleanup be8d93f92e99d480d18cc736384ac0575dfbca7d47fc4efe4bd66caeb6398132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T01:38:08.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:08 smithi027 bash[99582]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a 2024-04-25T01:38:09.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T01:38:09.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:08 smithi027 podman[99597]: 2024-04-25 01:38:08.95741924 +0000 UTC m=+0.651629713 container remove be8d93f92e99d480d18cc736384ac0575dfbca7d47fc4efe4bd66caeb6398132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T01:38:09.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:09 smithi027 systemd[1]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T01:38:09.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:09 smithi027 systemd[1]: Stopped Ceph mon.a for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:38:09.194 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 01:38:09 smithi027 systemd[1]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.a.service: Consumed 24.119s CPU time. 2024-04-25T01:38:09.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:09.242 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T01:38:09.242 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T01:38:09.242 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.b 2024-04-25T01:38:09.541 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:09 smithi139 systemd[1]: Stopping Ceph mon.b for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:09.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:09 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-b[32756]: 2024-04-25T01:38:09.560+0000 7f0f5ba24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:09.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:09 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-b[32756]: 2024-04-25T01:38:09.560+0000 7f0f5ba24700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T01:38:09.791 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:09 smithi139 podman[69489]: 2024-04-25 01:38:09.74555072 +0000 UTC m=+0.289038595 container died 4dc38a28c2a890fa97914af1b6b4eeddbecf67c0cec51c46506256b63bf1fe2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-b, ceph=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T01:38:10.291 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:10 smithi139 podman[69489]: 2024-04-25 01:38:10.12926019 +0000 UTC m=+0.672748064 container cleanup 4dc38a28c2a890fa97914af1b6b4eeddbecf67c0cec51c46506256b63bf1fe2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T01:38:10.291 INFO:journalctl@ceph.mon.b.smithi139.stdout:Apr 25 01:38:10 smithi139 bash[69489]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-b 2024-04-25T01:38:10.538 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T01:38:10.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:10.579 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T01:38:10.579 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T01:38:10.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.c 2024-04-25T01:38:10.862 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:10 smithi153 systemd[1]: Stopping Ceph mon.c for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:11.179 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:10 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-c[32617]: 2024-04-25T01:38:10.882+0000 7ff05391f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:11.179 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:10 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-c[32617]: 2024-04-25T01:38:10.882+0000 7ff05391f700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T01:38:11.180 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:10 smithi153 podman[69691]: 2024-04-25 01:38:10.903520312 +0000 UTC m=+0.106529102 container died 4c8282d0292ed28b7b19313bda77e22c9404acfcb02faf68f402193f4692236d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:38:11.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:11 smithi153 podman[69691]: 2024-04-25 01:38:11.283476479 +0000 UTC m=+0.486485256 container cleanup 4c8282d0292ed28b7b19313bda77e22c9404acfcb02faf68f402193f4692236d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T01:38:11.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 25 01:38:11 smithi153 bash[69691]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mon-c 2024-04-25T01:38:11.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T01:38:11.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:11.758 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T01:38:11.758 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T01:38:11.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.x 2024-04-25T01:38:12.109 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:38:11 smithi027 systemd[1]: Stopping Ceph mgr.x for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:12.399 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:38:12 smithi027 podman[99705]: 2024-04-25 01:38:12.133861608 +0000 UTC m=+0.151587955 container died 63362b3c482ce2f56f3b187a2c898c0d40173e6db67380bf294e0a24015b982b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9) 2024-04-25T01:38:12.694 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:38:12 smithi027 podman[99705]: 2024-04-25 01:38:12.495181055 +0000 UTC m=+0.512907397 container cleanup 63362b3c482ce2f56f3b187a2c898c0d40173e6db67380bf294e0a24015b982b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T01:38:12.695 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 01:38:12 smithi027 bash[99705]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-x 2024-04-25T01:38:12.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T01:38:12.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:12.990 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T01:38:12.990 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T01:38:12.990 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.y 2024-04-25T01:38:13.291 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:38:13 smithi139 systemd[1]: Stopping Ceph mgr.y for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:13.655 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:38:13 smithi139 podman[69613]: 2024-04-25 01:38:13.379781074 +0000 UTC m=+0.177646871 container died 12df09f15ded9efe8bc0638105350d97d2ce7e9c58abc3cff67197040a56c6ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T01:38:14.006 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:38:13 smithi139 podman[69613]: 2024-04-25 01:38:13.768171212 +0000 UTC m=+0.566037010 container cleanup 12df09f15ded9efe8bc0638105350d97d2ce7e9c58abc3cff67197040a56c6ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:38:14.006 INFO:journalctl@ceph.mgr.y.smithi139.stdout:Apr 25 01:38:13 smithi139 bash[69613]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-y 2024-04-25T01:38:14.172 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T01:38:14.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:14.205 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T01:38:14.205 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T01:38:14.205 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.z 2024-04-25T01:38:14.525 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:38:14 smithi153 systemd[1]: Stopping Ceph mgr.z for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:14.869 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:38:14 smithi153 podman[69815]: 2024-04-25 01:38:14.559557593 +0000 UTC m=+0.144503689 container died f231c531f574c160c41e68ee25cecf7ac1cb5cc7aca806a4a55823610ab95c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:38:15.170 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:38:14 smithi153 podman[69815]: 2024-04-25 01:38:14.989629011 +0000 UTC m=+0.574575107 container cleanup f231c531f574c160c41e68ee25cecf7ac1cb5cc7aca806a4a55823610ab95c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T01:38:15.170 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 25 01:38:14 smithi153 bash[69815]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mgr-z 2024-04-25T01:38:15.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T01:38:15.423 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:15.424 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T01:38:15.424 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T01:38:15.424 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.0 2024-04-25T01:38:15.746 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:15 smithi027 systemd[1]: Stopping Ceph osd.0 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:16.195 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0[43531]: 2024-04-25T01:38:15.765+0000 7f92d2621700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:16.195 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0[43531]: 2024-04-25T01:38:15.765+0000 7f92d2621700 -1 osd.0 111 *** Got signal Terminated *** 2024-04-25T01:38:16.195 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0[43531]: 2024-04-25T01:38:15.765+0000 7f92d2621700 -1 osd.0 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:38:21.356 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:21 smithi027 podman[99828]: 2024-04-25 01:38:21.090179121 +0000 UTC m=+5.447198200 container died 1775e1bbea119ffa3cde5bcc0469b9d5dc3f2f556c133f0b16545c2fd453fae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-25T01:38:21.695 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:21 smithi027 podman[99828]: 2024-04-25 01:38:21.48399691 +0000 UTC m=+5.841015974 container cleanup 1775e1bbea119ffa3cde5bcc0469b9d5dc3f2f556c133f0b16545c2fd453fae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:38:21.695 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:21 smithi027 bash[99828]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0 2024-04-25T01:38:22.049 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:21 smithi027 podman[99845]: 2024-04-25 01:38:21.727299506 +0000 UTC m=+0.638263484 container remove 1775e1bbea119ffa3cde5bcc0469b9d5dc3f2f556c133f0b16545c2fd453fae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T01:38:22.310 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:22 smithi027 podman[99937]: 2024-04-25 01:38:21.973681935 +0000 UTC m=+0.018369840 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:38:22.310 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:22 smithi027 podman[99937]: 2024-04-25 01:38:22.094933498 +0000 UTC m=+0.139621379 container create c93450e0a4b4fed3d9262a73a6837c6dfbbdcf8767d591101e836dfec3b1707e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9) 2024-04-25T01:38:22.310 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:22 smithi027 podman[99937]: 2024-04-25 01:38:22.33111409 +0000 UTC m=+0.375801969 container init c93450e0a4b4fed3d9262a73a6837c6dfbbdcf8767d591101e836dfec3b1707e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:38:22.619 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:22 smithi027 podman[99937]: 2024-04-25 01:38:22.335913539 +0000 UTC m=+0.380601418 container start c93450e0a4b4fed3d9262a73a6837c6dfbbdcf8767d591101e836dfec3b1707e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T01:38:22.619 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:22 smithi027 podman[99937]: 2024-04-25 01:38:22.451315864 +0000 UTC m=+0.496003759 container attach c93450e0a4b4fed3d9262a73a6837c6dfbbdcf8767d591101e836dfec3b1707e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T01:38:22.913 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 01:38:22 smithi027 podman[99937]: 2024-04-25 01:38:22.64043634 +0000 UTC m=+0.685124225 container died c93450e0a4b4fed3d9262a73a6837c6dfbbdcf8767d591101e836dfec3b1707e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T01:38:23.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T01:38:23.361 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:23.361 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T01:38:23.361 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T01:38:23.362 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.1 2024-04-25T01:38:23.660 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:23 smithi139 systemd[1]: Stopping Ceph osd.1 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:24.042 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1[37370]: 2024-04-25T01:38:23.680+0000 7ffbbf328700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:24.042 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1[37370]: 2024-04-25T01:38:23.680+0000 7ffbbf328700 -1 osd.1 111 *** Got signal Terminated *** 2024-04-25T01:38:24.042 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1[37370]: 2024-04-25T01:38:23.680+0000 7ffbbf328700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:38:29.278 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:29 smithi139 podman[69736]: 2024-04-25 01:38:29.002659954 +0000 UTC m=+5.431135021 container died dd8934475080d8c0bf678c359a53125f8c0b628be77e1f0b815ce467d676e401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:38:29.542 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:29 smithi139 podman[69736]: 2024-04-25 01:38:29.39753043 +0000 UTC m=+5.826005486 container cleanup dd8934475080d8c0bf678c359a53125f8c0b628be77e1f0b815ce467d676e401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T01:38:29.542 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:29 smithi139 bash[69736]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1 2024-04-25T01:38:29.990 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:29 smithi139 podman[69753]: 2024-04-25 01:38:29.657303786 +0000 UTC m=+0.645096281 container remove dd8934475080d8c0bf678c359a53125f8c0b628be77e1f0b815ce467d676e401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T01:38:30.292 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:30 smithi139 podman[69846]: 2024-04-25 01:38:29.913971569 +0000 UTC m=+0.018446818 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:38:30.293 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:30 smithi139 podman[69846]: 2024-04-25 01:38:30.041632732 +0000 UTC m=+0.146108009 container create 3115a387292b1d36d1ced1b2f9bc03ff6a9c22941e829efe30d29740a8302934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:38:30.293 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:30 smithi139 podman[69846]: 2024-04-25 01:38:30.235734671 +0000 UTC m=+0.340209902 container init 3115a387292b1d36d1ced1b2f9bc03ff6a9c22941e829efe30d29740a8302934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424) 2024-04-25T01:38:30.293 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:30 smithi139 podman[69846]: 2024-04-25 01:38:30.238843191 +0000 UTC m=+0.343318423 container start 3115a387292b1d36d1ced1b2f9bc03ff6a9c22941e829efe30d29740a8302934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T01:38:30.792 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:30 smithi139 podman[69846]: 2024-04-25 01:38:30.319068367 +0000 UTC m=+0.423543611 container attach 3115a387292b1d36d1ced1b2f9bc03ff6a9c22941e829efe30d29740a8302934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T01:38:30.792 INFO:journalctl@ceph.osd.1.smithi139.stdout:Apr 25 01:38:30 smithi139 podman[69846]: 2024-04-25 01:38:30.521940714 +0000 UTC m=+0.626415946 container died 3115a387292b1d36d1ced1b2f9bc03ff6a9c22941e829efe30d29740a8302934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T01:38:31.075 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T01:38:31.114 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:31.114 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T01:38:31.114 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T01:38:31.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.2 2024-04-25T01:38:31.415 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:31 smithi153 systemd[1]: Stopping Ceph osd.2 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:31.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:31 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2[37762]: 2024-04-25T01:38:31.434+0000 7fa370a21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:31.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:31 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2[37762]: 2024-04-25T01:38:31.434+0000 7fa370a21700 -1 osd.2 111 *** Got signal Terminated *** 2024-04-25T01:38:31.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:31 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2[37762]: 2024-04-25T01:38:31.434+0000 7fa370a21700 -1 osd.2 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:38:37.250 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:36 smithi153 podman[69939]: 2024-04-25 01:38:36.940713512 +0000 UTC m=+5.618538103 container stop bbf301f64991d12e6fbe4f687954d22d02d3ad9c34247d9460da145cd8ae57b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:38:37.250 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:36 smithi153 podman[69939]: 2024-04-25 01:38:36.966306001 +0000 UTC m=+5.644130589 container died bbf301f64991d12e6fbe4f687954d22d02d3ad9c34247d9460da145cd8ae57b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T01:38:37.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:37 smithi153 podman[69939]: 2024-04-25 01:38:37.385202339 +0000 UTC m=+6.063026927 container cleanup bbf301f64991d12e6fbe4f687954d22d02d3ad9c34247d9460da145cd8ae57b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:38:37.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:37 smithi153 bash[69939]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2 2024-04-25T01:38:37.992 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:37 smithi153 podman[69968]: 2024-04-25 01:38:37.586879919 +0000 UTC m=+0.638354165 container remove bbf301f64991d12e6fbe4f687954d22d02d3ad9c34247d9460da145cd8ae57b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T01:38:38.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:38 smithi153 podman[70058]: 2024-04-25 01:38:37.915799749 +0000 UTC m=+0.018100730 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:38:38.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:38 smithi153 podman[70058]: 2024-04-25 01:38:38.029539391 +0000 UTC m=+0.131840376 container create 27f0823f1eaa05e3d1cb1799daeac61b6e5dbdd3ea54fdbf5c5b1a14112039e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T01:38:38.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:38 smithi153 podman[70058]: 2024-04-25 01:38:38.28313333 +0000 UTC m=+0.385434323 container init 27f0823f1eaa05e3d1cb1799daeac61b6e5dbdd3ea54fdbf5c5b1a14112039e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T01:38:38.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:38 smithi153 podman[70058]: 2024-04-25 01:38:38.286849581 +0000 UTC m=+0.389150573 container start 27f0823f1eaa05e3d1cb1799daeac61b6e5dbdd3ea54fdbf5c5b1a14112039e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:38:38.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:38 smithi153 podman[70058]: 2024-04-25 01:38:38.382897464 +0000 UTC m=+0.485198473 container attach 27f0823f1eaa05e3d1cb1799daeac61b6e5dbdd3ea54fdbf5c5b1a14112039e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T01:38:38.816 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 25 01:38:38 smithi153 podman[70058]: 2024-04-25 01:38:38.574076984 +0000 UTC m=+0.676377966 container died 27f0823f1eaa05e3d1cb1799daeac61b6e5dbdd3ea54fdbf5c5b1a14112039e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T01:38:39.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T01:38:39.164 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:39.165 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T01:38:39.165 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T01:38:39.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.3 2024-04-25T01:38:39.446 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:39 smithi027 systemd[1]: Stopping Ceph osd.3 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:39.946 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:39 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3[50502]: 2024-04-25T01:38:39.505+0000 7fa2b4c1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:39.946 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:39 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3[50502]: 2024-04-25T01:38:39.505+0000 7fa2b4c1f700 -1 osd.3 111 *** Got signal Terminated *** 2024-04-25T01:38:39.946 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:39 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3[50502]: 2024-04-25T01:38:39.505+0000 7fa2b4c1f700 -1 osd.3 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:38:43.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:42 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:42.726+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:43.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:43.443+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:44.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:43 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:43.743+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:44.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:43.895+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:44.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:44.399+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:45.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:44 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:44.793+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:45.196 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:44 smithi027 podman[100043]: 2024-04-25 01:38:44.842157254 +0000 UTC m=+5.452833497 container died a61c45ad8b6986800fec6094842d9a641e001dd3aa868ccd6bf5307c711dbb07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T01:38:45.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:44.875+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:45.595 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:45 smithi027 podman[100043]: 2024-04-25 01:38:45.355223077 +0000 UTC m=+5.965899312 container cleanup a61c45ad8b6986800fec6094842d9a641e001dd3aa868ccd6bf5307c711dbb07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T01:38:45.595 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:45 smithi027 bash[100043]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3 2024-04-25T01:38:45.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:45.406+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:45.946 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:45 smithi027 podman[100063]: 2024-04-25 01:38:45.615060018 +0000 UTC m=+0.765917297 container remove a61c45ad8b6986800fec6094842d9a641e001dd3aa868ccd6bf5307c711dbb07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:38:46.276 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:45.890+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:46.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:45.867+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:46.292 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:45 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:45.819+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:46.374 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:46 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:46.073+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:46.374 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:46 smithi027 podman[100152]: 2024-04-25 01:38:45.944171358 +0000 UTC m=+0.018247718 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:38:46.375 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:46 smithi027 podman[100152]: 2024-04-25 01:38:46.174653963 +0000 UTC m=+0.248730304 container create 2ebda3f3654e13b64f68958c0d32dfcf125e36bfa90f0dc8203e1011b0a1038f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T01:38:46.375 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:46 smithi027 podman[100152]: 2024-04-25 01:38:46.394134341 +0000 UTC m=+0.468210690 container init 2ebda3f3654e13b64f68958c0d32dfcf125e36bfa90f0dc8203e1011b0a1038f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T01:38:46.677 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:46 smithi027 podman[100152]: 2024-04-25 01:38:46.397836058 +0000 UTC m=+0.471912412 container start 2ebda3f3654e13b64f68958c0d32dfcf125e36bfa90f0dc8203e1011b0a1038f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T01:38:46.677 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:46 smithi027 podman[100152]: 2024-04-25 01:38:46.502244631 +0000 UTC m=+0.576320987 container attach 2ebda3f3654e13b64f68958c0d32dfcf125e36bfa90f0dc8203e1011b0a1038f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T01:38:46.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:46.453+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:46.946 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 01:38:46 smithi027 podman[100152]: 2024-04-25 01:38:46.69769826 +0000 UTC m=+0.771774604 container died 2ebda3f3654e13b64f68958c0d32dfcf125e36bfa90f0dc8203e1011b0a1038f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:38:47.276 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:46.846+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:47.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:46.922+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:47.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:46 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:46.867+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:47.336 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:47 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:47.087+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:47.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T01:38:47.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:47.406 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T01:38:47.406 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T01:38:47.406 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.4 2024-04-25T01:38:47.722 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:47 smithi139 systemd[1]: Stopping Ceph osd.4 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:47.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:47.444+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:48.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:47 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:47.888+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:48.043 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:47 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4[43854]: 2024-04-25T01:38:47.741+0000 7f9da5619700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:48.043 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:47 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4[43854]: 2024-04-25T01:38:47.741+0000 7f9da5619700 -1 osd.4 111 *** Got signal Terminated *** 2024-04-25T01:38:48.043 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:47 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4[43854]: 2024-04-25T01:38:47.741+0000 7f9da5619700 -1 osd.4 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:38:48.276 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:47.887+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:48.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:47.917+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:48.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:48 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:48.075+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:48.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:48 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:48.487+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:49.276 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:48 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:48.860+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:49.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:48 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:48.901+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:49.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:48 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:48.837+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:49.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:49 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:49.056+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:49.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:49 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:49.520+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:50.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:49 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:49.808+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:50.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:49 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:49.808+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:50.276 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:49 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:49.855+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:50.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:49 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:49.921+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:50.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:49 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:49.921+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:50.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:50 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:50.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:50.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:50 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:50.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:50.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:50 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:50.204+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:50.776 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:50 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:50.539+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:51.138 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:50 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:50.828+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:51.138 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:50 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:50.828+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:51.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:50 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:50.896+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:51.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:50 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:50.904+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:51.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:50 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:50.904+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:51.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:51 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:51.021+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:51.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:51 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:51.021+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:51.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:51 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:51.156+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:51.920 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:51 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:51.583+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:51.920 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:51 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:51.583+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:38:52.135 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:51 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:51.833+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:52.136 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:51 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:51.833+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:52.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:51 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:51.939+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:52.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:51 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:51.954+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:52.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:51 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:51.954+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:52.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:51 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:51.977+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:52.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:51 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:51.977+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:52.543 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:52 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4[43854]: 2024-04-25T01:38:52.154+0000 7f9d9dc05700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.644786+0000 front 2024-04-25T01:38:26.644846+0000 (oldest deadline 2024-04-25T01:38:51.344999+0000) 2024-04-25T01:38:52.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:52 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:52.197+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:52.903 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:52 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:52.619+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:52.903 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:52 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:52.619+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:38:53.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:52 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:52.794+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:38:53.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:52 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:52.984+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:53.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:52 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:52.984+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:53.222 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:52 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:52.878+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:53.223 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:52 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:52.878+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:53.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:52 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:52.949+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:53.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:52 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:52.922+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:53.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:52 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:52.922+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:53.540 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:53 smithi139 podman[69965]: 2024-04-25 01:38:53.257044225 +0000 UTC m=+5.631558889 container died 645b44028640868929b0994257677b9241e1650a2fb050ca998adb97db0fc69c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4, ceph=True, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T01:38:53.540 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:53 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:53.240+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:53.793 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:53 smithi139 podman[69965]: 2024-04-25 01:38:53.658294078 +0000 UTC m=+6.032808751 container cleanup 645b44028640868929b0994257677b9241e1650a2fb050ca998adb97db0fc69c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:38:53.793 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:53 smithi139 bash[69965]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4 2024-04-25T01:38:53.908 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:53 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:53.612+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:53.908 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:53 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:53.612+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:38:54.196 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:53 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:53.776+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:38:54.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:54 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:54.000+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:54.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:54 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:54.000+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:54.200 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:53 smithi139 podman[69983]: 2024-04-25 01:38:53.926409955 +0000 UTC m=+0.668155562 container remove 645b44028640868929b0994257677b9241e1650a2fb050ca998adb97db0fc69c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424) 2024-04-25T01:38:54.200 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:53 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:53.837+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:54.200 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:53 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:53.837+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:54.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:53 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:53.927+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:54.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:53 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:53.947+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:54.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:53 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:53.947+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:54.468 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:54 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:54.218+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:54.468 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:54 smithi139 podman[70073]: 2024-04-25 01:38:54.182148549 +0000 UTC m=+0.018054779 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:38:54.468 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:54 smithi139 podman[70073]: 2024-04-25 01:38:54.285633115 +0000 UTC m=+0.121539325 container create b8063bebd9ee87a496ba4564f6147cfd3c31fc146110e4411b6ebfcc8d42339d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T01:38:54.758 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:54 smithi139 podman[70073]: 2024-04-25 01:38:54.488022225 +0000 UTC m=+0.323928437 container init b8063bebd9ee87a496ba4564f6147cfd3c31fc146110e4411b6ebfcc8d42339d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T01:38:54.758 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:54 smithi139 podman[70073]: 2024-04-25 01:38:54.491600259 +0000 UTC m=+0.327506475 container start b8063bebd9ee87a496ba4564f6147cfd3c31fc146110e4411b6ebfcc8d42339d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T01:38:54.758 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:54 smithi139 podman[70073]: 2024-04-25 01:38:54.563091914 +0000 UTC m=+0.398998143 container attach b8063bebd9ee87a496ba4564f6147cfd3c31fc146110e4411b6ebfcc8d42339d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T01:38:54.863 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:54 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:54.589+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:54.864 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:54 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:54.589+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:38:54.993 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:54 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:54.741+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:38:55.043 INFO:journalctl@ceph.osd.4.smithi139.stdout:Apr 25 01:38:54 smithi139 podman[70073]: 2024-04-25 01:38:54.778170066 +0000 UTC m=+0.614076277 container died b8063bebd9ee87a496ba4564f6147cfd3c31fc146110e4411b6ebfcc8d42339d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T01:38:55.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:54 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:54.808+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:55.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:54 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:54.808+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:55.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:54 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:54.883+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:55.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:54 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:54.960+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:55.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:54 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:54.960+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:55.339 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:55 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:55.261+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:55.359 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T01:38:55.400 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:38:55.400 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T01:38:55.400 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T01:38:55.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.5 2024-04-25T01:38:55.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:55 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:55.011+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:55.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:55 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:55.011+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:55.712 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:55 smithi153 systemd[1]: Stopping Ceph osd.5 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:38:55.712 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:55 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:55.627+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:55.712 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:55 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:55.627+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:38:55.712 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:55 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:55.731+0000 7fdab01c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:38:55.712 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:55 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:55.731+0000 7fdab01c5700 -1 osd.5 111 *** Got signal Terminated *** 2024-04-25T01:38:55.712 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:55 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:55.731+0000 7fdab01c5700 -1 osd.5 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:38:56.027 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:55 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:55.843+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:56.027 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:55 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:55.911+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:56.027 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:55 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:55.911+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:56.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:55 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:55.759+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:56.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:55 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:55.759+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:56.196 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:55 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:55.780+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:38:56.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:56 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:56.017+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:56.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:56 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:56.017+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:56.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:56 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:56.267+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:56.872 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:56 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:56.640+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:56.872 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:56 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:56.640+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:38:57.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:56 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:56.801+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:57.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:56 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:56.801+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:57.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:56 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:56.791+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:38:57.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:57 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:57.031+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:57.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:57 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:57.031+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:57.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:56 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:56.891+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:57.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:56 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:56.946+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:57.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:56 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:56.946+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:57.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:57 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:57.268+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:57.895 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:57 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:57.884+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:57.895 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:57 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:57.642+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:57.895 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:57 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:57.642+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:38:58.018 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:57 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:57.784+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:38:58.215 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:57 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:57.824+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:58.216 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:57 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:57.824+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:58.216 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:57 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:57.824+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:38:58.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:57 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:57.913+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:58.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:57 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:57.913+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:58.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:57 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:57.913+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:38:58.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:58.037+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:58.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:58.037+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:58.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:58 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:58.234+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:58.847 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:58.604+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:58.847 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:58.604+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:38:58.946 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:58.804+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:38:59.239 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:58 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:58.871+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:38:59.239 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:58 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:58.871+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:38:59.239 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:58 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:58.871+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:38:59.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:58.866+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:38:59.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:58.921+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:38:59.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:58.921+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:38:59.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:58 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:58.921+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:38:59.446 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:59 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:59.002+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:38:59.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:59 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:59.002+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:38:59.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:38:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:38:59.258+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:38:59.858 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:59.621+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:38:59.859 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:38:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:38:59.621+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:39:00.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:59 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:59.772+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:00.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:38:59 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:38:59.772+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:00.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:59 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:59.962+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:00.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:38:59 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:38:59.962+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:00.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:59.877+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:00.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:38:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:38:59.877+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:00.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:59.908+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:00.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:59.908+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:00.277 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:38:59 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:38:59.908+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:00.288 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:59.847+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:00.288 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:59.847+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:00.288 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:38:59 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:38:59.847+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:00.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:00 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:00.306+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:00.544 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:00 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:00.306+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:00.865 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:39:00.621+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:18.698992+0000 front 2024-04-25T01:38:18.698756+0000 (oldest deadline 2024-04-25T01:38:43.398738+0000) 2024-04-25T01:39:00.866 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5[44259]: 2024-04-25T01:39:00.621+0000 7fdaa87b1700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.400658+0000 front 2024-04-25T01:38:27.400712+0000 (oldest deadline 2024-04-25T01:38:51.500550+0000) 2024-04-25T01:39:01.176 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:00.924+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:01.176 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:00.924+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:01.176 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:00.885+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:01.176 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:00.885+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:01.177 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:00 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:00.885+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:01.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:00.779+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:01.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:00.779+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:01.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:00.928+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:01.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:00.928+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:01.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:00 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:00.928+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:01.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:00 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:00.827+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:01.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:00 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:00.827+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:01.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:00 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:00.827+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:01.479 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:01 smithi153 podman[70165]: 2024-04-25 01:39:01.195004947 +0000 UTC m=+5.583986433 container died 6e3c7881334b6b6f38e0e184991f96201bdd95ce99dd4846a885348e7c6c4c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-25T01:39:01.777 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:01 smithi153 podman[70165]: 2024-04-25 01:39:01.597898413 +0000 UTC m=+5.986879900 container cleanup 6e3c7881334b6b6f38e0e184991f96201bdd95ce99dd4846a885348e7c6c4c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:39:01.777 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:01 smithi153 bash[70165]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5 2024-04-25T01:39:01.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:01.328+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:01.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:01.328+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:02.177 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:01 smithi153 podman[70182]: 2024-04-25 01:39:01.841037876 +0000 UTC m=+0.647899042 container remove 6e3c7881334b6b6f38e0e184991f96201bdd95ce99dd4846a885348e7c6c4c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T01:39:02.177 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:01.941+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:02.177 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:01.941+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:02.178 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:01.841+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:02.178 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:01.841+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:02.178 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:01 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:01.841+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:02.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:01 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:01.770+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:02.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:01 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:01.770+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:02.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:01 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:01.953+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:02.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:01 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:01.953+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:02.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:01 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:01.953+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:02.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:01.832+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:02.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:01.832+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:02.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:01 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:01.832+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:02.450 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:02 smithi153 podman[70279]: 2024-04-25 01:39:02.099893929 +0000 UTC m=+0.019243352 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:02.450 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:02 smithi153 podman[70279]: 2024-04-25 01:39:02.241928619 +0000 UTC m=+0.161278042 container create 2360c4d2c9008997817698626bb89362735634c97e6ac3b111baa7656a843f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T01:39:02.450 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:02 smithi153 podman[70279]: 2024-04-25 01:39:02.469740575 +0000 UTC m=+0.389089999 container init 2360c4d2c9008997817698626bb89362735634c97e6ac3b111baa7656a843f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T01:39:02.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:02.311+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:02.543 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:02.311+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:02.755 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:02 smithi153 podman[70279]: 2024-04-25 01:39:02.472892508 +0000 UTC m=+0.392241937 container start 2360c4d2c9008997817698626bb89362735634c97e6ac3b111baa7656a843f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T01:39:02.755 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:02 smithi153 podman[70279]: 2024-04-25 01:39:02.569872226 +0000 UTC m=+0.489221665 container attach 2360c4d2c9008997817698626bb89362735634c97e6ac3b111baa7656a843f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:39:03.027 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 25 01:39:02 smithi153 podman[70279]: 2024-04-25 01:39:02.774710708 +0000 UTC m=+0.694060135 container died 2360c4d2c9008997817698626bb89362735634c97e6ac3b111baa7656a843f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T01:39:03.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:02 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:02.911+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:03.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:02 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:02.911+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:03.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:02 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:02.827+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:03.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:02 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:02.827+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:03.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:02 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:02.827+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:03.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:02 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:02.726+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:03.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:02 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:02.726+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:03.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:02 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:02.955+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:03.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:02 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:02.955+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:03.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:02 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:02.955+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:03.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:02.828+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:03.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:02.828+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:03.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:02 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:02.828+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:03.394 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T01:39:03.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:03.434 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T01:39:03.434 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T01:39:03.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.6 2024-04-25T01:39:03.757 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:03 smithi027 systemd[1]: Stopping Ceph osd.6 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:03.757 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:03.715+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:03.757 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:03.715+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:03.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:03.334+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:03.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:03.334+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:04.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:03.775+0000 7fbb10e7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:04.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:03.775+0000 7fbb10e7f700 -1 osd.6 111 *** Got signal Terminated *** 2024-04-25T01:39:04.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:03.775+0000 7fbb10e7f700 -1 osd.6 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:39:04.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:03.996+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:04.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:03.996+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:04.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:03.996+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:04.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:03.902+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:04.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:03.902+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:04.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:03.859+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:04.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:03.859+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:04.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:03 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:03.859+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:04.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:03.860+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:04.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:03.860+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:04.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:03 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:03.860+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:04.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:04.368+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:04.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:04.368+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:04.947 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:04 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:04.704+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:04.947 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:04 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:04.704+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:05.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:04 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:04.895+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:05.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:04 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:04.895+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:05.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:04 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:04.827+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:05.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:04 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:04.827+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:05.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:04 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:04.827+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:05.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:04.882+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:05.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:04.882+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:05.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:04 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:04.882+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:05.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:04 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:04.998+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:05.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:04 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:04.998+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:05.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:04 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:04.998+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:05.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:05 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:05.398+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:05.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:05 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:05.398+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:05.947 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:05 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:05.713+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:05.947 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:05 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:05.713+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:06.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:05 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:05.889+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:06.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:05 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:05.889+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:06.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:05 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:05.889+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:06.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:05 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:05.848+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:06.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:05 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:05.848+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:06.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:05 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:05.848+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:06.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:05 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:05.870+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:06.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:05 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:05.870+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:06.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:05 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:05.870+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:06.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:06 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:06.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:06.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:06 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:06.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:06.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:06 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:06.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:06.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:06 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:06.446+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:06.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:06 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:06.446+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:07.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:06 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:06.753+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:07.197 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:06 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:06.753+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:07.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:06 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:06.984+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:07.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:06 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:06.984+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:07.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:06 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:06.984+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:07.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:06 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:06.904+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:07.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:06 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:06.904+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:07.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:06 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:06.904+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:07.279 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:06 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:06.845+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:07.279 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:06 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:06.845+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:07.279 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:06 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:06.845+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:07.293 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:06 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:06.836+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:07.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:06 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:06.836+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:07.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:06 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:06.836+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:07.772 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:07 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:07.480+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:07.772 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:07 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:07.480+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:07.997 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:07 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:07.746+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:07.997 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:07 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:07.746+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:08.043 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:07 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:07.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:08.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:07 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:07.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:08.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:07 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:07.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:08.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:07 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:07.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:08.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:07 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:07.907+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:08.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:07 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:07.907+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:08.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:07 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:07.907+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:08.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:07 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:07.873+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:08.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:07 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:07.873+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:08.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:07 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:07.873+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:08.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:08.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:08.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:08.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:08.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:08.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:08.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:08.016+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:08.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:08 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:08.525+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:08.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:08 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:08.525+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:09.034 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:08.741+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:27.932181+0000 front 2024-04-25T01:38:27.932162+0000 (oldest deadline 2024-04-25T01:38:52.632156+0000) 2024-04-25T01:39:09.034 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:08.741+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.234257+0000 front 2024-04-25T01:38:34.234085+0000 (oldest deadline 2024-04-25T01:38:59.534194+0000) 2024-04-25T01:39:09.034 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:08 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6[57329]: 2024-04-25T01:39:08.741+0000 7fbb0d473700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.535768+0000 front 2024-04-25T01:38:43.535725+0000 (oldest deadline 2024-04-25T01:39:08.235749+0000) 2024-04-25T01:39:09.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:08 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:08.867+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:09.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:08 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:08.867+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:09.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:08 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:08.867+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:09.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:08 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:08.855+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:09.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:08 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:08.855+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:09.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:08 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:08.855+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:09.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:08 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:08.826+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:09.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:08 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:08.826+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:09.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:08 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:08.826+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:09.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:08 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:08.826+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:09.447 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:09 smithi027 podman[100268]: 2024-04-25 01:39:09.207424029 +0000 UTC m=+5.546805977 container died d8c06dd9b0ae22fb6c5129ad0efd151d03167a99349c971c4ed0913e8b04cbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T01:39:09.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:09 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:09.052+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:09.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:09 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:09.052+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:09.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:09 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:09.052+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:09.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:09 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:09.052+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:09.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:09 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:09.496+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:09.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:09 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:09.496+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:09.947 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:09 smithi027 podman[100268]: 2024-04-25 01:39:09.717980704 +0000 UTC m=+6.057362646 container cleanup d8c06dd9b0ae22fb6c5129ad0efd151d03167a99349c971c4ed0913e8b04cbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T01:39:09.947 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:09 smithi027 bash[100268]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6 2024-04-25T01:39:10.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:09 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:09.876+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:10.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:09 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:09.876+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:10.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:09 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:09.876+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:10.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:09 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:09.865+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:10.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:09 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:09.865+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:10.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:09 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:09.865+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:10.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:09 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:09.833+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:10.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:09 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:09.833+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:10.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:09 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:09.833+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:10.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:09 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:09.833+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:10.322 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:09 smithi027 podman[100285]: 2024-04-25 01:39:09.977682645 +0000 UTC m=+0.761786150 container remove d8c06dd9b0ae22fb6c5129ad0efd151d03167a99349c971c4ed0913e8b04cbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T01:39:10.323 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:10.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:10.323 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:10.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:10.323 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:10.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:10.323 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:10.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:10.697 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:10 smithi027 podman[100375]: 2024-04-25 01:39:10.245250991 +0000 UTC m=+0.018768841 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:10.697 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:10 smithi027 podman[100375]: 2024-04-25 01:39:10.358086947 +0000 UTC m=+0.131604846 container create 092c1b858a1d3b40609ecc4b26d9552462d92d0850a97bceaae3be3ae466e4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:39:10.697 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:10 smithi027 podman[100375]: 2024-04-25 01:39:10.581281588 +0000 UTC m=+0.354799418 container init 092c1b858a1d3b40609ecc4b26d9552462d92d0850a97bceaae3be3ae466e4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:39:10.697 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:10 smithi027 podman[100375]: 2024-04-25 01:39:10.584942474 +0000 UTC m=+0.358460307 container start 092c1b858a1d3b40609ecc4b26d9552462d92d0850a97bceaae3be3ae466e4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:39:10.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:10 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:10.513+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:10.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:10 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:10.514+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:10.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:10 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:10.514+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:44.079783+0000 front 2024-04-25T01:38:44.079816+0000 (oldest deadline 2024-04-25T01:39:09.979755+0000) 2024-04-25T01:39:11.148 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:10 smithi027 podman[100375]: 2024-04-25 01:39:10.789568493 +0000 UTC m=+0.563086325 container attach 092c1b858a1d3b40609ecc4b26d9552462d92d0850a97bceaae3be3ae466e4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T01:39:11.149 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 01:39:10 smithi027 podman[100375]: 2024-04-25 01:39:10.886550181 +0000 UTC m=+0.660068014 container died 092c1b858a1d3b40609ecc4b26d9552462d92d0850a97bceaae3be3ae466e4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_CLEAN=True) 2024-04-25T01:39:11.149 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:10.986+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:11.149 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:10.986+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:11.149 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:10.986+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:11.149 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:10 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:10.986+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:11.277 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:10 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:10.902+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:11.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:10 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:10.902+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:11.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:10 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:10.902+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:11.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:10 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:10.874+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:11.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:10 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:10.874+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:11.278 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:10 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:10.874+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:11.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:10 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:10.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:11.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:10 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:10.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:11.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:10 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:10.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:11.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:10 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:10.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:11.478 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T01:39:11.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:11.517 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T01:39:11.517 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T01:39:11.517 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.7 2024-04-25T01:39:11.792 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:11.538+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:11.792 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:11.538+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:11.792 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:11.538+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:44.079783+0000 front 2024-04-25T01:38:44.079816+0000 (oldest deadline 2024-04-25T01:39:09.979755+0000) 2024-04-25T01:39:11.792 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:11 smithi139 systemd[1]: Stopping Ceph osd.7 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:12.044 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:11.812+0000 7f2fae11f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:12.044 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:11.812+0000 7f2fae11f700 -1 osd.7 111 *** Got signal Terminated *** 2024-04-25T01:39:12.044 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:11.812+0000 7f2fae11f700 -1 osd.7 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:39:12.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:11.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:12.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:11.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:12.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:11.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:12.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:11 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:11.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:12.163 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:11 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:11.906+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:12.163 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:11 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:11.906+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:12.164 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:11 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:11.906+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:12.164 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:11 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:11.856+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:12.164 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:11 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:11.856+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:12.164 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:11 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:11.856+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:12.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:11.998+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:12.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:11.998+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:12.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:11.998+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:12.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:11 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:11.998+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:12.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:12.554+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:12.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:12.554+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:12.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:12.554+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:44.079783+0000 front 2024-04-25T01:38:44.079816+0000 (oldest deadline 2024-04-25T01:39:09.979755+0000) 2024-04-25T01:39:12.795 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:12.782+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:12.795 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:12.782+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:12.795 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:12.782+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:12.795 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:12.782+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:13.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:12 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:12.943+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:13.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:12 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:12.943+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:13.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:12 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:12.943+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:13.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:12 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:12.807+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:13.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:12 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:12.807+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:13.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:12 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:12.807+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:13.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:12 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:12.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:13.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:12 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:12.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:13.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:12 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:12.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:13.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:12 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:12.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:13.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:13 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:13.505+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:13.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:13 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:13.505+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:13.793 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:13 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:13.505+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:44.079783+0000 front 2024-04-25T01:38:44.079816+0000 (oldest deadline 2024-04-25T01:39:09.979755+0000) 2024-04-25T01:39:14.027 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:13 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:13.964+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:14.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:13 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:13.964+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:14.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:13 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:13.964+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:14.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:13 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:13.785+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:14.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:13 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:13.785+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:14.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:13 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:13.785+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:14.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:13 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:13.811+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:14.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:13 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:13.811+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:14.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:13 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:13.811+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:14.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:13 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:13.811+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:14.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:13.996+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:14.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:13.996+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:14.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:13.996+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:14.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:13 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:13.996+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:14.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:14 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:14.538+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:14.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:14 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:14.538+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:14.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:14 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:14.538+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:44.079783+0000 front 2024-04-25T01:38:44.079816+0000 (oldest deadline 2024-04-25T01:39:09.979755+0000) 2024-04-25T01:39:14.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:14 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:14.538+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.581119+0000 front 2024-04-25T01:38:51.581196+0000 (oldest deadline 2024-04-25T01:39:13.881262+0000) 2024-04-25T01:39:15.027 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:15.003+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:15.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:15.003+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:15.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:15.003+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:15.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:14 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:14.775+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:15.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:14 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:14.775+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:15.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:14 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:14.775+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:15.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:14 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:14.775+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:15.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:14 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:14.859+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:15.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:14 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:14.859+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:15.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:14 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:14.859+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:15.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:14 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:14.859+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:15.348 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:15.015+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:15.349 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:15.015+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:15.349 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:15.015+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:15.349 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:15 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:15.015+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:15.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:15 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:15.516+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:15.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:15 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:15.516+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:15.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:15 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:15.516+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:44.079783+0000 front 2024-04-25T01:38:44.079816+0000 (oldest deadline 2024-04-25T01:39:09.979755+0000) 2024-04-25T01:39:15.795 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:15 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:15.516+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.581119+0000 front 2024-04-25T01:38:51.581196+0000 (oldest deadline 2024-04-25T01:39:13.881262+0000) 2024-04-25T01:39:16.027 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:15.962+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:16.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:15.962+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:16.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:15.962+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:16.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:15.962+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:16.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:15.750+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:16.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:15.750+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:16.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:15.750+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:16.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:15 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:15.750+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:16.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:15 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:15.842+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:16.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:15 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:15.842+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:16.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:15 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:15.842+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:16.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:15 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:15.842+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:16.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:16.028+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:16.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:16.028+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:16.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:16.028+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:16.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:16.028+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:16.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:16.509+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.476455+0000 front 2024-04-25T01:38:25.476735+0000 (oldest deadline 2024-04-25T01:38:49.576679+0000) 2024-04-25T01:39:16.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:16.509+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.778261+0000 front 2024-04-25T01:38:34.778271+0000 (oldest deadline 2024-04-25T01:38:59.478235+0000) 2024-04-25T01:39:16.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:16.509+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:44.079783+0000 front 2024-04-25T01:38:44.079816+0000 (oldest deadline 2024-04-25T01:39:09.979755+0000) 2024-04-25T01:39:16.794 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7[50840]: 2024-04-25T01:39:16.509+0000 7f2faa713700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.581119+0000 front 2024-04-25T01:38:51.581196+0000 (oldest deadline 2024-04-25T01:39:13.881262+0000) 2024-04-25T01:39:17.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:16 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:16.938+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:17.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:16 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:16.938+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:17.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:16 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:16.938+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:17.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:16 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:16.938+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:17.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:16 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:16.741+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:17.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:16 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:16.741+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:17.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:16 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:16.741+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:17.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:16 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:16.741+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:17.135 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:16.879+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:17.136 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:16.879+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:17.136 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:16.879+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:17.136 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:16.879+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:17.136 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:16.879+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:17.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:16.980+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:17.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:16.980+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:17.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:16.980+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:17.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:16 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:16.980+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:17.468 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:17 smithi139 podman[70194]: 2024-04-25 01:39:17.153892327 +0000 UTC m=+5.435862995 container died f44a9f64f6a34a3db77664be002a803852640e4a82fc96512b2932590f2d0d6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T01:39:17.468 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:17 smithi139 podman[70194]: 2024-04-25 01:39:17.487261857 +0000 UTC m=+5.769232531 container cleanup f44a9f64f6a34a3db77664be002a803852640e4a82fc96512b2932590f2d0d6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:39:17.720 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:17 smithi139 bash[70194]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7 2024-04-25T01:39:18.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:17.954+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:18.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:17.954+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:18.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:17.954+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:18.028 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:17.954+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:18.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:17.719+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:18.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:17.719+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:18.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:17.719+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:18.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:17 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:17.719+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:18.044 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:17 smithi139 podman[70211]: 2024-04-25 01:39:17.738577603 +0000 UTC m=+0.578027715 container remove f44a9f64f6a34a3db77664be002a803852640e4a82fc96512b2932590f2d0d6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:39:18.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:17.888+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:18.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:17.888+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:18.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:17.888+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:18.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:17.888+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:18.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:17 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:17.888+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:18.417 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:18 smithi139 podman[70304]: 2024-04-25 01:39:18.068258573 +0000 UTC m=+0.018208340 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:18.417 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:18 smithi139 podman[70304]: 2024-04-25 01:39:18.181130236 +0000 UTC m=+0.131080036 container create a9c8fece0da8856c4519b97d265c8ad345d0ee27b50a6303ebce585414bc7978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T01:39:18.418 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:18 smithi139 podman[70304]: 2024-04-25 01:39:18.434067468 +0000 UTC m=+0.384017222 container init a9c8fece0da8856c4519b97d265c8ad345d0ee27b50a6303ebce585414bc7978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T01:39:18.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:18.005+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:18.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:18.005+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:18.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:18.005+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:18.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:18.005+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:18.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:18 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:18.005+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:18.708 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:18 smithi139 podman[70304]: 2024-04-25 01:39:18.437150311 +0000 UTC m=+0.387100066 container start a9c8fece0da8856c4519b97d265c8ad345d0ee27b50a6303ebce585414bc7978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T01:39:18.709 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:18 smithi139 podman[70304]: 2024-04-25 01:39:18.526141614 +0000 UTC m=+0.476091393 container attach a9c8fece0da8856c4519b97d265c8ad345d0ee27b50a6303ebce585414bc7978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD) 2024-04-25T01:39:18.937 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:18.671+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:18.937 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:18.671+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:18.937 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:18.671+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:18.937 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:18.671+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:19.019 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:18 smithi139 podman[70304]: 2024-04-25 01:39:18.727489091 +0000 UTC m=+0.677438854 container died a9c8fece0da8856c4519b97d265c8ad345d0ee27b50a6303ebce585414bc7978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T01:39:19.019 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:18 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:18.850+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:19.019 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:18 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:18.850+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:19.019 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:18 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:18.850+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:19.019 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:18 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:18.850+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:19.019 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:18 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:18.850+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:19.268 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T01:39:19.269 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:19 smithi139 podman[70329]: 2024-04-25 01:39:19.271304466 +0000 UTC m=+0.538087366 container remove a9c8fece0da8856c4519b97d265c8ad345d0ee27b50a6303ebce585414bc7978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-25T01:39:19.269 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:19 smithi139 systemd[1]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-25T01:39:19.269 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:19 smithi139 systemd[1]: Stopped Ceph osd.7 for 1469ec2e-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:39:19.269 INFO:journalctl@ceph.osd.7.smithi139.stdout:Apr 25 01:39:19 smithi139 systemd[1]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.7.service: Consumed 14.898s CPU time. 2024-04-25T01:39:19.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:18.954+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:19.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:18.954+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:19.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:18.954+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:19.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:18 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:18.954+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:19.306 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:19.306 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T01:39:19.306 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T01:39:19.306 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.8 2024-04-25T01:39:19.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:19 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:19.000+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:19.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:19 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:19.000+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:19.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:19 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:19.000+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:19.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:19 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:19.000+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:19.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:19 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:19.000+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:19.615 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:19 smithi153 systemd[1]: Stopping Ceph osd.8 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:19.917 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:19.682+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:19.917 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:19.682+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:19.917 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:19.682+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:19.917 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:19.682+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:19.917 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:19.632+0000 7f1d602e3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:19.918 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:19.633+0000 7f1d602e3700 -1 osd.8 111 *** Got signal Terminated *** 2024-04-25T01:39:19.918 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:19.633+0000 7f1d602e3700 -1 osd.8 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:39:19.918 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:19.934+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:19.918 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:19.934+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:19.918 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:19.934+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:19.918 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:19 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:19.934+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:20.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:19 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:19.865+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:20.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:19 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:19.865+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:20.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:19 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:19.865+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:20.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:19 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:19.865+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:20.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:19 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:19.865+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:20.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:20.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:20.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:20.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:20.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.007+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:20.938 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:20.652+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:20.938 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:20.652+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:20.938 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:20.652+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:20.938 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:20.652+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:21.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:20.956+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:21.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:20.956+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:21.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:20.956+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:21.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:20 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:20.956+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:21.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:20.838+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:21.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:20.838+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:21.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:20.838+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:21.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:20.838+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:21.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:20.838+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:21.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.982+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:21.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.982+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:21.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.982+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:21.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.982+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:21.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:20 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:20.982+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:21.935 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:21.671+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:21.935 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:21.671+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:21.935 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:21.671+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:21.935 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:21.671+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:22.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:21.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:22.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:21.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:22.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:21.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:22.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:21.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:22.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:21.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:22.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:21 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:21.809+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:22.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:21.952+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:22.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:21.952+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:22.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:21.952+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:22.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:21 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:21.952+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:22.447 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:21 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:21.974+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:22.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:21 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:21.974+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:22.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:21 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:21.974+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:22.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:21 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:21.974+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:22.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:21 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:21.974+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:22.910 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:22.627+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:22.910 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:22.627+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:22.910 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:22.627+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:22.910 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:22.627+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:22.910 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:22.627+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:23.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:22.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:23.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:22.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:23.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:22.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:23.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:22.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:23.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:22.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:23.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:22 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:22.790+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:23.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:22.940+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:23.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:22.940+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:23.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:22.940+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:23.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:22.940+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:23.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:22.940+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:23.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:22 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:22.940+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:23.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:22.927+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:23.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:22.927+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:23.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:22.927+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:23.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:22.927+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:23.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:22 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:22.927+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.609942+0000 front 2024-04-25T01:39:00.609997+0000 (oldest deadline 2024-04-25T01:39:22.910019+0000) 2024-04-25T01:39:23.895 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:23.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:23.895 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:23.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:23.895 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:23.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:23.895 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:23.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:23.895 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:23.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:24.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:23.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:24.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:23.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:24.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:23.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:24.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:23.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:24.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:23.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:24.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:23 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:23.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:24.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:23.921+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:24.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:23.921+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:24.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:23.921+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:24.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:23.921+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:24.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:23.921+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:24.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:23 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:23.921+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:24.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:23.911+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.703902+0000 front 2024-04-25T01:38:19.703825+0000 (oldest deadline 2024-04-25T01:38:45.603839+0000) 2024-04-25T01:39:24.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:23.911+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:34.405332+0000 front 2024-04-25T01:38:34.405359+0000 (oldest deadline 2024-04-25T01:38:59.705646+0000) 2024-04-25T01:39:24.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:23.912+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.206765+0000 front 2024-04-25T01:38:43.206833+0000 (oldest deadline 2024-04-25T01:39:05.506810+0000) 2024-04-25T01:39:24.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:23.912+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.009098+0000 front 2024-04-25T01:38:50.009073+0000 (oldest deadline 2024-04-25T01:39:15.308974+0000) 2024-04-25T01:39:24.279 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:23 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8[51245]: 2024-04-25T01:39:23.912+0000 7f1d5c8d7700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.609942+0000 front 2024-04-25T01:39:00.609997+0000 (oldest deadline 2024-04-25T01:39:22.910019+0000) 2024-04-25T01:39:25.001 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:24.618+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:25.002 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:24.618+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:25.002 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:24.618+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:25.002 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:24.618+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:25.002 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:24.618+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:25.197 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:24.954+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:25.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:24.954+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:25.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:24.954+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:25.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:24.954+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:25.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:24.954+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:25.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:24 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:24.954+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:25.278 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:25 smithi153 podman[70392]: 2024-04-25 01:39:25.020047921 +0000 UTC m=+5.504048546 container died 2e1a75727449f6687dcb61d0353d85ff3fdbec4955862b8ec03ddc41d1c9edf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:39:25.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:24 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:24.848+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:25.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:24 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:24.848+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:25.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:24 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:24.848+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:25.295 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:24 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:24.848+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:25.295 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:24 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:24.848+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:25.295 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:24 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:24.848+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:25.632 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:25 smithi153 podman[70392]: 2024-04-25 01:39:25.402596118 +0000 UTC m=+5.886596742 container cleanup 2e1a75727449f6687dcb61d0353d85ff3fdbec4955862b8ec03ddc41d1c9edf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T01:39:25.632 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:25 smithi153 bash[70392]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8 2024-04-25T01:39:25.998 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:25 smithi153 podman[70409]: 2024-04-25 01:39:25.661930279 +0000 UTC m=+0.632096506 container remove 2e1a75727449f6687dcb61d0353d85ff3fdbec4955862b8ec03ddc41d1c9edf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T01:39:25.998 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:25 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:25.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:25.998 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:25 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:25.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:25.998 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:25 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:25.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:25.998 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:25 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:25.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:25.998 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:25 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:25.649+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:26.255 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:26 smithi153 podman[70502]: 2024-04-25 01:39:25.919221861 +0000 UTC m=+0.017950107 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:26.255 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:26 smithi153 podman[70502]: 2024-04-25 01:39:26.020809564 +0000 UTC m=+0.119537802 container create 7f7c3b39a38aa16227dc935349a7be73a93d84ecc93af242ff5eba1a9cbd1514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9) 2024-04-25T01:39:26.294 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:25 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:25.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:26.295 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:25 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:25.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:26.295 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:25 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:25.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:26.295 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:25 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:25.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:26.295 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:25 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:25.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:26.295 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:25 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:25.818+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:26.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:25.977+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:26.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:25.977+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:26.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:25.977+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:26.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:25.977+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:26.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:25.977+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:26.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:25 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:25.977+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:26.528 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:26 smithi153 podman[70502]: 2024-04-25 01:39:26.273305547 +0000 UTC m=+0.372033790 container init 7f7c3b39a38aa16227dc935349a7be73a93d84ecc93af242ff5eba1a9cbd1514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9) 2024-04-25T01:39:26.529 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:26 smithi153 podman[70502]: 2024-04-25 01:39:26.277712059 +0000 UTC m=+0.376440299 container start 7f7c3b39a38aa16227dc935349a7be73a93d84ecc93af242ff5eba1a9cbd1514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=squid-d2c45d9, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:39:26.529 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:26 smithi153 podman[70502]: 2024-04-25 01:39:26.348305546 +0000 UTC m=+0.447033801 container attach 7f7c3b39a38aa16227dc935349a7be73a93d84ecc93af242ff5eba1a9cbd1514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T01:39:26.900 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:26 smithi153 podman[70502]: 2024-04-25 01:39:26.559928848 +0000 UTC m=+0.658657096 container died 7f7c3b39a38aa16227dc935349a7be73a93d84ecc93af242ff5eba1a9cbd1514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T01:39:26.900 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:26 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:26.629+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:26.900 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:26 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:26.629+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:26.900 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:26 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:26.629+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:26.900 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:26 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:26.629+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:26.900 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:26 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:26.629+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:27.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:26 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:26.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:27.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:26 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:26.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:27.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:26 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:26.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:27.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:26 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:26.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:27.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:26 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:26.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:27.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:26 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:26.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:27.157 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 25 01:39:27 smithi153 podman[70528]: 2024-04-25 01:39:27.160600235 +0000 UTC m=+0.594833644 container remove 7f7c3b39a38aa16227dc935349a7be73a93d84ecc93af242ff5eba1a9cbd1514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T01:39:27.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T01:39:27.199 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:27.200 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T01:39:27.200 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T01:39:27.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.9 2024-04-25T01:39:27.227 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:26.992+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:27.228 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:26.992+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:27.228 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:26.992+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:27.228 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:26.992+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:27.228 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:26.992+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:27.228 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:26 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:26.992+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:27.511 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 systemd[1]: Stopping Ceph osd.9 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:27.778 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:27 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:27.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:27.778 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:27 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:27.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:27.778 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:27 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:27.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:27.778 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:27 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:27.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:27.779 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:27 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:27.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:27.948 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.528+0000 7f12bcb01700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:27.948 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.528+0000 7f12bcb01700 -1 osd.9 111 *** Got signal Terminated *** 2024-04-25T01:39:27.948 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.528+0000 7f12bcb01700 -1 osd.9 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:39:28.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:27 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:27.803+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:28.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:27 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:27.803+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:28.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:27 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:27.803+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:28.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:27 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:27.803+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:28.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:27 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:27.803+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:28.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:27 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:27.803+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:28.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.976+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:28.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.976+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:28.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.976+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:28.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.976+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:28.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.976+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:28.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:27 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:27.976+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:29.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:28 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:28.673+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:29.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:28 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:28.673+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:29.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:28 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:28.673+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:29.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:28 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:28.673+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:29.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:28 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:28.673+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:29.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:28 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:28.799+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:29.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:28 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:28.799+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:29.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:28 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:28.799+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:29.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:28 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:28.799+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:29.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:28 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:28.799+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:29.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:28 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:28.799+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:29.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.012+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:29.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.012+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:29.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.012+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:29.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.012+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:29.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.012+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:29.448 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.012+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:30.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:29 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:29.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:30.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:29 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:29.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:30.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:29 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:29.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:30.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:29 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:29.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:30.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:29 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:29.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:30.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:29 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:29.779+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:30.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:29 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:29.779+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:30.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:29 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:29.779+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:30.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:29 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:29.779+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:30.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:29 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:29.779+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:30.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:29 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:29.779+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:30.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:30.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:30.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:30.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:30.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:30.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:29 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:29.964+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:31.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:30 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:30.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:31.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:30 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:30.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:31.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:30 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:30.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:31.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:30 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:30.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:31.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:30 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:30.700+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:31.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:30 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:30.747+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:31.044 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:30 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:30.747+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:31.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:30 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:30.747+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:31.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:30 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:30.747+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:31.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:30 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:30.747+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:31.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:30 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:30.747+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:31.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:30 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:30.916+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:31.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:30 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:30.916+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:31.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:30 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:30.916+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:31.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:30 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:30.916+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:31.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:30 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:30.916+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:31.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:30 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:30.916+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:31.929 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:31 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:31.695+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:31.930 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:31 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:31.695+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:31.930 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:31 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:31.695+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:31.930 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:31 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:31.695+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:31.930 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:31 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:31.695+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:32.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:31 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:31.762+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:32.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:31 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:31.762+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:32.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:31 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:31.762+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:32.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:31 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:31.762+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:32.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:31 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:31.762+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:32.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:31 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:31.762+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:32.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:31 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:31.762+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:32.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:31 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:31.933+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.882879+0000 front 2024-04-25T01:38:19.882845+0000 (oldest deadline 2024-04-25T01:38:45.183025+0000) 2024-04-25T01:39:32.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:31 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:31.933+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:26.185003+0000 front 2024-04-25T01:38:26.185221+0000 (oldest deadline 2024-04-25T01:38:49.685114+0000) 2024-04-25T01:39:32.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:31 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:31.933+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:35.986827+0000 front 2024-04-25T01:38:35.986979+0000 (oldest deadline 2024-04-25T01:39:00.686977+0000) 2024-04-25T01:39:32.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:31 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:31.933+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:43.488821+0000 front 2024-04-25T01:38:43.488841+0000 (oldest deadline 2024-04-25T01:39:07.589031+0000) 2024-04-25T01:39:32.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:31 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:31.933+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:51.590483+0000 front 2024-04-25T01:38:51.590512+0000 (oldest deadline 2024-04-25T01:39:17.490624+0000) 2024-04-25T01:39:32.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:31 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9[63915]: 2024-04-25T01:39:31.933+0000 7f12b90f5700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:57.491424+0000 front 2024-04-25T01:38:57.491410+0000 (oldest deadline 2024-04-25T01:39:22.191216+0000) 2024-04-25T01:39:33.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:32 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:32.650+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:33.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:32 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:32.650+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:33.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:32 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:32.650+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:33.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:32 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:32.650+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:33.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:32 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:32.650+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:33.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:32 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:32.805+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:33.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:32 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:32.805+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:33.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:32 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:32.805+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:33.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:32 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:32.805+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:33.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:32 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:32.805+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:33.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:32 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:32.805+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:33.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:32 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:32.805+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:33.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:32 smithi027 podman[100493]: 2024-04-25 01:39:32.962616413 +0000 UTC m=+5.545806620 container died bf7d2bc6bf8074395cf6a8eb968d7c61999864a79a4e466f5d03d666f26cdafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T01:39:33.561 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:33 smithi027 podman[100493]: 2024-04-25 01:39:33.320611679 +0000 UTC m=+5.903801927 container cleanup bf7d2bc6bf8074395cf6a8eb968d7c61999864a79a4e466f5d03d666f26cdafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T01:39:33.561 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:33 smithi027 bash[100493]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9 2024-04-25T01:39:33.887 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:33 smithi027 podman[100510]: 2024-04-25 01:39:33.57987395 +0000 UTC m=+0.612900823 container remove bf7d2bc6bf8074395cf6a8eb968d7c61999864a79a4e466f5d03d666f26cdafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T01:39:34.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:33 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:33.640+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:34.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:33 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:33.640+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:34.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:33 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:33.640+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:34.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:33 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:33.640+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:34.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:33 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:33.640+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:34.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:33 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:33.640+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:34.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:33 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:33.778+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:34.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:33 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:33.778+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:34.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:33 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:33.778+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:34.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:33 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:33.778+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:34.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:33 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:33.778+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:34.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:33 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:33.778+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:34.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:33 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:33.778+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:34.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:33 smithi027 podman[100599]: 2024-04-25 01:39:33.807823176 +0000 UTC m=+0.018704247 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:34.198 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:33 smithi027 podman[100599]: 2024-04-25 01:39:33.913974255 +0000 UTC m=+0.124855305 container create 42d78a8620fcbfac6cc088cfbbcd5024c35c8eb479c5d20bfd7047b24ef012bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T01:39:34.199 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:34 smithi027 podman[100599]: 2024-04-25 01:39:34.150007239 +0000 UTC m=+0.360888286 container init 42d78a8620fcbfac6cc088cfbbcd5024c35c8eb479c5d20bfd7047b24ef012bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:39:34.199 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:34 smithi027 podman[100599]: 2024-04-25 01:39:34.153444974 +0000 UTC m=+0.364326021 container start 42d78a8620fcbfac6cc088cfbbcd5024c35c8eb479c5d20bfd7047b24ef012bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T01:39:34.698 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:34 smithi027 podman[100599]: 2024-04-25 01:39:34.224643726 +0000 UTC m=+0.435524787 container attach 42d78a8620fcbfac6cc088cfbbcd5024c35c8eb479c5d20bfd7047b24ef012bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:39:34.698 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 01:39:34 smithi027 podman[100599]: 2024-04-25 01:39:34.452879149 +0000 UTC m=+0.663760204 container died 42d78a8620fcbfac6cc088cfbbcd5024c35c8eb479c5d20bfd7047b24ef012bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:39:35.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:34 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:34.664+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:35.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:34 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:34.664+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:35.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:34 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:34.664+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:35.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:34 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:34.664+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:35.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:34 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:34.664+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:35.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:34 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:34.664+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:35.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:34 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:34.787+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:35.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:34 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:34.787+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:35.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:34 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:34.787+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:35.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:34 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:34.787+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:35.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:34 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:34.787+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:35.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:34 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:34.787+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:35.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:34 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:34.787+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:35.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T01:39:35.173 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:35.173 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T01:39:35.173 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T01:39:35.173 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.10 2024-04-25T01:39:35.464 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 systemd[1]: Stopping Ceph osd.10 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:35.783 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.481+0000 7f5ad75b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:35.783 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.481+0000 7f5ad75b8700 -1 osd.10 111 *** Got signal Terminated *** 2024-04-25T01:39:35.783 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.481+0000 7f5ad75b8700 -1 osd.10 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:39:36.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:35 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:35.714+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:36.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:35 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:35.714+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:36.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:35 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:35.714+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:36.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:35 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:35.714+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:36.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:35 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:35.714+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:36.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:35 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:35.714+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:36.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:36.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:36.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:36.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:36.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:36.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:36.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:35 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:35.800+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:37.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:36 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:36.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:37.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:36 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:36.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:37.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:36 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:36.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:37.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:36 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:36.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:37.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:36 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:36.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:37.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:36 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:36.677+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:37.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:36.780+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:37.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:36.781+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:37.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:36.781+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:37.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:36.781+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:37.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:36.781+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:37.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:36.781+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:37.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:36 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:36.781+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:38.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:37 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:37.635+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:38.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:37 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:37.635+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:38.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:37 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:37.635+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:38.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:37 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:37.635+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:38.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:37 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:37.635+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:38.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:37 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:37.635+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:38.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:37 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:37.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:38.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:37 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:37.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:38.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:37 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:37.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:38.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:37 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:37.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:38.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:37 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:37.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:38.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:37 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:37.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:38.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:37 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:37.810+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:38.779 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:38 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:38.637+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:38.779 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:38 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:38.637+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:38.779 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:38 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:38.637+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:38.779 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:38 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:38.637+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:38.779 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:38 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:38.637+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:38.779 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:38 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:38.637+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:39.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:38 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:38.769+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:39.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:38 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:38.769+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:39.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:38 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:38.769+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:39.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:38 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:38.769+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:39.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:38 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:38.769+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:39.046 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:38 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:38.769+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:39.046 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:38 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:38.769+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:40.028 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:39 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:39.676+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:40.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:39 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:39.676+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:40.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:39 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:39.676+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:40.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:39 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:39.676+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:40.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:39 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:39.676+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:40.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:39 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:39.676+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:40.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:39 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:39.735+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:19.785017+0000 front 2024-04-25T01:38:19.785054+0000 (oldest deadline 2024-04-25T01:38:42.684881+0000) 2024-04-25T01:39:40.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:39 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:39.735+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:25.586051+0000 front 2024-04-25T01:38:25.586005+0000 (oldest deadline 2024-04-25T01:38:49.086153+0000) 2024-04-25T01:39:40.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:39 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:39.735+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:36.087288+0000 front 2024-04-25T01:38:36.087350+0000 (oldest deadline 2024-04-25T01:38:57.787358+0000) 2024-04-25T01:39:40.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:39 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:39.735+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T01:38:41.888584+0000 front 2024-04-25T01:38:41.888691+0000 (oldest deadline 2024-04-25T01:39:07.188526+0000) 2024-04-25T01:39:40.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:39 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:39.735+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.689495+0000 front 2024-04-25T01:38:50.689394+0000 (oldest deadline 2024-04-25T01:39:15.989592+0000) 2024-04-25T01:39:40.045 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:39 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:39.735+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:39:00.090839+0000 front 2024-04-25T01:39:00.090814+0000 (oldest deadline 2024-04-25T01:39:21.790613+0000) 2024-04-25T01:39:40.046 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:39 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10[57425]: 2024-04-25T01:39:39.735+0000 7f5acfba4700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:06.093256+0000 front 2024-04-25T01:39:06.093383+0000 (oldest deadline 2024-04-25T01:39:30.793151+0000) 2024-04-25T01:39:41.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:40 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:40.644+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:41.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:40 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:40.644+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:41.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:40 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:40.644+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:41.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:40 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:40.644+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:41.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:40 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:40.644+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:41.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:40 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:40.644+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:41.264 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:41 smithi139 podman[70426]: 2024-04-25 01:39:41.019749997 +0000 UTC m=+5.634969033 container died c1f8fc2196752b815ea4e82182a2e46944f742adb8264e204f7e73279a221a05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T01:39:41.545 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:41 smithi139 podman[70426]: 2024-04-25 01:39:41.381171687 +0000 UTC m=+5.996390762 container cleanup c1f8fc2196752b815ea4e82182a2e46944f742adb8264e204f7e73279a221a05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T01:39:41.545 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:41 smithi139 bash[70426]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10 2024-04-25T01:39:41.966 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:41 smithi139 podman[70442]: 2024-04-25 01:39:41.632756089 +0000 UTC m=+0.614043944 container remove c1f8fc2196752b815ea4e82182a2e46944f742adb8264e204f7e73279a221a05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:39:42.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:41 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:41.626+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:42.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:41 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:41.626+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:42.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:41 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:41.626+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:42.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:41 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:41.626+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:42.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:41 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:41.626+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:42.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:41 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:41.626+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:42.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:41 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:41.626+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-04-25T01:39:16.418219+0000 front 2024-04-25T01:39:16.418298+0000 (oldest deadline 2024-04-25T01:39:41.118149+0000) 2024-04-25T01:39:42.260 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:41 smithi139 podman[70535]: 2024-04-25 01:39:41.983825825 +0000 UTC m=+0.108759686 container create 4a657f09caf21f3a6d0e26f4412f54a6d31339f7ef1012b41378d9858abad84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T01:39:42.260 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:41 smithi139 podman[70535]: 2024-04-25 01:39:41.893124778 +0000 UTC m=+0.018058649 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:42.260 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:42 smithi139 podman[70535]: 2024-04-25 01:39:42.202715372 +0000 UTC m=+0.327649232 container init 4a657f09caf21f3a6d0e26f4412f54a6d31339f7ef1012b41378d9858abad84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T01:39:42.260 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:42 smithi139 podman[70535]: 2024-04-25 01:39:42.20615189 +0000 UTC m=+0.331085750 container start 4a657f09caf21f3a6d0e26f4412f54a6d31339f7ef1012b41378d9858abad84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T01:39:42.260 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:42 smithi139 podman[70535]: 2024-04-25 01:39:42.277583691 +0000 UTC m=+0.402517560 container attach 4a657f09caf21f3a6d0e26f4412f54a6d31339f7ef1012b41378d9858abad84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:39:42.545 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:42 smithi139 podman[70535]: 2024-04-25 01:39:42.496383759 +0000 UTC m=+0.621317612 container died 4a657f09caf21f3a6d0e26f4412f54a6d31339f7ef1012b41378d9858abad84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-25T01:39:43.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:42 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:42.665+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:43.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:42 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:42.665+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:43.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:42 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:42.665+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:43.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:42 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:42.665+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:43.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:42 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:42.665+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:43.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:42 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:42.665+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:43.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:42 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:42.665+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-04-25T01:39:16.418219+0000 front 2024-04-25T01:39:16.418298+0000 (oldest deadline 2024-04-25T01:39:41.118149+0000) 2024-04-25T01:39:43.040 INFO:journalctl@ceph.osd.10.smithi139.stdout:Apr 25 01:39:43 smithi139 podman[70561]: 2024-04-25 01:39:43.039274217 +0000 UTC m=+0.537535687 container remove 4a657f09caf21f3a6d0e26f4412f54a6d31339f7ef1012b41378d9858abad84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:39:43.045 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T01:39:43.081 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:43.081 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T01:39:43.081 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T01:39:43.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.11 2024-04-25T01:39:43.370 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 systemd[1]: Stopping Ceph osd.11 for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:43.696 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.387+0000 7fd300f27700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:43.696 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.387+0000 7fd300f27700 -1 osd.11 111 *** Got signal Terminated *** 2024-04-25T01:39:43.696 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.387+0000 7fd300f27700 -1 osd.11 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:39:44.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.713+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:44.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.713+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:44.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.713+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:44.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.713+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:44.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.713+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:44.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.713+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:44.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:43 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:43.713+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-04-25T01:39:16.418219+0000 front 2024-04-25T01:39:16.418298+0000 (oldest deadline 2024-04-25T01:39:41.118149+0000) 2024-04-25T01:39:45.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:44.732+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:45.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:44.732+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:45.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:44.732+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:45.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:44.732+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:45.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:44.732+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:45.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:44.732+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:45.030 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:44 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:44.732+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-04-25T01:39:16.418219+0000 front 2024-04-25T01:39:16.418298+0000 (oldest deadline 2024-04-25T01:39:41.118149+0000) 2024-04-25T01:39:46.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:45.699+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:46.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:45.699+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:46.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:45.699+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:46.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:45.699+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:46.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:45.699+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:46.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:45.699+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:46.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:45 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:45.699+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-04-25T01:39:16.418219+0000 front 2024-04-25T01:39:16.418298+0000 (oldest deadline 2024-04-25T01:39:41.118149+0000) 2024-04-25T01:39:47.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:46.680+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:47.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:46.680+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:47.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:46.680+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:47.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:46.680+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:47.030 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:46.680+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:47.030 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:46.680+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:47.030 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:46.680+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-04-25T01:39:16.418219+0000 front 2024-04-25T01:39:16.418298+0000 (oldest deadline 2024-04-25T01:39:41.118149+0000) 2024-04-25T01:39:47.030 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:46 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:46.680+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6820 osd.8 since back 2024-04-25T01:39:21.118597+0000 front 2024-04-25T01:39:21.118575+0000 (oldest deadline 2024-04-25T01:39:46.418592+0000) 2024-04-25T01:39:48.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:47.729+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T01:38:20.310687+0000 front 2024-04-25T01:38:20.310710+0000 (oldest deadline 2024-04-25T01:38:43.810696+0000) 2024-04-25T01:39:48.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:47.729+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6804 osd.1 since back 2024-04-25T01:38:23.811552+0000 front 2024-04-25T01:38:23.811325+0000 (oldest deadline 2024-04-25T01:38:49.111392+0000) 2024-04-25T01:39:48.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:47.729+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6804 osd.2 since back 2024-04-25T01:38:33.812510+0000 front 2024-04-25T01:38:33.812487+0000 (oldest deadline 2024-04-25T01:38:57.912439+0000) 2024-04-25T01:39:48.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:47.729+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6812 osd.4 since back 2024-04-25T01:38:50.114692+0000 front 2024-04-25T01:38:50.114587+0000 (oldest deadline 2024-04-25T01:39:14.214668+0000) 2024-04-25T01:39:48.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:47.729+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6812 osd.5 since back 2024-04-25T01:38:56.515621+0000 front 2024-04-25T01:38:56.515665+0000 (oldest deadline 2024-04-25T01:39:21.815759+0000) 2024-04-25T01:39:48.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:47.729+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T01:39:05.916831+0000 front 2024-04-25T01:39:05.916876+0000 (oldest deadline 2024-04-25T01:39:32.917369+0000) 2024-04-25T01:39:48.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:47.729+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.139:6820 osd.7 since back 2024-04-25T01:39:16.418219+0000 front 2024-04-25T01:39:16.418298+0000 (oldest deadline 2024-04-25T01:39:41.118149+0000) 2024-04-25T01:39:48.030 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:47 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11[58242]: 2024-04-25T01:39:47.729+0000 7fd2f9513700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.153:6820 osd.8 since back 2024-04-25T01:39:21.118597+0000 front 2024-04-25T01:39:21.118575+0000 (oldest deadline 2024-04-25T01:39:46.418592+0000) 2024-04-25T01:39:49.029 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:48 smithi153 podman[70619]: 2024-04-25 01:39:48.773491384 +0000 UTC m=+5.485237545 container died 1a923186da809e9c7e6be101f0c1dfd6021530b58c78e6463b77c88afa9c57b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9) 2024-04-25T01:39:49.314 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:49 smithi153 podman[70619]: 2024-04-25 01:39:49.129898638 +0000 UTC m=+5.841644802 container cleanup 1a923186da809e9c7e6be101f0c1dfd6021530b58c78e6463b77c88afa9c57b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T01:39:49.314 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:49 smithi153 bash[70619]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11 2024-04-25T01:39:49.705 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:49 smithi153 podman[70636]: 2024-04-25 01:39:49.330480079 +0000 UTC m=+0.548620160 container remove 1a923186da809e9c7e6be101f0c1dfd6021530b58c78e6463b77c88afa9c57b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:39:49.990 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:49 smithi153 podman[70731]: 2024-04-25 01:39:49.722697611 +0000 UTC m=+0.092108250 container create e127a2dffd2f183abe9d3b431496ee0ca9eecf9e0b1fc57977368b89ae6e0993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:39:49.991 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:49 smithi153 podman[70731]: 2024-04-25 01:39:49.648724584 +0000 UTC m=+0.018135256 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:49.991 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:49 smithi153 podman[70731]: 2024-04-25 01:39:49.933218163 +0000 UTC m=+0.302628803 container init e127a2dffd2f183abe9d3b431496ee0ca9eecf9e0b1fc57977368b89ae6e0993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T01:39:49.991 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:49 smithi153 podman[70731]: 2024-04-25 01:39:49.936784527 +0000 UTC m=+0.306195177 container start e127a2dffd2f183abe9d3b431496ee0ca9eecf9e0b1fc57977368b89ae6e0993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T01:39:49.991 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:50 smithi153 podman[70731]: 2024-04-25 01:39:50.008025179 +0000 UTC m=+0.377435828 container attach e127a2dffd2f183abe9d3b431496ee0ca9eecf9e0b1fc57977368b89ae6e0993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:39:50.279 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:50 smithi153 podman[70731]: 2024-04-25 01:39:50.227513537 +0000 UTC m=+0.596924185 container died e127a2dffd2f183abe9d3b431496ee0ca9eecf9e0b1fc57977368b89ae6e0993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T01:39:50.749 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 25 01:39:50 smithi153 podman[70758]: 2024-04-25 01:39:50.752716722 +0000 UTC m=+0.519569704 container remove e127a2dffd2f183abe9d3b431496ee0ca9eecf9e0b1fc57977368b89ae6e0993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T01:39:50.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T01:39:50.793 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:50.793 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T01:39:50.793 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T01:39:50.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.a 2024-04-25T01:39:51.113 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:39:50 smithi027 systemd[1]: Stopping Ceph mds.a for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:51.449 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:39:51 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a[70964]: 2024-04-25T01:39:51.130+0000 7fad7c647700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:51.449 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:39:51 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a[70964]: 2024-04-25T01:39:51.130+0000 7fad7c647700 -1 mds.a *** got signal Terminated *** 2024-04-25T01:39:53.805 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:39:53 smithi027 podman[100720]: 2024-04-25 01:39:53.556952199 +0000 UTC m=+2.539440138 container died ff31018298463f9cc3b50c0a3fa27d5e1af3ec23b3d61cef4a3b1c2703f45ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T01:39:54.199 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:39:53 smithi027 podman[100720]: 2024-04-25 01:39:53.921671004 +0000 UTC m=+2.904158943 container cleanup ff31018298463f9cc3b50c0a3fa27d5e1af3ec23b3d61cef4a3b1c2703f45ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T01:39:54.199 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 01:39:53 smithi027 bash[100720]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-a 2024-04-25T01:39:54.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T01:39:54.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:54.519 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T01:39:54.519 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T01:39:54.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.d 2024-04-25T01:39:54.875 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:39:54 smithi027 systemd[1]: Stopping Ceph mds.d for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:55.199 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:39:54 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d[69965]: 2024-04-25T01:39:54.892+0000 7f5cf9aa6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:55.199 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:39:54 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d[69965]: 2024-04-25T01:39:54.892+0000 7f5cf9aa6700 -1 mds.d *** got signal Terminated *** 2024-04-25T01:39:57.367 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:39:57 smithi027 podman[100842]: 2024-04-25 01:39:57.098145949 +0000 UTC m=+2.337584665 container died b7ee065b9d57978bf98177a772f75907e95e792f9c79d2edda615d47cca41568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T01:39:57.699 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:39:57 smithi027 podman[100842]: 2024-04-25 01:39:57.475932265 +0000 UTC m=+2.715370957 container cleanup b7ee065b9d57978bf98177a772f75907e95e792f9c79d2edda615d47cca41568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T01:39:57.699 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 01:39:57 smithi027 bash[100842]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-d 2024-04-25T01:39:57.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T01:39:57.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:57.994 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T01:39:57.994 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T01:39:57.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.g 2024-04-25T01:39:58.199 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 01:39:58 smithi027 systemd[1]: Stopping Ceph mds.g for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:58.699 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 01:39:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-g[70571]: 2024-04-25T01:39:58.337+0000 7fd6708f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:58.699 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 01:39:58 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-g[70571]: 2024-04-25T01:39:58.337+0000 7fd6708f1700 -1 mds.g *** got signal Terminated *** 2024-04-25T01:40:02.596 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 01:40:02 smithi027 podman[100964]: 2024-04-25 01:40:02.345985498 +0000 UTC m=+4.115443467 container died 1b992ee592615932c813eb2319d1fa7bfadfbae4e9a6fae834fd3941e69eba88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=) 2024-04-25T01:40:02.855 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 01:40:02 smithi027 podman[100964]: 2024-04-25 01:40:02.696097738 +0000 UTC m=+4.465555707 container cleanup 1b992ee592615932c813eb2319d1fa7bfadfbae4e9a6fae834fd3941e69eba88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T01:40:02.855 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 01:40:02 smithi027 bash[100964]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-g 2024-04-25T01:40:03.106 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 01:40:02 smithi027 podman[100980]: 2024-04-25 01:40:02.872280432 +0000 UTC m=+0.522076800 container remove 1b992ee592615932c813eb2319d1fa7bfadfbae4e9a6fae834fd3941e69eba88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T01:40:03.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T01:40:03.142 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:03.143 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T01:40:03.143 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T01:40:03.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.j 2024-04-25T01:40:03.449 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:40:03 smithi027 systemd[1]: Stopping Ceph mds.j for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:03.949 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:40:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j[69614]: 2024-04-25T01:40:03.490+0000 7f8cd95ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:03.949 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:40:03 smithi027 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j[69614]: 2024-04-25T01:40:03.490+0000 7f8cd95ab700 -1 mds.j *** got signal Terminated *** 2024-04-25T01:40:06.449 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:40:06 smithi027 podman[101087]: 2024-04-25 01:40:06.142627055 +0000 UTC m=+2.762850924 container died 7a740179891c491d8603db77ffe031c065a1ec74bc8574d81f5113f3d1a74118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T01:40:06.917 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:40:06 smithi027 podman[101087]: 2024-04-25 01:40:06.591407266 +0000 UTC m=+3.211631081 container cleanup 7a740179891c491d8603db77ffe031c065a1ec74bc8574d81f5113f3d1a74118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T01:40:06.917 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:40:06 smithi027 bash[101087]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j 2024-04-25T01:40:06.917 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 01:40:06 smithi027 podman[101101]: 2024-04-25 01:40:06.734122412 +0000 UTC m=+0.584541774 container remove 7a740179891c491d8603db77ffe031c065a1ec74bc8574d81f5113f3d1a74118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T01:40:06.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T01:40:06.968 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:06.968 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T01:40:06.968 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T01:40:06.968 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.b 2024-04-25T01:40:07.273 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:40:07 smithi139 systemd[1]: Stopping Ceph mds.b for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:07.546 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:40:07 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b[63104]: 2024-04-25T01:40:07.288+0000 7f7702aaa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:07.546 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:40:07 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b[63104]: 2024-04-25T01:40:07.288+0000 7f7702aaa700 -1 mds.b *** got signal Terminated *** 2024-04-25T01:40:11.531 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:40:11 smithi139 podman[70642]: 2024-04-25 01:40:11.29493699 +0000 UTC m=+4.112353629 container died ffb82643ef8d98b16d1c8b6cb78a688ec9f52505b91bb94dda1bdc99ca1b7ddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T01:40:11.796 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:40:11 smithi139 podman[70642]: 2024-04-25 01:40:11.663785779 +0000 UTC m=+4.481202428 container cleanup ffb82643ef8d98b16d1c8b6cb78a688ec9f52505b91bb94dda1bdc99ca1b7ddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:40:11.796 INFO:journalctl@ceph.mds.b.smithi139.stdout:Apr 25 01:40:11 smithi139 bash[70642]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-b 2024-04-25T01:40:12.073 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T01:40:12.107 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:12.107 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T01:40:12.108 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T01:40:12.108 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.e 2024-04-25T01:40:12.418 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:40:12 smithi139 systemd[1]: Stopping Ceph mds.e for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:12.796 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:40:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e[62486]: 2024-04-25T01:40:12.433+0000 7fa9531af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:12.796 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:40:12 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e[62486]: 2024-04-25T01:40:12.433+0000 7fa9531af700 -1 mds.e *** got signal Terminated *** 2024-04-25T01:40:15.943 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:40:15 smithi139 podman[70762]: 2024-04-25 01:40:15.691825087 +0000 UTC m=+3.354904544 container died 9d7cc3edf7cf7f5bce374332e4f84fcf2b19a46cfa59a4bad3b7826a933e5993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=) 2024-04-25T01:40:16.296 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:40:16 smithi139 podman[70762]: 2024-04-25 01:40:16.059509336 +0000 UTC m=+3.722588793 container cleanup 9d7cc3edf7cf7f5bce374332e4f84fcf2b19a46cfa59a4bad3b7826a933e5993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T01:40:16.296 INFO:journalctl@ceph.mds.e.smithi139.stdout:Apr 25 01:40:16 smithi139 bash[70762]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-e 2024-04-25T01:40:16.480 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T01:40:16.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:16.511 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T01:40:16.511 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T01:40:16.512 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.h 2024-04-25T01:40:17.046 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:40:16 smithi139 systemd[1]: Stopping Ceph mds.h for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:17.046 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:40:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h[62797]: 2024-04-25T01:40:16.845+0000 7fea543cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:17.047 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:40:16 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h[62797]: 2024-04-25T01:40:16.845+0000 7fea543cb700 -1 mds.h *** got signal Terminated *** 2024-04-25T01:40:19.939 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:40:19 smithi139 podman[70882]: 2024-04-25 01:40:19.694081151 +0000 UTC m=+2.953981994 container died e5e6102d99a620215e55b8d410fb9fceafedccafbd7a198f5bfb45ca39033370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T01:40:20.296 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:40:20 smithi139 podman[70882]: 2024-04-25 01:40:20.029821165 +0000 UTC m=+3.289722006 container cleanup e5e6102d99a620215e55b8d410fb9fceafedccafbd7a198f5bfb45ca39033370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T01:40:20.297 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:40:20 smithi139 bash[70882]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h 2024-04-25T01:40:20.297 INFO:journalctl@ceph.mds.h.smithi139.stdout:Apr 25 01:40:20 smithi139 podman[70895]: 2024-04-25 01:40:20.180749883 +0000 UTC m=+0.484168510 container remove e5e6102d99a620215e55b8d410fb9fceafedccafbd7a198f5bfb45ca39033370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-04-25T01:40:20.385 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T01:40:20.418 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:20.418 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T01:40:20.418 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T01:40:20.418 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.k 2024-04-25T01:40:20.725 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:40:20 smithi139 systemd[1]: Stopping Ceph mds.k for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:21.046 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:40:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k[62166]: 2024-04-25T01:40:20.741+0000 7eff37478700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:21.047 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:40:20 smithi139 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k[62166]: 2024-04-25T01:40:20.741+0000 7eff37478700 -1 mds.k *** got signal Terminated *** 2024-04-25T01:40:23.409 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:40:23 smithi139 podman[71005]: 2024-04-25 01:40:23.078837041 +0000 UTC m=+2.420494362 container died 7b6f4eeef614a87f2c5a1a82db7318877cdd89a3f418d6ff7ccd07809358e8b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T01:40:23.767 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:40:23 smithi139 podman[71005]: 2024-04-25 01:40:23.524824834 +0000 UTC m=+2.866482144 container cleanup 7b6f4eeef614a87f2c5a1a82db7318877cdd89a3f418d6ff7ccd07809358e8b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T01:40:23.768 INFO:journalctl@ceph.mds.k.smithi139.stdout:Apr 25 01:40:23 smithi139 bash[71005]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-k 2024-04-25T01:40:23.939 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T01:40:23.968 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:23.969 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T01:40:23.969 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T01:40:23.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.c 2024-04-25T01:40:24.278 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 01:40:24 smithi153 systemd[1]: Stopping Ceph mds.c for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:24.530 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 01:40:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-c[63707]: 2024-04-25T01:40:24.292+0000 7f40d3f48700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:24.530 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 01:40:24 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-c[63707]: 2024-04-25T01:40:24.292+0000 7f40d3f48700 -1 mds.c *** got signal Terminated *** 2024-04-25T01:40:27.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 01:40:27 smithi153 podman[70840]: 2024-04-25 01:40:27.695539271 +0000 UTC m=+3.521870267 container died dd126713b2e74fecc8d44597ed16fc567a53e37d0a8d21d2ff3c7849c5d5c05c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-25T01:40:28.280 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 01:40:28 smithi153 podman[70840]: 2024-04-25 01:40:28.059432548 +0000 UTC m=+3.885763534 container cleanup dd126713b2e74fecc8d44597ed16fc567a53e37d0a8d21d2ff3c7849c5d5c05c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T01:40:28.280 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 25 01:40:28 smithi153 bash[70840]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-c 2024-04-25T01:40:28.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T01:40:28.466 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:28.466 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T01:40:28.467 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T01:40:28.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.f 2024-04-25T01:40:29.030 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:40:28 smithi153 systemd[1]: Stopping Ceph mds.f for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:29.031 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:40:28 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f[63398]: 2024-04-25T01:40:28.795+0000 7f9c41bf6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:29.031 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:40:28 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f[63398]: 2024-04-25T01:40:28.795+0000 7f9c41bf6700 -1 mds.f *** got signal Terminated *** 2024-04-25T01:40:32.014 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:40:31 smithi153 podman[70973]: 2024-04-25 01:40:31.694589664 +0000 UTC m=+2.987382651 container died 80297802d4b3fa7f0e69b986469141df334fc4c27b41fa2cc418c7a4c7d2df48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T01:40:32.280 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:40:32 smithi153 podman[70973]: 2024-04-25 01:40:32.030143706 +0000 UTC m=+3.322936691 container cleanup 80297802d4b3fa7f0e69b986469141df334fc4c27b41fa2cc418c7a4c7d2df48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T01:40:32.281 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:40:32 smithi153 bash[70973]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f 2024-04-25T01:40:32.281 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 25 01:40:32 smithi153 podman[70987]: 2024-04-25 01:40:32.206079633 +0000 UTC m=+0.508933193 container remove 80297802d4b3fa7f0e69b986469141df334fc4c27b41fa2cc418c7a4c7d2df48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:40:32.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T01:40:32.464 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:32.465 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T01:40:32.465 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T01:40:32.465 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.i 2024-04-25T01:40:32.776 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:40:32 smithi153 systemd[1]: Stopping Ceph mds.i for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:33.030 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:40:32 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i[64015]: 2024-04-25T01:40:32.790+0000 7fe8679ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:33.031 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:40:32 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i[64015]: 2024-04-25T01:40:32.790+0000 7fe8679ed700 -1 mds.i *** got signal Terminated *** 2024-04-25T01:40:35.943 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:40:35 smithi153 podman[71094]: 2024-04-25 01:40:35.688788803 +0000 UTC m=+2.995260656 container died 4b77951ab54e6b431f164bd567c6857fe3ade63e551310cf25f03cbdae7d6df9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T01:40:36.277 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:40:36 smithi153 podman[71094]: 2024-04-25 01:40:36.041827487 +0000 UTC m=+3.348299334 container cleanup 4b77951ab54e6b431f164bd567c6857fe3ade63e551310cf25f03cbdae7d6df9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T01:40:36.278 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 25 01:40:36 smithi153 bash[71094]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-i 2024-04-25T01:40:36.397 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T01:40:36.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:36.428 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T01:40:36.428 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T01:40:36.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.l 2024-04-25T01:40:36.530 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:40:36 smithi153 systemd[1]: Stopping Ceph mds.l for 1469ec2e-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:37.030 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:40:36 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l[63078]: 2024-04-25T01:40:36.745+0000 7fd7bf389700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:37.031 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:40:36 smithi153 ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l[63078]: 2024-04-25T01:40:36.745+0000 7fd7bf389700 -1 mds.l *** got signal Terminated *** 2024-04-25T01:40:40.364 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:40:40 smithi153 podman[71214]: 2024-04-25 01:40:40.129816921 +0000 UTC m=+3.466529229 container died 5ec7d31dad34259e92f4362d6d3489039f83cb762b7d226e2ca9dcc7e0ebd2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T01:40:40.615 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:40:40 smithi153 podman[71214]: 2024-04-25 01:40:40.479646147 +0000 UTC m=+3.816358447 container cleanup 5ec7d31dad34259e92f4362d6d3489039f83cb762b7d226e2ca9dcc7e0ebd2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T01:40:40.616 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 25 01:40:40 smithi153 bash[71214]: ceph-1469ec2e-02a1-11ef-bc93-c7b262605968-mds-l 2024-04-25T01:40:40.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1469ec2e-02a1-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T01:40:40.867 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:40.867 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T01:40:40.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:40:41.067 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:40:43.604 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:40:43.793 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:40:46.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:40:46.427 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:40:48.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:40:48.804 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:40:48.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:40:48.862 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T01:40:48.864 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi027/crash 2024-04-25T01:40:48.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:40:48.906 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:40:48.906 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:40:48.908 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi139/crash 2024-04-25T01:40:48.909 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:40:48.948 INFO:teuthology.orchestra.run.smithi139.stderr:tar: /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:40:48.948 INFO:teuthology.orchestra.run.smithi139.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:40:48.950 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi153/crash 2024-04-25T01:40:48.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:40:48.987 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:40:48.987 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:40:48.988 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T01:40:48.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T01:40:49.030 INFO:tasks.cephadm:Compressing logs... 2024-04-25T01:40:49.030 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:40:49.072 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:40:49.075 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:40:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log 2024-04-25T01:40:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:: No such file or directory 2024-04-25T01:40:49.097 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:40:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log: -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T01:40:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T01:40:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T01:40:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T01:40:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T01:40:49.112 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log 2024-04-25T01:40:49.112 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-04-25T01:40:49.112 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T01:40:49.112 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T01:40:49.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:40:49.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.c.log: /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.6% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T01:40:49.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.1% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T01:40:49.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.a.log: 87.6% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr: 90.0% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log: 94.6% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr: 84.1% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.a.log: 94.7%gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T01:40:49.116 INFO:teuthology.orchestra.run.smithi027.stderr: 94.5% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T01:40:49.117 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:40:49.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-25T01:40:49.120 INFO:teuthology.orchestra.run.smithi139.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log 2024-04-25T01:40:49.120 INFO:teuthology.orchestra.run.smithi139.stderr:: No such file or directory 2024-04-25T01:40:49.120 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:40:49.120 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T01:40:49.120 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T01:40:49.120 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.h.log: 87.6% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr: 90.1% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.e.log: 92.0% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T01:40:49.121 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:40:49.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-25T01:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:40:49.127 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T01:40:49.128 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:40:49.128 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:40:49.130 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T01:40:49.134 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.5% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T01:40:49.136 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/cephadm.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:40:49.138 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:40:49.154 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 94.6% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T01:40:49.156 INFO:teuthology.orchestra.run.smithi153.stderr: 91.0% 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:40:49.156 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:40:49.168 INFO:teuthology.orchestra.run.smithi153.stderr: 93.1% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T01:40:49.169 INFO:teuthology.orchestra.run.smithi153.stderr: 94.6% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T01:40:49.209 INFO:teuthology.orchestra.run.smithi139.stderr: 92.7% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T01:40:49.275 INFO:teuthology.orchestra.run.smithi139.stderr: 94.9% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T01:40:49.466 INFO:teuthology.orchestra.run.smithi139.stderr: 92.7% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T01:40:49.503 INFO:teuthology.orchestra.run.smithi027.stderr: 90.3% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T01:40:49.593 INFO:teuthology.orchestra.run.smithi139.stderr: 92.2% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T01:40:49.603 INFO:teuthology.orchestra.run.smithi153.stderr: 91.9% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T01:40:49.907 INFO:teuthology.orchestra.run.smithi027.stderr: 92.5% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T01:40:49.926 INFO:teuthology.orchestra.run.smithi153.stderr: 93.0% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T01:40:50.031 INFO:teuthology.orchestra.run.smithi027.stderr: 90.5% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T01:40:50.105 INFO:teuthology.orchestra.run.smithi139.stderr: 92.2% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T01:40:50.207 INFO:teuthology.orchestra.run.smithi027.stderr: 92.6% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T01:40:50.252 INFO:teuthology.orchestra.run.smithi027.stderr: 92.3% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T01:40:50.316 INFO:teuthology.orchestra.run.smithi027.stderr: 92.4% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T01:40:50.317 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-04-25T01:40:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m1.231s 2024-04-25T01:40:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m5.494s 2024-04-25T01:40:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m0.192s 2024-04-25T01:40:50.402 INFO:teuthology.orchestra.run.smithi153.stderr: 91.9% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T01:40:50.645 INFO:teuthology.orchestra.run.smithi153.stderr: 92.6% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T01:40:50.683 INFO:teuthology.orchestra.run.smithi153.stderr: 92.9% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T01:40:50.720 INFO:teuthology.orchestra.run.smithi139.stderr: 91.8% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T01:40:51.096 INFO:teuthology.orchestra.run.smithi139.stderr: 92.3% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T01:40:51.206 INFO:teuthology.orchestra.run.smithi153.stderr: 92.4% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T01:40:51.208 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-25T01:40:51.208 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m2.119s 2024-04-25T01:40:51.208 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m7.735s 2024-04-25T01:40:51.208 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.272s 2024-04-25T01:40:51.552 INFO:teuthology.orchestra.run.smithi139.stderr: 91.8% -- replaced with /var/log/ceph/1469ec2e-02a1-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T01:40:51.554 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-04-25T01:40:51.554 INFO:teuthology.orchestra.run.smithi139.stderr:real 0m2.464s 2024-04-25T01:40:51.554 INFO:teuthology.orchestra.run.smithi139.stderr:user 0m7.924s 2024-04-25T01:40:51.554 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m0.287s 2024-04-25T01:40:51.554 INFO:tasks.cephadm:Archiving logs... 2024-04-25T01:40:51.554 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi027/log 2024-04-25T01:40:51.555 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:40:52.377 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi139/log 2024-04-25T01:40:52.378 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:40:53.630 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi153/log 2024-04-25T01:40:53.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:40:54.805 INFO:tasks.cephadm:Removing cluster... 2024-04-25T01:40:54.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 --force 2024-04-25T01:40:55.006 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:40:55.278 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 --force 2024-04-25T01:40:55.474 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:40:55.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1469ec2e-02a1-11ef-bc93-c7b262605968 --force 2024-04-25T01:40:55.934 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: 1469ec2e-02a1-11ef-bc93-c7b262605968 2024-04-25T01:40:56.199 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T01:40:56.199 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:40:56.219 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:40:56.235 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:40:56.250 INFO:tasks.cephadm:Teardown complete 2024-04-25T01:40:56.251 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T01:40:56.265 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T01:40:56.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:40:56.268 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:40:56.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:40:56.387 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T01:40:56.397 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T01:40:56.398 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:40:56.400 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:40:56.403 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:40:56.414 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:40:56.415 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:40:56.416 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:40:56.419 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:40:56.419 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-04-25T01:40:56.419 INFO:teuthology.orchestra.run.smithi139.stdout:^- hv01.front.sepia.ceph.com 2 6 377 58 +1353us[+1353us] +/- 580ms 2024-04-25T01:40:56.419 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -232us[ -237us] +/- 62ms 2024-04-25T01:40:56.419 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +229us[ +229us] +/- 69ms 2024-04-25T01:40:56.419 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:56.420 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:40:56.420 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-25T01:40:56.420 INFO:teuthology.orchestra.run.smithi027.stdout:^- hv01.front.sepia.ceph.com 2 6 377 57 +1366us[+1358us] +/- 580ms 2024-04-25T01:40:56.420 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -214us[ -223us] +/- 62ms 2024-04-25T01:40:56.420 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +238us[ +229us] +/- 69ms 2024-04-25T01:40:56.420 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:56.422 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:40:56.422 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-25T01:40:56.422 INFO:teuthology.orchestra.run.smithi153.stdout:^- hv01.front.sepia.ceph.com 2 6 377 57 +1373us[+1373us] +/- 580ms 2024-04-25T01:40:56.422 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -209us[ -207us] +/- 62ms 2024-04-25T01:40:56.422 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +235us[ +235us] +/- 69ms 2024-04-25T01:40:56.422 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:56.422 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T01:40:56.433 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T01:40:56.434 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T01:40:56.450 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:40:56.547 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:40:56.611 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:40:56.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:40:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:40:56.705 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:40:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:40:56.732 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:40:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:40:56.758 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T01:40:56.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T01:40:56.786 INFO:teuthology.task.internal:Duration was 1990.284468 seconds 2024-04-25T01:40:56.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T01:40:56.803 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T01:40:56.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:40:56.806 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:40:56.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:40:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:40:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:40:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:40:57.335 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T01:40:57.335 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T01:40:57.335 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:40:57.358 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2024-04-25T01:40:57.358 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:40:57.400 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-04-25T01:40:57.400 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:40:57.424 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T01:40:57.424 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:40:57.427 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:40:57.442 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:40:57.491 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T01:40:57.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:40:57.495 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:40:57.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:40:57.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T01:40:57.905 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T01:40:57.905 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:40:57.929 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:40:57.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:40:57.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T01:40:57.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:40:57.994 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:40:57.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:40:58.014 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-04-25T01:40:58.016 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2024-04-25T01:40:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-04-25T01:40:58.105 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:40:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:40:58.120 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:40:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:40:58.135 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:40:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:40:58.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T01:40:58.171 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T01:40:58.172 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi027 2024-04-25T01:40:58.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:40:58.215 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi139 2024-04-25T01:40:58.215 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:40:58.253 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672311/remote/smithi153 2024-04-25T01:40:58.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:40:58.290 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T01:40:58.291 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:40:58.293 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:40:58.295 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:40:58.339 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T01:40:58.353 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T01:40:58.353 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T01:40:58.366 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T01:40:58.366 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:40:58.369 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:40:58.370 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:40:58.381 INFO:teuthology.orchestra.run.smithi027.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:40 /home/ubuntu/cephtest 2024-04-25T01:40:58.382 INFO:teuthology.orchestra.run.smithi139.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:40 /home/ubuntu/cephtest 2024-04-25T01:40:58.393 INFO:teuthology.orchestra.run.smithi153.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:40 /home/ubuntu/cephtest 2024-04-25T01:40:58.394 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T01:40:58.406 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T01:40:58.437 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} duration: 1990.2844681739807 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T01:40:58.437 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:40:58.523 INFO:teuthology.run:pass